Image encoder optimization method and device, equipment and medium

Abstract

The invention relates to the technical field of machine learning, and discloses an image encoder optimization method and device, equipment and a medium, and the method comprises the steps: obtaining a target encoding image, carrying out the block embedding of the target encoding image, and carrying out the linear projection transformation, and obtaining a multi-channel feature map; performing convolution up-sampling on the multi-channel feature map to obtain a pixel-level attention feature map; performing regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix; optimizing the loss function according to the sparse attention feature matrix to obtain a sparse regularization loss function; and performing feature recombination on the sparse attention feature matrix and the multi-channel feature map to obtain target image features, and optimizing the image encoder by using the target image features and a sparse regularization loss function to obtain a target image encoder. The method can realize structured attention guidance and characteristic interpretability of the encoder, and can be applied to business system platforms of financial science and technology, medical health and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to an image encoder optimization method, device, equipment and medium. BACKGROUND

[0002] Image encoder representation learning has made rapid progress in recent years. On the one hand, traditional attention mechanisms have difficulty in accurately focusing on semantic key regions in images when extracting image features. Most methods lack effective guidance mechanisms and cannot naturally concentrate attention weights on truly semantically valuable parts. Artificial labeling or complex post-processing operations are often needed to determine key regions, which not only increases cost and complexity but also limits the model's application and generalization ability in unannotated data scenarios. On the other hand, the encoder training process lacks optimization and explanation synergy. Existing image encoding model training usually separates feature learning from explanation analysis of model decision logic. Without directly incorporating explainability analysis (such as gradient and sparsity intensity) into the training process, the model cannot adjust feature learning according to decision logic during optimization, making it difficult to achieve unified optimization of model decision logic and feature learning, thereby affecting the model's performance and explainability.

[0003] For example, in the medical health field, taking medical image diagnosis as an example, traditional attention mechanisms have difficulty in accurately focusing on semantic key regions such as lesions when extracting medical image features. For example, in identifying early lung cancer nodules in lung CT images, existing methods lack effective guidance mechanisms, and attention weights cannot naturally focus on the micro-nodule area. Some normal tissues are often misjudged as key regions, requiring radiologists to manually label key regions or use complex post-processing algorithms for correction. This not only increases the time and labor costs of diagnosis but also limits the model's application in scenarios such as primary care where there is a lack of sufficient professional annotation resources, affecting the accuracy and timeliness of early lung cancer diagnosis.

[0004] For example, in the financial technology field, taking financial bill recognition and risk assessment as an example, existing image encoding model training separates feature learning from explanation analysis of model decision logic. When identifying the authenticity of bank checks, the encoder only focuses on image feature extraction without incorporating explainability analysis such as gradient into the training process. This makes it difficult for the model to adjust feature learning according to decision logic when faced with new counterfeit bills, making it difficult to clearly explain why the model determines it as counterfeit, affecting the financial regulatory department's ability to accurately control risks and make quick decisions, and hindering the model's ability to continuously optimize and improve in the face of changing financial fraud methods.

[0005] Therefore, how to realize an image encoder with structured attention guidance and model explainability has become a problem to be solved. SUMMARY

[0006] The application provides an image encoder optimization method, device, equipment and medium, which mainly aims to solve the problems that the image encoder does not have structured attention guidance and is not strong in explainability.

[0007] In a first aspect, to achieve the above-mentioned object, the application provides an image encoder optimization method, which comprises the following steps: obtaining a target encoded image, performing multi-scale image block embedding on the target encoded image to obtain an image block sequence, and performing linear projection transformation on the image block sequence to obtain a multi-channel feature map; performing convolution upsampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map; performing sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix; performing function optimization on a pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a sparse regularization loss function; performing adaptive feature recombination on the sparse attention feature matrix and the multi-channel feature map to obtain a target image feature, and performing parameter optimization on a preset image encoder by using the target image feature and the sparse regularization loss function to obtain a target image encoder.

[0008] In a second aspect, the application further provides an image encoder optimization device, which comprises the following modules: a channel feature map generation module, configured to obtain a target encoded image, perform multi-scale image block embedding on the target encoded image to obtain an image block sequence, and perform linear projection transformation on the image block sequence to obtain a multi-channel feature map; an attention feature map convolution module, configured to perform convolution upsampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map; a feature regularization constraint module, configured to perform sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix; a function loss optimization module, configured to perform function optimization on a pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a sparse regularization loss function; a target encoder optimization module, configured to perform adaptive feature recombination on the sparse attention feature matrix and the multi-channel feature map to obtain a target image feature, and perform parameter optimization on a preset image encoder by using the target image feature and the sparse regularization loss function to obtain a target image encoder.

[0009] In a third aspect, the present application also provides an electronic device, comprising: at least one processor; and, a memory connected with the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the image encoder optimization method described above.

[0010] In a fourth aspect, the present application also provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the image encoder optimization method described above.

[0011] In the embodiments of the present application, the target coding image is subjected to a multi-scale image block embedding operation to be disassembled into an image block sequence, breaking the overall structure of the image and presenting in the form of a finer-grained image block, preparing for extracting local features; the image block sequence is subjected to a linear projection transformation to be mapped to a specific feature space, thereby obtaining a multi-channel feature map, which contains feature information of the image block in different dimensions; in order to further mine key information in the image, the multi-channel feature map is subjected to a convolution upsampling processing, through which local patterns and spatial relationships in the feature map can be captured, and the upsampling is helpful to restore the detail information of the image, finally obtaining a pixel-level attention feature map, which can reflect the importance of each attention channel in the image; the pixel-level attention feature map is subjected to a sparse regularization constraint, the purpose of which is to make the attention weight more reasonable and sparse, avoiding excessive dispersion of attention, thereby obtaining a sparse attention feature matrix, highlighting the key regions and features in the image; the pre-constructed contrast loss function is optimized by using the sparse attention feature matrix, obtaining a sparse regularization loss function, combining the attention mechanism with the loss function, making the encoder pay more attention to key features in the training process, and improving the discriminability of the features; the sparse attention feature matrix and the multi-channel feature map are weighted and fused to obtain a target image feature, and the preset image encoder is subjected to parameter optimization by using the target image feature and the sparse regularization loss function, constantly adjusting the parameters of the encoder to make it better extract image features, finally obtaining a target image encoder with excellent performance, to improve the coding effect and further enhance the accuracy of subsequent image processing tasks, realizing the structured attention guidance and feature interpretability of the image encoder. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings described in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0013] Figure 1 An application environment schematic diagram of an image encoder optimization method in an embodiment of the present application is provided. Figure 2 A flowchart schematic diagram of an image encoder optimization method provided in an embodiment of the present application is provided. Figure 3 A flowchart schematic diagram of sparse regularization constraint on a pixel-level attention feature map provided in an embodiment of the present application is provided. Figure 4 A module schematic diagram of an image encoder optimization device provided in an embodiment of the present application is provided. Figure 5 A structure schematic diagram of an electronic device for implementing an image encoder optimization method provided in an embodiment of the present application is provided. Figure 6 Another structure schematic diagram of an electronic device for implementing an image encoder optimization method provided in an embodiment of the present application is provided.

[0014] The purpose of the present application, functional characteristics and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0015] In order to make the person skilled in the art better understand the technical solutions of the present disclosure, and to fully understand and implement the implementation process of the present disclosure how to apply technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in the embodiments of the present disclosure with reference to the drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, not all embodiments. The embodiments of the present disclosure and each feature in the embodiments can be combined with each other without conflict, and the technical solutions formed thereby are all within the protection scope of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present disclosure.

[0016] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present disclosure and above-described drawings are intended to distinguish similar objects and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or apparatus including a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products or apparatus.

[0017] The embodiments of the present application provide an image encoder optimization method. The execution subject of the image encoder optimization method includes but is not limited to at least one of electronic devices capable of being configured to execute the apparatus provided by the embodiments of the present application, such as a server and a terminal. In other words, the image encoder optimization method can be executed by software or hardware installed in a terminal device or a server device. The server includes but is not limited to a single server, a server cluster, a cloud server, or a cloud server cluster, etc. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0018] The image encoder optimization method of the present application can be applied to, for example Figure 1In the application environment, the client communicates with the server through the network. The server can obtain the target coding image through the client, perform multi-scale image block embedding on the target coding image, decompose the complete image into an ordered image block sequence, which is helpful to capture local detail information of the image; perform linear projection transformation on the image block sequence to map it to a specific feature space, which reflects the feature information of the image block in different dimensions; for further mining of key information in the image, perform convolution upsampling processing on the multi-channel feature map, through convolution operation, local patterns and spatial relationships in the feature map can be captured, and upsampling is helpful to restore the detail information of the image, which can reflect the importance of each attention channel in the image; sparse regularization constraint is applied to the pixel-level attention feature map, the purpose is to make the attention weight more reasonable and sparse, avoid attention overdispersion, and highlight the key areas and features in the image; the pre-constructed contrast loss function is optimized by using the sparse attention feature matrix, and the attention mechanism is combined with the loss function, so that the encoder pays more attention to the key features in the training process, and the discriminability of the features is improved; the sparse attention feature matrix and the multi-channel feature map are weighted and fused, and the pre-set image encoder is parameter optimized by using the target image feature and the sparse regularization loss function, the parameters of the encoder are continuously adjusted, so that the image features can be better extracted, and finally a target image encoder with excellent performance is obtained to improve the coding effect, and finally the output of the target image encoder is fed back to the client.

[0019] The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The application will be described in detail through specific embodiments.

[0020] Referring to Figure 2 Fig. 1 shows a flowchart of an image encoder optimization method provided by an embodiment of the application. In this embodiment, the image encoder optimization method comprises: S1, obtaining a target coding image, performing multi-scale image block embedding on the target coding image to obtain an image block sequence, and performing linear projection transformation on the image block sequence to obtain a multi-channel feature map.

[0021] In the embodiment of the application, the target coding image refers to an image form generated by processing an original image through a specific coding technology, which contains target feature information and is convenient for computer processing and recognition; the core is to convert image data into more compact and more discriminative feature representation, while retaining key information to support subsequent tasks (such as target detection, classification or optimization).

[0022] In detail, the traditional encoding methods include Huffman encoding, run-length encoding, etc. Under the deep learning framework, the encoder maps the initial image to a low-dimensional hidden space to generate a compact encoded image, and the decoder attempts to reconstruct the original image from the encoding to optimize the encoding quality through reconstruction error. For example, the autoencoder (Autoencoder) compresses features through unsupervised learning, the variational autoencoder (VAE) introduces a probability distribution constraint to enhance the generation ability, and the mask autoencoder (MAE) improves the understanding of local information by random mask training to improve the understanding ability of the model to complex scenes.

[0023] In the embodiment of the application, the multi-scale image block embedding of the target encoded image comprises: performing image enhancement processing on the target encoded image to obtain an enhanced encoded image; performing adaptive block processing on the enhanced encoded image according to a preset pyramid block window to obtain image blocks that do not overlap with each other; performing local feature extraction on each image block to obtain an initial feature vector of the image block; mapping the initial feature vector to a preset high-dimensional image space to generate an embedding vector; adding position encoding information to the embedding vector, and performing spatial dimension reconstruction on the embedding vector after adding the position encoding information according to the original spatial order of the target encoded image to obtain an image block sequence.

[0024] In detail, the image enhancement aims to optimize the visual quality of the image through nonlinear transformation to provide more robust input for subsequent block processing and feature extraction. Histogram equalization or adaptive contrast enhancement (such as CLAHE) technology is adopted to expand the dynamic range of the image by redistributing the pixel gray value distribution.

[0025] For example, in medical CT images, the density difference between lesions and normal tissues can be highlighted; in financial bill recognition, the contrast between text and background can be enhanced.

[0026] Specifically, Gaussian filtering or non-local mean denoising algorithm is used to eliminate random noise by weighted average of neighborhood pixel values. For example, in medical endoscope images captured under low light conditions, granular noise interference can be reduced; at the same time, Laplace operator is applied to enhance high-frequency components to improve edge definition.

[0027] Among them, the enhanced image is divided into regular non-overlapping blocks with the aim of converting the global problem into a local feature aggregation problem. The pyramid block size is dynamically adjusted according to the optimization task requirements. For example, in medical images, small-size blocks (such as 16×16 pixels) are suitable for capturing the local texture of tiny lesions, while large-size blocks (such as 64×64 pixels) are more suitable for overall organ contour analysis. Sliding window or grid division method is used to ensure that there is no overlap between blocks to avoid feature redundancy.

[0028] Furthermore, the initial feature vector is extracted for each image block to achieve the conversion from low-level pixels to high-level semantics. Statistics such as contrast and correlation are calculated through the gray-level co-occurrence matrix (GLCM) to describe the pixel distribution pattern within the image block. At the same time, the Hough transform or contour detection algorithm is used to extract geometric parameters (such as area, perimeter, and aspect ratio).

[0029] Among them, you can also use the shallow convolutional layer of the pre-trained model (such as ResNet, VGG) to extract local receptive field features, map the initial feature vector to a high-dimensional space, and improve feature discrimination through dimensional expansion. Specifically, the linear projection of the feature space is achieved through the fully connected layer; for example, the 128-dimensional initial vector is mapped to a 512-dimensional high-dimensional space to increase the possibility of feature combination.

[0030] Specifically, position information is added to the embedding vector to ensure that the image encoder (such as Transformer) can perceive the spatial order of image blocks, a unique position identifier is assigned to each image block, a position vector is generated through the sine / cosine function, and the embedding vectors with added position encoding are arranged according to the spatial order of the original image (such as from left to right, from top to bottom) to generate an image block sequence.

[0031] For example, in the medical and health scenario, taking the lesion classification task of medical images (such as lung CT) as an example, after the original CT image is generated by an encoder such as U-Net to generate a target coded image, image enhancement processing is first performed. The contrast is enhanced by histogram equalization, and Gaussian filtering is combined to remove noise to generate an enhanced coded image to highlight the lesion area; then, the enhanced image is divided into non-overlapping image blocks according to a preset block size (such as 16×16 pixels), and each image block covers the local tissue structure (such as lung nodules and blood vessels).

[0032] Specifically, for each image block, a 3D convolution kernel is used to extract local features (such as texture density, edge gradient), generate an initial feature vector, and then map the vector to a 512-dimensional high-dimensional space through a fully connected layer to form an embedding vector to capture more complex semantic associations. In order to preserve spatial position information, two-dimensional position encoding is added to the embedding vector, horizontal direction encoding records the column position of the image block in the CT slice, and vertical direction encoding records the row position, and finally the embedding vectors are arranged in the original spatial order to obtain an image block sequence; Wherein, the sequence input into the image encoder model can effectively model the spatial relationship between the lesion and the surrounding tissue.

[0033] Illustratively, in a bank bill anti-fraud system, the authenticity of the user uploaded bill images such as ID cards and business licenses needs to be verified. After the original bill image is extracted by the OCR encoder, the text and layout information is generated to form a target coded image. In the enhancement processing stage, CLAHE (Contrast Limited Adaptive Histogram Equalization) is used to enhance the contrast between the text and the background, and morphological operations (such as dilation and erosion) are used to repair the image edges to generate an enhanced coded image. The image is divided into non-overlapping blocks according to the preset block size (such as 32x32 pixels), and each block covers the key area of the bill (such as name, ID number, and official seal). For each image block, a shallow convolution layer is used to extract local features (such as text strokes and seal textures) to generate an initial feature vector, and then the vector is mapped to a 256-dimensional high-dimensional space through linear transformation to form an embedding vector to distinguish between real and fake features (such as print marks and material reflections). In the ID forgery detection task, the model detection efficiency and accuracy are improved.

[0034] In the embodiment of the application, the linear projection transformation is performed on the image block sequence to obtain a multi-channel feature map, comprising: The pixel dimension of each image block in the image block sequence is flattened to obtain a target image block sequence; The pixel variance and edge density of each target image block in the target image block sequence are calculated, and the vector dimension and feature channel number of the target image block are obtained; A multi-channel linear projection matrix is constructed according to the pixel variance, edge density, vector dimension, and feature channel number; A matrix multiplication operation is performed on each target image block according to the multi-channel linear projection matrix to obtain a projection result; A bias term is added to the projection result, and a nonlinear transformation is performed on the projection result after adding the bias term to obtain a multi-channel feature map.

[0035] In detail, when the image block is pixel dimensionally flattened, a dimension reshaping method is adopted to convert the pixel matrix of the two-dimensional image block into a one-dimensional vector in row or column priority order, to retain the spatial arrangement information of the pixel values and to ensure that the pixel sequence after flattening has a mapping relationship with the pixel distribution of the original image block.

[0036] In the calculation of the pixel variance, the deviation of the pixel value from the mean in the one-dimensional vector is counted to quantify the brightness fluctuation of the image block, and the edge density is calculated by using an edge detection operator (such as a Sobel operator) to extract the edge features of the image block and to count the proportion of the edge pixels, thereby reflecting the richness of the details of the image block. Both of them are used as quantitative indicators of the complexity of the image block.

[0037] Further, the dimension (such as 512 dimensions) of each target image block in the sequence of image blocks and the preset number of feature channels (such as 64 channels) are obtained, the number of rows of the multi-channel linear projection matrix is equal to the dimension of the target image block, and the number of columns is equal to the number of target channels; the initial weight value of the projection matrix is randomly generated according to the variance distribution law of the input and output dimensions by using the Xavier initialization or He initialization method. If multiple independent feature channels (such as RGB three channels expanded to multi-scale features) need to be generated, the projection matrix can be divided into multiple sub-matrices for processing the transformation of different channel groups.

[0038] Further, the target image block is mapped to a high-dimensional feature space through matrix multiplication, and the matrix multiplication of multiple image blocks is simultaneously processed by using the SIMD (Single Instruction Multiple Data) instruction set or the parallel computing unit of the GPU. If there are a large number of zero values in the projection matrix (such as the matrix optimized by the pruning technique), a sparse matrix storage format (such as the CSR format) and a special multiplication algorithm are used to reduce invalid calculations. For a super large dimension projection matrix (such as a 1024-dimensional input), the matrix is divided into several sub-matrix blocks, and step-by-step calculation is used to avoid memory overflow.

[0039] In the projection result, a bias term is added to adjust the distribution center of the feature value, enhance the expression ability of the model, and dynamically calculate the bias value according to the statistical characteristics (such as the mean and variance) of the projection result. For example, in medical image classification, if the feature value of a certain channel is generally low, the bias term can be used to improve its activation level. An independent bias parameter can also be assigned to each feature channel to allow different channels to have different distribution offsets. The bias term is included in the model training process, and the bias value is automatically optimized through backpropagation.

[0040] Specifically, the projection result after adding the bias is activated nonlinearly, which gives the model the ability to fit complex functions, introduces sparsity through threshold truncation to speed up model convergence, or combines Gaussian error linear units to enhance smoothness while preserving negative value information, dynamically adjusts the activation curve shape (such as Swish function) through parameterization to adapt to different task requirements, and improves the processing ability of the encoder for complex data.

[0041] Illustratively, in a medical image intelligent diagnosis system, accurate analysis of medical images is crucial for disease diagnosis. For example, in lung CT images, the original CT images as target encoding images may have problems such as noise and insufficient contrast, which affect subsequent analysis. Image enhancement processing is performed on the target encoding image, histogram equalization technology is used to improve the image contrast, and the lung tissue and possible lesion area are made clearer to obtain an enhanced encoding image. Then, according to a preset block size, for example, the image is divided into 16x16 pixel blocks, the enhanced encoding image is divided into multiple scales of grid, and non-overlapping image blocks are obtained. These small blocks can focus on the local area of the image.

[0042] Further, local feature extraction is performed on each image block, and a convolutional neural network (CNN) is used to automatically learn the texture, edge and other features in the image block to obtain an initial feature vector of the image block. The initial feature vector is mapped to a preset high-dimensional image space, which can better represent the complex features of the image, and an embedding vector is generated to make the features of different image blocks more distinguishable. Based on this sequence, the intelligent diagnosis system can more accurately locate the lung lesions, such as distinguishing between ordinary nodules and malignant tumors, to provide more reliable diagnostic reference for doctors and improve the accuracy and efficiency of disease diagnosis.

[0043] In the embodiment of the present application, after obtaining the target encoding image, multi-scale image block embedding is performed on it, and the complete image is disassembled into an ordered image block sequence, which helps to capture local detail information of the image, so that the encoder can analyze the image content more carefully, and avoid missing local key features due to overall processing. At the same time, linear projection can fully exploit the potential features of the image, enrich the feature representation, effectively improve the understanding and recognition ability of the encoder for the image, and enhance the accuracy of computer vision tasks.

[0044] S2, performing convolution up-sampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map.

[0045] In the embodiment of the present application, the multi-channel feature map is converted into a pixel-level attention feature map through convolution up-sampling processing combined with a channel attention mechanism, which can gradually extract more abstract and representative features. Through the combination of convolution spatial resolution recovery and attention mechanism, the capture ability of the encoder for microscopic features is significantly improved.

[0046] In the embodiment of the present application, the convolution up-sampling processing on the multi-channel feature map is performed to obtain a pixel-level attention feature map corresponding to the multi-channel feature map, which comprises: The multi-layer dilated convolution processing is performed on the multi-channel feature map to obtain a dilated convolution feature map, and the pixel standardization processing is performed on the dilated convolution feature map to obtain a standardized feature map; The up-sampling model is initialized, and the deep feature extraction is performed on the standardized feature map according to the feature extraction sub-module of the up-sampling model to obtain a deep feature map; The image size of the standardized feature map is obtained, and the up-sampling multiple is calculated according to the preset up-sampling target size and the image size; The deep feature map is up-sampled according to the up-sampling multiple to obtain an up-sampled feature map; The up-sampled feature map and the multi-channel feature map are added pixel by pixel to generate a fusion feature map; The channel attention analysis is performed on the fusion feature map to obtain a channel attention weight matrix, and the fusion feature map is multiplied by the channel attention weight matrix to obtain a pixel-level attention feature map.

[0047] In detail, the channel attention weight matrix is specifically as follows:

[0048] wherein, represents the channel attention weight matrix, represents a preset total number of attention channels, represents a query matrix in the channel attention weight matrix, represents a key matrix in the channel attention weight matrix, represents a preset scaling factor, represents transposition, represents an activation function.

[0049] In detail, different sizes of dilated convolution kernels (such as 3x3, 5x5) are combined to capture local to global feature patterns, the standard convolution is decomposed into depth convolution and point-by-point convolution to reduce the amount of calculation, the skip connection is introduced between the convolution layers to alleviate the gradient vanishing problem of the deep network, the mean-variance normalization is performed on the feature map of each batch to stabilize the input distribution, or the statistics are calculated along the channel dimension to avoid the influence of the fluctuation of the statistics between batches.

[0050] Specifically, when initializing the up-sampling model, the feature extraction sub-module usually adopts a convolutional neural network (CNN) architecture, which gradually extracts deep semantic information of standardized feature maps by stacking convolutional layers and activation functions, and uses pooling layers to retain key features while reducing dimensions, achieving progressive extraction from shallow to deep features; when calculating the up-sampling multiple, based on the width-to-height ratio of the standardized feature map and the target size, the size ratio calculation method is used to determine the magnification; the up-sampling processing commonly uses methods such as bilinear interpolation, deconvolution or transpose convolution, among which the deconvolution learns the convolution kernel parameters to restore details while enlarging the feature map size, so that the up-sampled feature map matches the target size and retains key features.

[0051] Among them, the high-resolution feature map can be reconstructed through a learnable up-sampling kernel, combined with a 1x1 convolution to adjust the number of channels, to achieve a non-parameterized resolution improvement. The up-sampled feature map is combined with the original multi-channel feature map to achieve information complementation, superimpose feature values at the same spatial position, and retain original low-level features and high-level semantic features.

[0052] Among them, by modeling the inter-channel dependency to generate a weight matrix, the feature map of each channel is compressed into a scalar, and the channel statistics are calculated, or by modeling the channel correlation through a two-layer fully connected network to generate attention weights; the channel attention weights are applied to the fused feature map to achieve feature recalibration, and the attention weights are applied to the channel features of each spatial position to enhance key features and suppress redundant information.

[0053] Further, on the basis of channel attention, spatial dimension modeling (such as a 3D attention matrix) is added to achieve pixel-level fine control, for example, in financial document forgery prevention, the pixel-level abnormalities of the local tampered area can be located.

[0054] Illustratively, in medical image analysis, accurate segmentation of lesion areas is crucial for disease diagnosis and treatment plan development. Taking brain magnetic resonance imaging (MRI) images as an example, after preliminary processing of the original images, multi-channel feature maps are obtained, which contain information about different tissue structures in the brain, but may have problems such as feature blurring and unclear boundaries; multi-layer convolution processing is performed on the multi-channel feature maps to extract features at different levels through different size convolution kernels, obtaining convolution feature maps, which can capture local and global features of brain tissue such as tumor shape and texture; then, pixel standardization processing is performed on the convolution feature maps to normalize the value of each pixel to a certain range, obtaining standardized feature maps, which helps to improve the stability and convergence speed of the model.

[0055] Further, the standardized feature map is up-sampled to restore its size close to that of the original image to obtain an up-sampled feature map; the up-sampling can supplement the details lost due to the convolution operation, making the feature map more detailed, and then the up-sampled feature map and the original multi-channel feature map are added pixel by pixel to generate a fusion feature map, and the channel attention analysis is performed on the fusion feature map; the channel attention mechanism can automatically learn the importance of different channel features, and the channel attention weight matrix is obtained by calculation, which reflects the contribution degree of each channel feature to the lesion segmentation; the fusion feature map is weighted and multiplied according to the channel attention weight matrix, the features of important channels are highlighted, and the features of irrelevant channels are suppressed, so as to obtain a pixel-level attention feature map; based on this feature map, doctors can more accurately segment brain lesions, such as accurately distinguishing tumor tissues from normal brain tissues, to provide a reliable basis for subsequent treatment.

[0056] In the embodiment of the application, multi-layer convolution is performed on the multi-channel feature map to enhance the ability of the model to capture key information of the image, and the convolution up-sampling can restore the spatial resolution of the feature map and retain more details, and the combination of spatial resolution restoration and attention mechanism significantly improves the ability of the encoder to capture microscopic features.

[0057] For example, in a financial bill processing system, quickly and accurately identifying key information on the bill, such as account number, amount, date, etc., is the key to improving business processing efficiency and accuracy. Taking a bank check as an example, after pre-processing the check image, a multi-channel feature map is obtained, which contains information such as text, seal, and background on the check. Similarly, multi-layer convolution is performed on the multi-channel feature map to extract local and global features of the check image, such as strokes of text and shapes of seals. Then, pixel standardization processing is performed to obtain a standardized feature map, and then the standardized feature map is up-sampled to restore image details. The up-sampled feature map and the multi-channel feature map are added pixel by pixel to generate a fusion feature map.

[0058] Specifically, the fusion feature map is subjected to channel attention analysis, and since the importance of different information on the check is different, for example, the account number information is more important than the background information, the channel attention mechanism can automatically assign different weights to different channels, and the channel attention weight matrix is obtained by calculation, and the fusion feature map is weighted and multiplied according to the matrix to obtain a pixel-level attention feature map, based on which the key information on the check can be more accurately identified, the recognition error is reduced, and the processing efficiency and security of financial business are improved.

[0059] S3, performing sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix.

[0060] In the embodiment of the present application, the pixel-level attention feature map is converted into a sparse attention feature matrix through global average pooling, sparse-smooth constraint and norm regularization.

[0061] As shown in Figure 3 , in the embodiment of the present application, the sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix comprises: performing spatial dimension analysis on the pixel-level attention feature map to obtain a height pixel number, a width pixel number and a channel number; constructing a multi-scale pooling window according to the height pixel number, the width pixel number and the channel number; performing sliding average pooling on the pixel-level attention feature map using the multi-scale pooling window to obtain a pooled feature map; calculating the mean value of each attention channel in the pooled feature map, and determining a feature dense area and a feature sparse area in the pooled feature map according to the mean value; generating an attention sparse term and an attention smooth term based on the feature dense area and the feature sparse area; performing norm sparse regularization constraint on each attention channel according to the attention sparse term and the attention smooth term to obtain a sparse attention weight vector; performing pixel-by-pixel weighted normalization on the pixel-level attention feature map according to the sparse attention weight vector to obtain a sparse attention feature matrix.

[0062] In detail, the sparse attention weight vector is specifically as follows:

[0063] wherein, represents the channel attention weight matrix, represents the sparse attention weight vector, represents a preset sparse intensity weight coefficient, represents a preset smooth coherence weight coefficient, represents the attention sparse term corresponding to the attention channel in the i-th row and the j-th column, represents the attention smooth term corresponding to the attention channel in the i-th row and the j-th column, represents the attention sparse term corresponding to the attention channel in the i-th row and the j-th column. represents the attention smooth term corresponding to the attention channel in the i-th row and the j-th column. represents the attention sparse term corresponding to the attention channel in the i-th row and the j-th column. represents the attention smooth term corresponding to the attention channel in the i-th row and the j-th column.

[0064] In detail, the spatial dimension resolution extracts the height, width and channel number of the feature map through tensor dimension index, explicitly the spatial scale and depth information of the feature map, and provides dimension basis for subsequent pooling; the multi-scale pooling window construction adopts dynamic size design, generates different proportion of pooling windows (such as 1x1, 3x3, 5x5) based on the height and width obtained by resolution, adapts to different scale feature extraction requirements, when sliding average pooling, the window traverses the feature map by step, takes the average of the covered area pixels, through multi-window parallel processing and fusion of the results, both retains the global feature trend and captures local details, improves the representation ability of the pooled feature map, that is, for each channel in the pixel-level attention feature map, the average value of all pixel points in the channel is calculated, which is equivalent to "compression" of the entire feature map, and the pooled feature map obtained by global average pooling converts the pixel-level attention feature map with spatial information into a more compact representation.

[0065] Among them, for each attention channel in the pooled feature map, the average value of all values in the channel is directly calculated, and the average value reflects the overall intensity level of the attention feature in the channel; the generation of the attention sparsity term is based on some processing of the channel average value, the purpose is to make the attention feature matrix have sparsity, and the region higher than the threshold value is determined as the feature dense area (containing key information), and the region lower than the threshold value is the sparse area (redundant information), the sparsity term uses L1 regularization to suppress the weight of the sparse area, and the smoothing term smoothes the dense area in space through Gaussian filtering or bilateral filtering, reduces the feature jump, enhances the consistency within the area, and finally realizes the fine regulation and control of the attention weight.

[0066] Specifically, sparsity means that only part of the attention weights of the channels are high, and the weights of the remaining channels are close to zero. The attention sparsity term can be obtained by converting or screening the channel average value through a specific rule; for example, a sparsity threshold can be set to suppress the attention weight of the channel whose average value is lower than the sparsity threshold, so as to encourage the encoder to focus attention on a small number of important channels; the generation of the attention smoothing term is to make the attention weight change between adjacent channels more smooth, so as to avoid sharp fluctuations, for example, using the method of moving average, considering the relationship between adjacent channel averages, so that the generated attention smoothing term can guide the attention weight to present a continuous and smooth change trend in the channel dimension.

[0067] Further, norm regularization is a commonly used regularization method for controlling the sparsity and smoothness of vectors, different norms have different characteristics, for example, L1 norm tends to produce sparse solutions, and L2 norm helps to make the solution more smooth, the attention sparsity term and the attention smoothing term are combined to perform norm sparse regularization constraint on the weight vector of each attention channel.

[0068] Specifically, guided by the attention sparsity term and the attention smoothness term, the weight vector of each attention channel is optimized by adjusting the parameter of the norm regularization. After the norm sparsity regularization constraint, the sparse attention weight vector obtained has sparsity and can maintain the continuity of the weights between channels to a certain extent.

[0069] In detail, the pixel-wise weighting and normalization refers to weighting each pixel point in the pixel-level attention feature map according to the weight of the corresponding channel in the sparse attention weight vector, and then performing normalization processing. According to the sparse attention weight vector, the weight of each channel is multiplied by the value of the pixel point in the channel to obtain a weighted value. The weighted pixel point value is normalized, which can usually use linear normalization or other normalization methods to map the pixel point value to a fixed interval, such as [0, 1] or [-1, 1], which can avoid the influence of too large or too small values on subsequent optimization, and make the values between different pixel points comparable.

[0070] Illustratively, in the field of medical image analysis, accurate classification of diseases is a key link to assist doctors in diagnosis and treatment. Taking the diagnosis of pneumonia from chest X-ray images as an example, a pixel-level attention feature map has been obtained through a series of pre-processing. Although this feature map highlights the regions that may be related to pneumonia, it may have feature redundancy and noise interference, affecting the classification accuracy. Global average pooling is performed on the pixel-level attention feature map to compress the entire feature map into a vector to obtain a pooled feature map. The mean value of each attention channel in the pooled feature map is calculated, and the attention sparsity term and the attention smoothness term are generated based on these mean values. The attention sparsity term aims to make the model focus on a few key feature channels and reduce redundant information, and the attention smoothness term ensures that the attention weights between adjacent pixels change smoothly to avoid excessive localization.

[0071] Further, according to the attention sparsity term and the attention smoothness term, each attention channel is subjected to norm sparsity regularization constraint, and the channel weight is adjusted through an optimization algorithm to obtain a sparse attention weight vector. The weights in this vector reflect the importance of each channel to pneumonia classification, and the greater the weight, the more critical the channel feature. Finally, the pixel-level attention feature map is subjected to pixel-wise weighting and normalization according to the sparse attention weight vector, highlighting important features and suppressing irrelevant and noise features to obtain a sparse attention feature matrix. This matrix is input into a classification model, which can more accurately identify pneumonia images and distinguish between different types of pneumonia, such as bacterial pneumonia and viral pneumonia, providing more accurate diagnostic recommendations for doctors.

[0072] In the embodiment of the present application, global average pooling is performed on the pixel-level attention feature map, which can compress information, and at the same time, the mean of each attention channel of the pooled feature map is calculated to generate a sparse term and a smooth term. The former promotes feature sparsification, reduces redundancy, and highlights key features, and the latter ensures feature smoothing and avoids excessive sparsification to damage information integrity. The sparse attention weight vector is obtained through norm sparse regularization constraint, which can accurately measure the importance of each channel, improve the image key information capture ability of the encoder, and enhance the performance, feature learning accuracy and effectiveness of the encoder.

[0073] S4, function optimization is performed on the pre-constructed contrast loss function according to the sparse attention feature matrix, to obtain a sparse regularization loss function.

[0074] In the embodiment of the present application, the sparse attention feature matrix is optimized into a sparse regularization loss function through contrast loss analysis, gradient back propagation and sparse regularization adjustment, which includes three key steps of loss calculation, gradient update and weight optimization.

[0075] In the embodiment of the present application, the function optimization is performed on the pre-constructed contrast loss function according to the sparse attention feature matrix, to obtain a sparse regularization loss function, which includes: loss value analysis is performed on the pre-constructed contrast loss function according to the sparse attention feature matrix, to obtain a contrast loss term; a mathematical expression of the contrast loss term is analyzed, and the attention feature parameters in the loss value analysis process and the correlation between the attention feature parameters are determined according to the mathematical expression; partial derivative analysis is performed on each feature element in the sparse attention feature matrix according to the attention feature parameters and the correlation between the attention feature parameters, to obtain an initial gradient matrix; gradient threshold filtering is performed on the initial gradient matrix, to obtain a filtered gradient matrix; back propagation is performed on the filtered gradient matrix, to obtain a target attention feature matrix; a sparse regularization term of the target attention feature matrix is obtained, and weight adjustment is performed on the contrast loss function according to the sparse regularization term, to obtain a sparse regularization loss function.

[0076] In detail, the sparse regularization term is specifically as shown in the following formula:

[0077] wherein, the sparse regularization term is represented by, the target attention feature matrix is represented by, the channel attention weight matrix is represented by, the attention channel is represented by, representing a degree of sparsity of the attention channel.

[0078] In detail, before the encoder training starts, a contrastive loss function is pre-constructed, which is usually used to measure the similarity and difference between samples, and the core idea is to make similar samples closer in the feature space and dissimilar samples farther apart; for example, in the image retrieval task, similar image pairs should have smaller loss values, while dissimilar image pairs should have larger loss values.

[0079] The sparse attention feature matrix is a special matrix obtained by the model during data processing, which weights the important features in the data through the attention mechanism, and through sparse processing, only part of the important features get higher weights, and the weights of the remaining features are close to zero, improving the efficiency and generalization ability of the encoder.

[0080] Specifically, the sparse attention feature matrix is input into the pre-constructed contrastive loss function to calculate the corresponding loss value, that is, to analyze the difference between the feature representation represented by the current sparse attention feature matrix and the feature distribution expected by the contrastive loss function. By calculating and analyzing the loss values of a large number of sparse attention feature matrices of samples, a comprehensive contrastive loss term can be obtained, which reflects the gap between the overall performance of the encoder under the current sparse attention feature representation and the target.

[0081] Further, when analyzing the mathematical expression of the contrastive loss term, by decomposing the feature similarity measurement term (such as cosine distance, Euclidean distance) and the positive and negative sample pair weight coefficient in the formula, the attention feature parameters (including channel weight, spatial attention coefficient and feature similarity threshold) are determined. The channel weight and the spatial coefficient jointly determine the feature response strength, and the similarity threshold constrains the loss contribution proportion of the positive and negative sample pairs.

[0082] Wherein, when doing partial derivative analysis on each element of the sparse attention feature matrix, combined with the derivative formula of the contrastive loss term with respect to the feature element, the gradient contribution of each element to the total loss is calculated to generate an initial gradient matrix; the gradient threshold filtering adopts the truncation method, sets the upper and lower limits of the gradient amplitude, and eliminates abnormal gradient values (such as outliers exceeding 3 times the standard deviation), to obtain a filtered gradient matrix to avoid gradient explosion.

[0083] In detail, the gradient of each layer parameter is calculated layer by layer along the network structure in the reverse direction according to the filtering gradient matrix, in the calculation process, the complex gradient calculation is decomposed into multiple simple gradient calculation steps by using the chain rule, the gradient of each element in the sparse attention feature matrix is calculated by using the gradient back propagation algorithm, and the gradient represents the change rate of the contrast loss term with respect to each element in the sparse attention feature matrix, that is, the influence degree of each element on the contrast loss term.

[0084] Specifically, the sparse attention feature matrix is updated by using an optimization algorithm (such as a stochastic gradient descent) according to the calculated gradient, the optimization algorithm adjusts the element value in the sparse attention feature matrix according to the direction and size of the gradient, so that the contrast loss term gradually decreases, and after multiple iterations, the obtained target attention feature matrix is the result of optimizing the original sparse attention feature matrix under the guidance of the contrast loss function, and can better meet the sample similarity and difference requirements defined by the contrast loss function.

[0085] The sparse regularization term is introduced to further strengthen the sparsity of the target attention feature matrix, the contrast loss function mainly focuses on the similarity and difference between samples, and the sparse regularization term focuses on controlling the sparsity of the target attention feature matrix, and the contrast loss function and the sparse regularization term after weight adjustment are combined to obtain a sparse regularization loss function.

[0086] Specifically, this new loss function contains both the constraints of the original contrast loss function on the sample relationship and the requirements of the sparse regularization term on the sparsity of the feature matrix, and in the subsequent encoder optimization process, the encoder will optimize the similarity and difference between samples and the sparsity of the target attention feature matrix at the same time, so as to obtain the encoder parameters and feature representation that meet the actual requirements.

[0087] In the embodiment of the application, the contrast loss term is obtained by analyzing the loss value of the contrast loss function by using the sparse attention feature matrix, which can accurately measure the difference degree of the feature matrix in the contrast learning and provide a key basis for subsequent optimization, and the target attention feature matrix is obtained by using the contrast loss term to perform gradient back propagation, which can adjust the feature matrix in a direction that is more conducive to distinguishing different samples, enhance the distinguishability of the feature, further constrain the sparsity of the feature, reduce redundant information, and improve the generalization ability of the encoder.

[0088] S5, the sparse attention feature matrix and the multi-channel feature map are weighted and fused to obtain a target image feature, and a preset image encoder is parameter optimized by using the target image feature and the sparse regularization loss function to obtain a target image encoder.

[0089] In the embodiment of the present application, the sparse attention feature matrix makes the elements in the matrix present a sparse distribution, that is, the values of most elements are close to zero, and only a few key position elements have larger values, and the positions and sizes of these larger values reflect the importance degree of different regions or feature channels in the image; and the multi-channel feature map contains rich information of the image in different feature dimensions, and each channel describes the features of the image from a specific angle, for example, some channels may focus on edge information, and some channels may focus on color information.

[0090] Among them, according to the importance information provided by the sparse attention feature matrix, each element in the multi-channel feature map is adjusted in a targeted manner, and specifically, for each element in the multi-channel feature map, the value in the corresponding position of the sparse attention feature matrix is found, which will be used as a weight to weight the element in the multi-channel feature map.

[0091] Through this weighting operation, the feature information originally having different importance in the multi-channel feature map is redistributed and strengthened, the feature information of the important region is amplified, and the feature information of the relatively unimportant region is weakened. After integrating the multi-channel feature map after the weighting processing, the target image feature is obtained.

[0092] In the embodiment of the present application, the parameter optimization of the preset image encoder by using the target image feature and the sparse regularization loss function obtains a target image encoder, which comprises: According to the target image feature, an initial loss value of the sparse regularization loss function is calculated; According to the initial loss value, a parameter gradient of the preset image encoder is determined by using a back propagation algorithm; The parameter gradient is gradient-smoothed to obtain a smooth parameter gradient; According to the smooth parameter gradient, the image encoder is parameter-adjusted to obtain a preliminary optimization encoder; The optimization image feature of the target coding image is extracted by using the preliminary optimization encoder, and an optimization loss value of the sparse regularization loss function is calculated according to the optimization image feature; According to the optimization loss value and a preset loss threshold, the preliminary optimization encoder is iteratively optimized to obtain a target image encoder.

[0093] In detail, the target image features are representative information extracted from the target image by the image encoder. The target image features are input into the sparse regularization loss function. The loss function analyzes the feature matrix according to preset rules, compares the difference between the feature matrix and the expected sparse feature distribution, and comprehensively considers multiple aspects such as feature accuracy and sparsity, and finally calculates an initial loss value.

[0094] Among them, the backpropagation algorithm is used to calculate the gradient of the loss function with respect to the encoder parameters. Taking the calculated initial loss value as the starting point, the backpropagation algorithm starts from the sparse regularization loss function and passes through each layer of the image encoder in turn. At each layer, the algorithm will calculate the contribution of each parameter to the initial loss value based on the relationship between the parameters of the layer and the loss function, that is, the parameter gradient, which represents the degree of influence of each parameter on the loss value.

[0095] Furthermore, when smoothing parameter gradients, sliding average or Gaussian filtering is often used. Sliding average smoothes the current gradient by assigning attenuation weights to historical gradients; Gaussian filtering uses a Gaussian kernel to perform weighted smoothing on the gradient spatial distribution, reducing gradient fluctuations, avoiding parameter update oscillations, and enhancing model convergence stability to obtain a smooth parameter gradient.

[0096] Furthermore, the parameters of the image encoder are adjusted based on the smoothed parameter gradient using an optimization algorithm (such as stochastic gradient descent). The optimization algorithm determines the adjustment direction and amplitude of each parameter based on the direction and magnitude of the parameter gradient. If the gradient of a parameter is positive, it means that increasing the value of the parameter will increase the loss value, so the value of the parameter needs to be reduced; conversely, if the gradient is negative, the value of the parameter needs to be increased.

[0097] Specifically, after parameter adjustment, the obtained image encoder is the result of preliminary optimization of the original image encoder under the guidance of the initial loss value. Compared with the original image encoder, the preliminary optimized encoder can better meet the requirements of the sparse regularization loss function when extracting target image features, making the extracted features more accurate and sparse. However, the encoder at this time may not have reached the optimal state and still needs further optimization.

[0098] In detail, a preliminary optimized encoder is used to extract features of the target encoded image. The optimized image features pay more attention to the sparsity of the features while retaining the key information of the image, reducing the interference of redundant information. The extracted optimized image features are input into the sparse regularization loss function, and the loss value is calculated again; if the optimized loss value is smaller than the initial loss value, it means that the performance of the preliminary optimized encoder has improved. On the contrary, if the optimized loss value is still large, it means that the encoder parameters need to be further optimized.

[0099] The iterative optimization is a method for gradually improving the performance of the encoder by repeatedly optimizing the process multiple times, in each iteration, the encoder parameters are adjusted according to the current loss value, so that the loss value gradually decreases until the preset stopping condition is met, when the optimized loss value is less than the threshold, it is considered that the performance of the encoder has reached the requirement, the optimization can be stopped, and the obtained encoder is the target image encoder.

[0100] In the embodiment of the present application, the sparse regularization loss function can constrain the feature sparsity and avoid overfitting, by continuously adjusting the parameters, the encoder can more accurately extract the key features of the image, improve the encoding effect, and further enhance the accuracy of the subsequent image processing task, and improve the feature learning effectiveness and accuracy of the image encoder.

[0101] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0102] As shown in Figure 4 It is a functional module diagram of an image encoder optimization device provided by an embodiment of the present application.

[0103] In the embodiment of the present disclosure, an image encoder optimization device is provided, which corresponds one-to-one to the image encoder optimization method of the above embodiment. As shown in Figure 4 The image encoder optimization device 100 can be installed in an electronic device, according to the realized function, the image encoder optimization device 100 includes a channel feature map generation module 101, an attention feature map convolution module 102, a feature regularization constraint module 103, a function loss optimization module 104 and a target encoder optimization module 105. The detailed description of each functional module is as follows: The channel feature map generation module 101 is used for obtaining a target encoding image, performing multi-scale image block embedding on the target encoding image to obtain an image block sequence, and performing linear projection transformation on the image block sequence to obtain a multi-channel feature map; The attention feature map convolution module 102 is used for performing convolution upsampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map; The feature regularization constraint module 103 is used for performing sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix; The function loss optimization module 104 is used for performing function optimization on a pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a sparse regularization loss function; The target encoder optimization module 105 is configured to perform adaptive feature reorganization on the sparse attention feature matrix and the multi-channel feature map to obtain target image features, perform parameter optimization on a preset image encoder by using the target image features and a sparse regularization loss function, and obtain a target image encoder.

[0104] In an embodiment, the channel feature map generation module 101, when performing multi-scale image block embedding on the target encoding image to obtain an image block sequence, is configured to: perform image enhancement processing on the target encoding image to obtain an enhanced encoding image; perform adaptive block processing on the enhanced encoding image according to a preset pyramid block window to obtain image blocks that do not overlap with each other; perform local feature extraction on each of the image blocks to obtain an initial feature vector of the image block; map the initial feature vector to a preset image high-dimensional space to generate an embedding vector; add position encoding information to the embedding vector, and perform spatial dimension reorganization on the embedding vector after adding the position encoding information according to the original spatial order of the target encoding image to obtain an image block sequence.

[0105] In an embodiment, the channel feature map generation module 101, when performing linear projection transformation on the image block sequence to obtain a multi-channel feature map, is configured to: perform pixel dimension flattening processing on each image block in the image block sequence to obtain a target image block sequence; calculate the pixel variance and edge density of each target image block in the target image block sequence, and obtain the vector dimension and feature channel number of the target image block; construct a multi-channel linear projection matrix according to the pixel variance, the edge density, the vector dimension, and the feature channel number; perform matrix multiplication on each of the target image blocks according to the multi-channel linear projection matrix to obtain a projection result; add a bias term to the projection result, and perform nonlinear transformation on the projection result after adding the bias term to obtain a multi-channel feature map.

[0106] In an embodiment, the attention feature map convolution module 102, when performing convolution upsampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map, is configured to: perform multi-layer dilated convolution processing on the multi-channel feature map to obtain a dilated convolution feature map, and perform pixel standardization processing on the dilated convolution feature map to obtain a standardized feature map; Initialize the up-sampling model, perform deep feature extraction on the standardized feature map according to a feature extraction sub-module of the up-sampling model, and obtain a deep feature map; Obtain the image size of the standardized feature map, calculate the up-sampling multiple according to a preset up-sampling target size and the image size; Perform up-sampling processing on the deep feature map according to the up-sampling multiple, and obtain an up-sampled feature map; Pixel-by-pixel addition of the up-sampled feature map and the multi-channel feature map generates a fusion feature map; Perform channel attention analysis on the fusion feature map to obtain a channel attention weight matrix, and perform weighted multiplication on the fusion feature map according to the channel attention weight matrix to obtain a pixel-level attention feature map.

[0107] In an embodiment, the feature regularization constraint module 103, when performing sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix, is configured to: Perform spatial dimension analysis on the pixel-level attention feature map to obtain a height pixel number, a width pixel number and a channel number; Construct a multi-scale pooling window according to the height pixel number, the width pixel number and the channel number; Slide average pooling of the pixel-level attention feature map using the multi-scale pooling window to obtain a pooled feature map; Calculate the mean value of each attention channel in the pooled feature map, and determine the feature dense area and the feature sparse area in the pooled feature map according to the mean value; Generate an attention sparse term and an attention smoothing term based on the feature dense area and the feature sparse area; According to the attention sparse term and the attention smoothing term, each attention channel is subjected to norm sparse regularization constraint to obtain a sparse attention weight vector; According to the sparse attention weight vector, the pixel-level attention feature map is subjected to pixel-by-pixel weighted normalization to obtain a sparse attention feature matrix.

[0108] In an embodiment, the function loss optimization module 104, when performing function optimization on a pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a sparse regularization loss function, is configured to: According to the sparse attention feature matrix, loss value analysis is performed on a pre-constructed contrast loss function to obtain a contrast loss term; Analyze the mathematical expression of the contrast loss term, and determine the attention feature parameters in the loss value analysis process and the correlation between the attention feature parameters according to the mathematical expression; According to the attention feature parameters and the correlation between the attention feature parameters, partial derivative analysis is performed on each feature element in the sparse attention feature matrix to obtain an initial gradient matrix; The initial gradient matrix is subjected to gradient threshold filtering to obtain a filtered gradient matrix; The filtered gradient matrix is subjected to back propagation to obtain a target attention feature matrix; A sparse regularization term of the target attention feature matrix is obtained, and the contrast loss function is subjected to weight adjustment according to the sparse regularization term to obtain a sparse regularization loss function.

[0109] In an embodiment, the target encoder optimization module 105 performs parameter optimization on a preset image encoder by using the target image feature and the sparse regularization loss function to obtain a target image encoder, including: An initial loss value of the sparse regularization loss function is calculated according to the target image feature; A parameter gradient of the preset image encoder is determined according to the initial loss value by using a back propagation algorithm; The parameter gradient is subjected to gradient smoothing processing to obtain a smoothed parameter gradient; The image encoder is subjected to parameter adjustment according to the smoothed parameter gradient to obtain a preliminary optimized encoder; Optimized image features of the target encoded image are extracted by using the preliminary optimized encoder, and an optimized loss value of the sparse regularization loss function is calculated according to the optimized image features; The preliminary optimized encoder is subjected to iterative optimization according to the optimized loss value and a preset loss threshold to obtain a target image encoder.

[0110] In the present application, the specific limitations of the image encoder optimization device can be referred to the limitations of the image encoder optimization method in the foregoing, which will not be repeated here. Each module in the image encoder optimization device described above can be realized by software, hardware and combinations thereof in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0111] In an embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes non-volatile and / or volatile storage medium, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to realize the function or step of the server side of the image encoder optimization method.

[0112] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram can be as shown in the figure. Figure 6 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes non-volatile storage medium, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the function or step of the client side of the image encoder optimization method.

[0113] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the following steps: Obtaining a target encoded image, performing multi-scale image block embedding on the target encoded image to obtain an image block sequence, and performing linear projection transformation on the image block sequence to obtain a multi-channel feature map; Performing convolution upsampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map; Performing sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix; According to the sparse attention feature matrix, a pre-constructed contrast loss function is functionally optimized to obtain a sparse regularization loss function; Performing adaptive feature reorganization on the sparse attention feature matrix and the multi-channel feature map to obtain a target image feature, and performing parameter optimization on a preset image encoder using the target image feature and the sparse regularization loss function to obtain a target image encoder.

[0114] In several embodiments provided by the present application, it should be understood that the disclosed devices and apparatuses can be implemented in other manners. For example, the above-described system embodiments are merely illustrative. For example, the division of the modules is merely logical function division. In actual implementation, other division manners can be adopted.

[0115] In addition, each function module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software function modules.

[0116] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be regarded as limiting the claims involved.

[0117] In some embodiments of the present embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. The computer program is characterized in that when executed by a processor, the computer program implements the steps of the method described in the above embodiments.

[0118] The readable storage medium described in the present application stores a computer program, and the computer program, when executed by a processor of an electronic device, can realize: obtaining a target coding image, performing multi-scale image block embedding on the target coding image to obtain an image block sequence, and performing linear projection transformation on the image block sequence to obtain a multi-channel feature map; performing convolution upsampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map; performing sparse regularization constraint on the pixel-level attention feature map to obtain a sparse attention feature matrix; performing function optimization on a pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a sparse regularization loss function; performing adaptive feature reorganization on the sparse attention feature matrix and the multi-channel feature map to obtain a target image feature, and performing parameter optimization on a preset image encoder using the target image feature and the sparse regularization loss function to obtain a target image encoder.

[0119] It should be noted that the above functions or steps of the computer readable storage medium or the computer device can correspond to the above-mentioned method embodiments, and the server side and the client side are described in detail. To avoid repetition, they will not be described one by one here.

[0120] The computer readable storage medium can also store at least one computer executable program / instruction, such as computer readable instructions. The computer readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory, etc. The computer readable storage medium may, for example, include read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer readable storage medium can be connected to a computing device such as a computer, and then when the computing device runs the computer readable instructions stored on the computer readable storage medium, the various methods described above can be performed.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of the method. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units or modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-mentioned functions.

[0123] In the embodiments provided by the present disclosure, it should be understood that the disclosed apparatus and method can also be implemented in other manners. The embodiments described above are merely exemplary for describing the present disclosure. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowcharts and block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in some alternative implementations, the functions noted in the blocks can occur in a different order from that noted in the accompanying drawings. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a special-purpose hardware-based system that performs the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0124] The above-described embodiments are merely used to illustrate the technical solutions of the present disclosure, rather than limit the present disclosure; even though the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that modifications can still be made to the technical solutions recorded in the foregoing embodiments, or equivalent replacements can be made to some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

[0125] It should be noted that, if a software tool or component that is not from the company appears in the embodiments of the present disclosure, it is merely used for example introduction, and does not represent actual use.

Claims

1. A method for optimizing an image encoder, characterized in that: The method comprises: Acquire a target coded image, perform multi-scale image block embedding on the target coded image to obtain an image block sequence, and perform linear projection transformation on the image block sequence to obtain a multi-channel feature map; Performing convolution upsampling on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map; Performing sparse regularization constraints on the pixel-level attention feature map to obtain a sparse attention feature matrix; Optimizing the pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a sparse regularized loss function; The sparse attention feature matrix and the multi-channel feature map are adaptively reorganized to obtain target image features, and the target image features and the sparse regularization loss function are used to optimize the parameters of a preset image encoder to obtain a target image encoder.

2. The image encoder optimization method according to claim 1, wherein: The performing multi-scale image block embedding on the target coded image to obtain an image block sequence includes: performing image enhancement processing on the target coded image to obtain an enhanced coded image; Adaptively partitioning the enhanced coded image into blocks according to a preset pyramidal partitioning window to obtain non-overlapping image blocks; Performing local feature extraction on each of the image blocks to obtain an initial feature vector of the image block; Mapping the initial feature vector to a preset high-dimensional image space to generate an embedding vector; Position coding information is added to the embedding vector, and spatial dimension reorganization is performed on the embedding vector after adding the position coding information according to the original spatial order of the target coded image to obtain an image block sequence.

3. The image encoder optimization method according to claim 1, wherein: The performing linear projection transformation on the image block sequence to obtain a multi-channel feature map includes: Performing pixel dimension flattening processing on each image block in the image block sequence to obtain a target image block sequence; Calculating the pixel variance and edge density of each target image block in the target image block sequence, and obtaining the vector dimension and number of feature channels of the target image block; Constructing a multi-channel linear projection matrix according to the pixel variance, the edge density, the vector dimension, and the number of feature channels; Performing a matrix multiplication operation on each target image block according to the multi-channel linear projection matrix to obtain a projection result; A bias term is added to the projection result, and a nonlinear transformation is performed on the projection result after adding the bias term to obtain a multi-channel feature map.

4. The image encoder optimization method according to claim 1, wherein: The convolution upsampling process is performed on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map, including: Performing multi-layer dilated convolution processing on the multi-channel feature map to obtain a dilated convolution feature map, and performing pixel normalization processing on the dilated convolution feature map to obtain a normalized feature map; Initializing an upsampling model, and performing deep feature extraction on the standardized feature map according to a feature extraction submodule of the upsampling model to obtain a deep feature map; Obtaining the image size of the standardized feature map, and calculating an upsampling factor according to a preset upsampling target size and the image size; Performing upsampling processing on the deep feature map according to the upsampling multiple to obtain an upsampled feature map; Adding the upsampled feature map and the multi-channel feature map pixel by pixel to generate a fused feature map; Channel attention analysis is performed on the fused feature map to obtain a channel attention weight matrix, and weighted multiplication is performed on the fused feature map according to the channel attention weight matrix to obtain a pixel-level attention feature map.

5. The image encoder optimization method according to claim 1, wherein: The sparse regularization constraint is performed on the pixel-level attention feature map to obtain a sparse attention feature matrix, including: Performing spatial dimension analysis on the pixel-level attention feature map to obtain the number of height pixels, the number of width pixels, and the number of channels; Constructing a multi-scale pooling window according to the number of height pixels, the number of width pixels, and the number of channels; Performing sliding average pooling on the pixel-level attention feature map using the multi-scale pooling window to obtain a pooled feature map; Calculating the mean of each attention channel in the pooled feature map, and determining the feature-dense area and the feature-sparse area in the pooled feature map according to the mean; Generate an attention sparse term and an attention smooth term based on the feature-dense area and the feature-sparse area; Performing a norm sparse regularization constraint on each of the attention channels according to the attention sparse term and the attention smooth term to obtain a sparse attention weight vector; The pixel-level attention feature map is weighted and normalized pixel by pixel according to the sparse attention weight vector to obtain a sparse attention feature matrix.

6. The image encoder optimization method according to claim 1, wherein: The pre-constructed contrast loss function is optimized according to the sparse attention feature matrix to obtain a sparse regularized loss function, including: Performing loss value analysis on a pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a contrast loss term; parsing a mathematical expression of the contrast loss term, and determining, according to the mathematical expression, attention feature parameters in the loss value analysis process and correlation relationships between the attention feature parameters; Performing partial derivative analysis on each feature element in the sparse attention feature matrix according to the attention feature parameters and the correlation between the attention feature parameters to obtain an initial gradient matrix; Performing gradient threshold filtering on the initial gradient matrix to obtain a filtered gradient matrix; Backpropagating the filtered gradient matrix to obtain a target attention feature matrix; Obtain a sparse regularization term of the target attention feature matrix, and adjust the weight of the contrast loss function according to the sparse regularization term to obtain a sparse regularization loss function.

7. The image encoder optimization method according to claim 1, wherein: The method of optimizing parameters of a preset image encoder by using the target image features and the sparse regularization loss function to obtain a target image encoder includes: Calculating an initial loss value of the sparse regularization loss function according to the target image features; Determining a parameter gradient of a preset image encoder according to the initial loss value using a back propagation algorithm; Performing gradient smoothing on the parameter gradient to obtain a smoothed parameter gradient; Adjusting parameters of the image encoder according to the smoothing parameter gradient to obtain a preliminary optimized encoder; Extracting optimized image features of the target coded image using the preliminary optimized encoder, and calculating an optimized loss value of the sparse regularization loss function based on the optimized image features; The preliminary optimized encoder is iteratively optimized according to the optimized loss value and a preset loss threshold to obtain a target image encoder.

8. An image encoder optimization device, characterized in that: The device comprises: a channel feature map generation module, configured to obtain a target coded image, perform multi-scale image block embedding on the target coded image to obtain an image block sequence, and perform linear projection transformation on the image block sequence to obtain a multi-channel feature map; An attention feature map convolution module is used to perform convolution upsampling processing on the multi-channel feature map to obtain a pixel-level attention feature map corresponding to the multi-channel feature map; A feature regularization constraint module, configured to perform sparse regularization constraints on the pixel-level attention feature map to obtain a sparse attention feature matrix; A function loss optimization module, configured to perform function optimization on a pre-constructed contrast loss function according to the sparse attention feature matrix to obtain a sparse regularized loss function; The target encoder optimization module is used to adaptively reorganize the sparse attention feature matrix and the multi-channel feature map to obtain target image features, and use the target image features and the sparse regularization loss function to optimize the parameters of the preset image encoder to obtain the target image encoder.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the image encoder optimization method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image encoder optimization method according to any one of claims 1 to 7 is implemented.

Images