A method for reducing multimodal feature quantities of large models based on multi-level coding

Abstract

The present invention discloses a method for reducing the multimodal feature quantity of a large model based on multi-level coding, which relates to the fields of computer vision and deep learning, including collecting and preprocessing high-resolution images, extracting features and performing preliminary dimensionality reduction on the preprocessed high-resolution images, obtaining preliminary compressed feature maps, constructing multi-scale feature pyramids based on the preliminary compressed feature maps, generating features at different levels, applying hierarchical attention mechanisms to features at different levels, obtaining feature maps with local and global attention weights, using hierarchical fusion strategies on feature maps with local and global attention weights, generating multi-level attention optimization features, using image reconstruction processing on the multi-level attention optimization features, and obtaining reconstructed feature representations and images. The present invention maintains attention to the global structure when capturing images, solves the problems of insufficient fusion of multi-scale features and unbalanced application of local and global attention mechanisms, and achieves more efficient feature representation.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a method for reducing multimodal feature quantities of a large model based on multi-level coding. Background Art

[0002] In the field of computer vision and deep learning, processing high-resolution images has always been a hot topic. With the development of convolutional neural networks, especially the rise of large-scale pre-trained models, the ability to process multimodal data has been significantly improved.

[0003] Although the existing technology has made some progress, there are still two major problems: first, the fusion of multi-scale features is not sufficient; second, the application of local and global attention mechanisms is not balanced. Specifically, most methods only extract features at a single scale and cannot fully utilize the complementarity between features at different levels. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for reducing the multimodal feature quantity of a large model based on multi-level encoding to solve the problems of insufficient multi-scale feature fusion and unbalanced application of local and global attention mechanisms in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for reducing multimodal feature quantities of a large model based on multi-level coding, which comprises:

[0008] Acquire high-resolution images and perform preprocessing;

[0009] Perform feature extraction and preliminary dimension reduction on the preprocessed high-resolution image to obtain a preliminary compressed feature map;

[0010] Based on the preliminary compressed feature map, a multi-scale feature pyramid is constructed to generate features at different levels;

[0011] Apply hierarchical attention mechanism to different levels of features to obtain feature maps with local and global attention weights;

[0012] The feature maps with local and global attention weights are fused in layers to generate multi-level attention optimization features.

[0013] Use the multi-level attention optimization features to process the image reconstruction to obtain the reconstructed feature representation and image;

[0014] Collect high-resolution images and perform preprocessing, which includes the following steps:

[0015] Use scanners to capture high-resolution images;

[0016] Perform image denoising and normalization on high-resolution images;

[0017] The high-resolution image after denoising and normalization is divided into multiple regions, and an adaptive resolution selection algorithm is applied to obtain multi-resolution image blocks;

[0018] The multi-resolution image blocks are encoded and stored, and metadata records are created for each resolution image block to generate a pre-processed high-resolution image.

[0019] As a preferred solution of the method for reducing the multimodal feature quantity of a large model based on multi-level coding described in the present invention, wherein: the pre-processed high-resolution image is subjected to feature extraction and preliminary dimension reduction to obtain a preliminary compressed feature map, and the specific steps are as follows:

[0020] The preprocessed high-resolution image is resized to a uniform size, and a deep convolutional neural network is applied to extract features to obtain a multi-layer feature map;

[0021] Apply local response normalization to the multi-layer feature maps and perform preliminary dimensionality reduction using PCA principal component analysis;

[0022] The feature map after the preliminary dimensionality reduction by PCA principal component analysis is further compressed using a lightweight encoder to obtain a preliminary compressed feature map.

[0023] As a preferred solution of the method for reducing the multimodal feature quantity of a large model based on multi-level coding described in the present invention, wherein: based on the preliminary compressed feature map, a multi-scale feature pyramid is constructed to generate features at different levels. The specific steps are as follows:

[0024] Unify the size of the initial compressed feature map, and create a base layer of a multi-scale feature pyramid based on the compressed feature map after unification;

[0025] Use transposed convolution to upsample the base layer feature map to construct a high-resolution feature map;

[0026] Apply maximum pooling to downsample the base layer feature map to construct a low-resolution feature map;

[0027] Apply 1x1 convolution to the high-resolution feature map and the low-resolution feature map to adjust the number of channels, add the feature maps after the adjustment of the number of channels element by element, and obtain the fused multi-scale feature map;

[0028] Upsample the fused multi-scale feature map and add it element by element with the corresponding position features of the higher-resolution feature map. Apply a 3x3 convolution layer to eliminate the aliasing effect and generate a multi-scale fused high-resolution feature map.

[0029] The multi-scale fused high-resolution feature map of the current stage is used as the output of the current stage and the input of the next stage, gradually building a complete multi-scale feature pyramid;

[0030] Through the multi-scale feature pyramid, feature maps of different resolutions can be extracted and fused at each level to obtain different levels of features from low-level details to high-level semantics.

[0031] As a preferred solution of the method for reducing the multimodal feature quantity of a large model based on multi-level encoding described in the present invention, wherein: a hierarchical attention mechanism is applied to features at different levels to obtain a feature map with local and global attention weights, the specific steps are as follows:

[0032] Based on features at different levels, a local attention mechanism is designed, and a convolutional neural network is used to extract the importance weights of local areas and obtain feature maps with local attention weights.

[0033] Based on features at different levels, a global attention mechanism is designed, and the self-attention mechanism is used to capture global dependencies and obtain the feature map of global attention weights.

[0034] The feature map of local attention weights and the feature map of global attention weights are fused using a weighted summation method to generate a feature map with local and global attention weights.

[0035] As a preferred solution of the method for reducing the multimodal feature quantity of a large model based on multi-level coding described in the present invention, wherein: a hierarchical fusion strategy is adopted to generate multi-level attention optimization features using feature maps with local and global attention weights. The specific steps are as follows:

[0036] The feature maps of local and global attention weights are integrated through a hierarchical fusion strategy, a function to enhance the expressiveness of local features is defined, and the importance of local features is adjusted through convolution operations combined with nonlinear transformations.

[0037] Define a function to improve the quality of global features, and adjust the importance of global features through the SENet squeeze-excitation network combined with GAP global average pooling;

[0038] Define a function for calculating the cross relationship between the local feature map and the global feature map, and calculate the cross relationship between the local feature map and the global feature map through a multi-head attention mechanism and relative position encoding;

[0039] Define a dynamic adjustment function for the overall feature combination, and use weighted summation to adjust the weights of the local feature map and the global feature map;

[0040] By introducing an exponential decay coefficient, all the features obtained in the above steps are integrated into a multi-level attention optimization feature map, which is expressed as:

[0041] ;

[0042] in, represents a multi-level attention-optimized feature map, A feature map representing local attention weights, A feature map representing the global attention weights, is the amplitude coefficient of the sine function, is the amplitude coefficient of the cosine function, Is used to adjust The frequency of the sine function, Is used to adjust The frequency of the cosine function, yes The offset of yes The offset of is the weight coefficient of the influence of global features on local features, is the weight coefficient of the influence of local features on global features, is the intensity coefficient of the exponential decay term, Speed ​​parameter of the exponential decay.

[0043] As a preferred solution of the method for reducing the multimodal feature quantity of a large model based on multi-level coding described in the present invention, wherein: the multi-level attention optimization features are processed using image reconstruction to obtain the reconstructed feature representation and image, the specific steps are as follows:

[0044] The multi-level attention optimization feature map is normalized and the spatial size is increased using a transposed convolutional layer to obtain the processed feature map.

[0045] Extract the encoder and decoder from the pre-trained deep convolutional neural network, introduce skip connections, concatenate the features of the encoder stage level and the decoder stage level of the processed feature map, and obtain a feature map with skip connections;

[0046] Apply adaptive instance normalization to feature maps with skip connections to adjust the mean and variance of feature maps;

[0047] The feature map processed by adaptive instance normalization is further upsampled through multiple transposed convolutional layers to generate a feature map with a resolution close to the original input image. The tanh activation function is used to obtain an image of the same spatial size as the original input. The upsampled and activated feature maps are integrated to obtain the reconstructed feature representation and image.

[0048] In a second aspect, the present invention provides a system for reducing multimodal feature quantities of a large model based on multi-level coding, comprising:

[0049] Preprocessing module, collects original high-resolution images and performs preprocessing;

[0050] The dimension reduction module performs feature extraction and preliminary dimension reduction on the preprocessed original high-resolution image to obtain a preliminary compressed feature map;

[0051] Construct a pyramid module, based on the preliminary compressed feature map, to construct a multi-scale feature pyramid to obtain features at different levels;

[0052] Attention enhancement module, which applies hierarchical attention mechanism to features at different levels to obtain feature maps with local and global attention weights;

[0053] The feature fusion module uses a hierarchical fusion strategy to fuse feature maps with local and global attention weights to obtain multi-level attention optimization features;

[0054] The image reconstruction module uses the multi-level attention optimization features to reconstruct the image and obtain the reconstructed feature representation and image.

[0055] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the method for reducing the multimodal feature quantity of a large model based on multi-level coding as described in the first aspect of the present invention.

[0056] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the method for reducing the multimodal feature quantity of a large model based on multi-level coding as described in the first aspect of the present invention.

[0057] The beneficial effects of the present invention are as follows: by constructing a multi-scale feature pyramid and applying a hierarchical attention mechanism, effective fusion of features at different levels from low-level details to high-level semantics is achieved. This method can not only significantly reduce the number of features and reduce computational complexity, but also improve the model's ability to understand complex scenes. By introducing the joint optimization of local and global attention mechanisms, the present invention can capture key areas in the image while maintaining attention to the global structure, thereby solving the problems of insufficient multi-scale feature fusion and unbalanced application of local and global attention mechanisms in the prior art, and achieving more efficient feature representation and better task performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0059] Figure 1 This is a flow chart of the method for reducing the multimodal feature quantity of a large model based on multi-level coding in Example 1;

[0060] Figure 2 This is a system / schematic diagram of the hierarchical attention mechanism in Example 1. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0064] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a method for reducing the multimodal feature quantity of a large model based on multi-level coding, comprising the following steps:

[0065] S1. Collect high-resolution images. The specific steps are as follows:

[0066] Configure a dedicated scanner for OCR optical character recognition and set scanning parameters, including DPI resolution, grayscale, RGB, and CMYK color modes;

[0067] Place the original document or object on the scanner, start the scanning process, and obtain a high-resolution image.

[0068] S2. Preprocess the high-resolution image. The specific steps are as follows:

[0069] Select appropriate denoising algorithms according to image characteristics, including Gaussian filtering, bilateral filtering, NLM non-local mean filtering, and FFT fast Fourier transform denoising. Apply the selected denoising algorithm to the scanned image to remove random noise in the scanned image, improve image quality while maintaining key details and edge clarity of the image;

[0070] Standardize the high-resolution images after denoising. Adjust the brightness and contrast of the images through histogram equalization or CLAHE adaptive histogram equalization to make the image visually better and help enhance the visibility of image features. Then perform size normalization to adjust all images to a uniform size to ensure the consistency of subsequent processing. Finally, perform color space conversion to convert the images from one color space to another, which simplifies the processing steps, makes the features more obvious, and reduces the computational complexity.

[0071] The denoised and standardized high-resolution image is divided into several non-overlapping sub-regions according to the complexity of the image content using regular grid division and super-pixel segmentation methods to better capture the structural information in the image and provide the possibility of independent processing for each sub-region, thereby optimizing processing efficiency and result accuracy;

[0072] Use edge detection and texture analysis to evaluate the detail and information content of each area. Select the appropriate resolution for each area based on the evaluation results to ensure that important areas retain sufficient details while reducing the amount of data in unimportant areas. Resample each sub-area according to the selected resolution to generate multi-resolution image blocks, achieve efficient storage and fast transmission, and improve the overall visual quality of the image.

[0073] Select a JPEG compression algorithm based on the quality and storage requirements of the multi-resolution image block, encode each multi-resolution image block, and reduce storage space occupation;

[0074] Define the metadata structure, determine the metadata fields that need to be recorded, and create corresponding metadata records for each image block to ensure the association between the image block and its related information, which facilitates image retrieval and management, and also supports the long-term preservation and future use of image data;

[0075] All the encoded image blocks and their metadata records are integrated together to form the final pre-processed high-resolution image, ensuring the integrity and consistency of the image data for further analysis.

[0076] S3, extract features and perform preliminary dimension reduction on the preprocessed high-resolution image to obtain a preliminary compressed feature map. The specific steps are as follows:

[0077] The preprocessed high-resolution images are resized by bilinear interpolation to ensure that all images have the same width and height for subsequent processing;

[0078] Use a deep convolutional neural network to extract features from the resized high-resolution image. The specific steps are as follows:

[0079] The resized high-resolution image is passed as input to a deep convolutional neural network. The image data is passed through each layer in the network layer by layer. The convolutional layer uses multiple filters to slide on the input image to generate a feature map. At the same time, each filter captures the visual features of edges, textures, and shapes.

[0080] The ReLU nonlinear activation function is applied after each convolutional layer. It introduces nonlinear characteristics and can reflect complex mapping relationships. Pooling operations are performed after the convolutional layer is activated, which can reduce the spatial dimension of the feature map and prevent overfitting, thereby reducing the amount of calculation while maintaining important features.

[0081] According to the above steps, as the number of layers increases, the early layers mainly capture low-level features of color and edges, while the later layers gradually turn to capturing high-level semantic information of object parts and categories. Feature maps at different levels reflect different levels of abstraction of the image.

[0082] The ResNet architecture uses the SkipConnection jump connection method to allow the

[0083] Information is directly added, which promotes information flow and alleviates the gradient vanishing problem;

[0084] After a series of convolution, activation, pooling and other operations, multi-layer feature maps are generated, which contain rich spatial and semantic information and can be directly used for subsequent tasks;

[0085] LRN local response normalization and PCA principal component analysis are used to reduce the dimensionality of multi-layer feature maps. LRN local response normalization enhances the larger activation values ​​in the feature map while suppressing the smaller activation values ​​to achieve the purpose of dimensionality reduction and improve the generalization ability of the model. PCA principal component analysis finds the direction of the maximum variance of the data and projects it into a lower-dimensional space, thereby reducing the number of features while maintaining the information of the original data as much as possible.

[0086] Use a lightweight encoder to further compress the feature map after preliminary dimensionality reduction. The specific steps are as follows:

[0087] By downsampling, the spatial size of the feature map is gradually reduced by using convolution or maximum pooling operations with a step size of 2, thereby reducing the resolution;

[0088] Through BN batch normalization and activation function, nonlinear characteristics are introduced to help deep convolutional neural networks better learn and express complex patterns;

[0089] Through multi-scale feature fusion, combining feature maps at different levels, and skip connections, both low-level details and high-level semantic information can be effectively preserved;

[0090] Through global average pooling, the feature map of each channel is compressed into a single value to obtain a fixed-length feature vector, further reducing the feature dimension while maintaining important global information;

[0091] Through the bottleneck layer, one or more bottleneck layers are set at the deepest part of the encoder. These layers have fewer neurons or channels to force the compression of feature representation;

[0092] After the above processing, one or more groups of preliminarily compressed feature maps are generated. They greatly reduce the size and computational burden while retaining key features, making them suitable for subsequent task processing.

[0093] S4. Based on the preliminary compressed feature map, a multi-scale feature pyramid is constructed to generate features at different levels. The specific steps are as follows:

[0094] The feature map after preliminary compression is resized by bilinear interpolation, and the resized feature map is directly used as the base layer of the pyramid.

[0095] Use transposed convolution to upsample the base layer feature map, set the parameters of transposed convolution according to requirements, including kernel size, stride, padding, and number of output channels, to obtain a high-resolution feature map;

[0096] Apply maximum pooling to downsample the base layer feature map, set the maximum pooling parameters according to requirements, including pooling window size and stride, to obtain a low-resolution feature map;

[0097] Apply 1x1 convolution to the high-resolution feature map and the low-resolution feature map to adjust the number of channels, ensuring that they have the same number of channels, so as to change the depth of the feature map without changing its spatial dimension. Add the feature maps after adjusting the number of channels element by element to obtain the fused multi-scale feature map.

[0098] Upsample the fused multi-scale feature map, and add the upsampled multi-scale feature map and the features of the corresponding positions of the shallower high-resolution feature map element by element, that is, add each channel value of the two feature maps at the same spatial position, so as to effectively combine low-level detail information and high-level semantic information without increasing the computational cost;

[0099] Apply a 3x3 convolution layer to the multi-scale feature map after upsampling and element-wise addition to smooth the noise that may be introduced by upsampling and improve the quality of the final output, eliminate the checkerboard effect that may occur during the upsampling process, and obtain a multi-scale fused high-resolution feature map;

[0100] The multi-scale fused high-resolution feature map is used as the output of the current stage and as the input of the next stage, ensuring that the information of each layer can be fully utilized and that more and more abstract features can be captured as the number of pyramid layers increases;

[0101] According to the principle of coarse to fine, a complete multi-scale feature pyramid is gradually constructed, first processing the feature maps with lower resolution, and then gradually increasing the resolution until the required number of layers is reached or the requirements of a specific task are met, ensuring that information at all scales is taken into account;

[0102] The multi-scale feature pyramid is extracted using bottom-up and top-down paths to extract feature maps of different resolutions. The extracted feature maps of different resolutions are fused using the lateral connection method to obtain different levels of features from low-level details to high-level semantics.

[0103] S5. Apply the hierarchical attention mechanism to different levels of features to obtain feature maps with local and global attention weights. The specific steps are as follows:

[0104] Design a local attention mechanism, use convolutional neural networks to capture the spatial structure information of features at different levels, and use the Sigmoid function to obtain local weights, limit the weights to a specific range, and ensure that the importance weights of the generated local areas are non-negative and can be used directly as weighting coefficients;

[0105] Apply the importance weight of the local area to the original feature map, and obtain the feature map with local attention weight by element-by-element multiplication of the convolutional neural network;

[0106] Design a global attention mechanism to capture the long-distance dependencies between features at different levels, and introduce a self-attention mechanism to convert the feature map into three different vectors: query, key, and value;

[0107] Calculate the similarity between the query and the key to get the score matrix, and use the Softmax function to normalize the score matrix to form an attention distribution. According to this attention distribution, the values ​​are weighted and summed to get a feature map with global attention weights.

[0108] Combine the local attention feature map and the global attention feature map, mix the two feature maps together in a certain ratio, and create a new feature map. This new feature map takes both local and global characteristics into consideration. In the process of creating a new feature map, the ratio of the two feature maps is dynamically adjusted to realize the function of automatically deciding whether to pay more attention to local or global information according to the needs of different tasks.

[0109] S6. Use a hierarchical fusion strategy to generate multi-level attention optimization features using feature maps with local and global attention weights. The specific steps are as follows:

[0110] The feature maps of local and global attention weights are integrated through a hierarchical strategy, including:

[0111] Define a function to enhance the expressiveness of local features. Apply convolution operations to the local attention weights of the feature map to capture spatial structure information. This is done through one or more convolutional layers. Each convolutional layer can be followed by an activation function to introduce nonlinear transformations. The importance of local features is further adjusted through the Sigmoid function of the nonlinear transformation. The output of this step is the enhanced expressiveness of local features.

[0112] Define a function to improve the quality of global features. Use the SENet squeeze-excitation network combined with GAP global average pooling to adjust the importance of global features. Perform a global average pooling operation on the global attention weight of the feature map to obtain the average value on each channel. Use a fully connected layer and a ReLU activation function to build an "excitation" branch, which generates a weight for each channel. These weights are used to rescale the corresponding channels in the original feature map. The output of this step is the improved global feature quality.

[0113] Define a function to calculate the cross-relationship between the local feature map and the global feature map, and use a multi-head attention mechanism to extract the query, key, and value from the local attention weight and global attention weight of the feature map. Calculate the similarity between the query and the key to obtain a score matrix, and use the Softmax function to normalize the score matrix to form an attention distribution. According to this attention distribution, the values ​​are weighted and summed to obtain information with cross-relationships. The output of this step is a new feature that represents the cross-relationship between the local feature map and the global feature map.

[0114] Define a dynamic adjustment function for the overall feature combination, use a weighted summation method to adjust the weights of the feature map with local attention weights and the feature map with global attention weights, and dynamically adjust the proportion of the two feature maps. The output of this step is the overall feature after the dynamic adjustment function.

[0115] By introducing an exponential decay coefficient, all the features obtained in the above steps are integrated into a multi-level attention optimization feature map, which is expressed as:

[0116] ;

[0117] in, represents a multi-level attention-optimized feature map, A feature map representing local attention weights, A feature map representing the global attention weights, is the amplitude coefficient of the sine function, is the amplitude coefficient of the cosine function, Is used to adjust The frequency of the sine function, Is used to adjust The frequency of the cosine function, yes The offset of yes The offset of is the weight coefficient of the influence of global features on local features, is the weight coefficient of the influence of local features on global features, is the intensity coefficient of the exponential decay term, Speed ​​parameter of exponential decay;

[0118] The output of this process is a multi-level attention-optimized feature map, which comprehensively considers local and global features and achieves a dynamic balance between the two through mathematical formulas. This design not only improves the algorithm's ability to understand complex scenes, but also effectively reduces the number of multimodal features of large models, thereby reducing computational costs and improving performance.

[0119] S7, use image reconstruction to process the multi-level attention optimization features to obtain the reconstructed feature representation and image. The specific steps are as follows:

[0120] Batch normalization is used to standardize the multi-level attention optimization features, and a transposed convolution layer is used to increase the spatial size of the feature map. The transposed convolution layer can map the low-resolution features back to the high-resolution space through a filter. An activation function is added after the transposed convolution layer to help capture more complex patterns and obtain the processed feature map.

[0121] The encoder part of the deep convolutional neural network is used to capture high-level abstract features, and these features are converted into the same representation as the original input through the decoder part. In order to retain more detailed information, SkipConnections are introduced between the encoder and decoder to directly transfer low-level features to the corresponding level of the decoder layer. SkipConnections effectively solves the gradient vanishing problem in deep networks and promotes the preservation of fine-grained information. The features obtained in the encoder stage are spliced ​​together with the features in the decoder stage to form a feature map with skip connections;

[0122] Adaptive instance normalization is applied to feature maps with skip connections, which allows the mean and variance of feature maps to be dynamically adjusted to better match the target distribution. By independently calculating the mean and standard deviation for each channel and adjusting them to the target value, the feature representation between different samples is more consistent, thereby improving the robustness and generalization ability of the model.

[0123] The feature map after adaptive instance normalization is sent to multiple transposed convolutional layers for upsampling until a feature map with the same resolution as the original input image is generated. The tanh activation function is used to limit the output range to ensure that the color value of the generated image is in a reasonable range. The tanh activation function not only limits the output range, but also provides good nonlinear characteristics, which helps to improve visual quality.

[0124] The feature maps obtained through all the above steps are integrated to form the final reconstructed feature representation and image. This process ensures the continuity and consistency from multi-level attention optimized features to complete image reconstruction.

[0125] This embodiment also provides a system for reducing multimodal feature quantities of a large model based on multi-level coding, including: a preprocessing module, a dimensionality reduction module, a pyramid construction module, an attention enhancement module, a feature fusion module and an image reconstruction module;

[0126] Pre-processing module, collects high-resolution images and performs pre-processing;

[0127] The dimension reduction module performs feature extraction and preliminary dimension reduction on the preprocessed high-resolution image to obtain a preliminary compressed feature map;

[0128] Construct a pyramid module to build a multi-scale feature pyramid based on the preliminary compressed feature map and generate features at different levels;

[0129] Attention enhancement module, which applies hierarchical attention mechanism to features at different levels to obtain feature maps with local and global attention weights;

[0130] The feature fusion module uses a hierarchical fusion strategy to generate multi-level attention optimization features by combining feature maps with local and global attention weights;

[0131] The image reconstruction module uses the multi-level attention optimization features to reconstruct the image and obtain the reconstructed feature representation and image.

[0132] This embodiment also provides a computer device, which is suitable for the method of reducing the multimodal feature quantity of a large model based on multi-level coding, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the method of reducing the multimodal feature quantity of a large model based on multi-level coding as proposed in the above embodiment.

[0133] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0134] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for reducing the multimodal feature quantity of a large model based on multi-level coding as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage device, a flash memory, a disk or an optical disk.

[0135] In summary, the present invention achieves effective fusion of features at different levels from low-level details to high-level semantics by constructing a multi-scale feature pyramid and applying a hierarchical attention mechanism. This method can not only significantly reduce the number of features and reduce computational complexity, but also improve the model's ability to understand complex scenes. By introducing the joint optimization of local and global attention mechanisms, the present invention can capture key areas in the image while keeping an eye on the global structure, thereby solving the problems of insufficient multi-scale feature fusion and unbalanced application of local and global attention mechanisms in the prior art, and achieving more efficient feature representation and better task performance.

[0136] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for reducing multimodal feature quantities of a large model based on multi-level coding, characterized in that: include, Acquire high-resolution images and perform preprocessing; Perform feature extraction and preliminary dimension reduction on the preprocessed high-resolution image to obtain a preliminary compressed feature map; Based on the preliminary compressed feature map, a multi-scale feature pyramid is constructed to generate features at different levels; Apply hierarchical attention mechanism to different levels of features to obtain feature maps with local and global attention weights; The feature maps with local and global attention weights are fused in layers to generate multi-level attention optimization features. Use the multi-level attention optimization features to process the image reconstruction to obtain the reconstructed feature representation and image; The specific steps of collecting high-resolution images and preprocessing are as follows: Use scanners to capture high-resolution images; Perform image denoising and normalization on high-resolution images; The high-resolution image after denoising and normalization is divided into multiple regions, and an adaptive resolution selection algorithm is applied to obtain multi-resolution image blocks; Encode and store multi-resolution image blocks, and create metadata records for each resolution image block to generate a pre-processed high-resolution image; The features with local and global attention weights are fused in layers to generate a multi-level attention optimization feature map. The specific steps are as follows: A hierarchical fusion strategy is used to integrate the feature maps of local and global attention weights, a function to enhance the expressiveness of local features is defined, and the importance of local features is adjusted through convolution operations combined with nonlinear transformations. Define a function to improve the quality of global features, and adjust the importance of global features through the SENet squeeze-excitation network combined with GAP global average pooling; Define a function for calculating the cross relationship between the local feature map and the global feature map, and calculate the cross relationship between the local feature map and the global feature map through a multi-head attention mechanism and relative position encoding; Define a dynamic adjustment function for the overall feature combination, and use weighted summation to adjust the weights of the local feature map and the global feature map; By introducing an exponential decay coefficient, all the features obtained in the above steps are integrated into a multi-level attention optimization feature map, which is expressed as: ； in, represents a multi-level attention-optimized feature map, A feature map representing local attention weights, A feature map representing the global attention weights, is the amplitude coefficient of the sine function, is the amplitude coefficient of the cosine function, Is used to adjust The frequency of the sine function, Is used to adjust The frequency of the cosine function, yes The offset of yes The offset of is the weight coefficient of the influence of global features on local features, is the weight coefficient of the influence of local features on global features, is the intensity coefficient of the exponential decay term, Speed ​​parameter of the exponential decay.

2. The method for reducing multimodal feature quantities of large models based on multi-level coding according to claim 1, characterized in that: The pre-processed high-resolution image is subjected to feature extraction and preliminary dimension reduction to obtain a preliminary compressed feature map. The specific steps are as follows: The preprocessed high-resolution image is resized to a uniform size, and a deep convolutional neural network is applied to extract features to obtain a multi-layer feature map; Apply local response normalization to the multi-layer feature maps and perform preliminary dimensionality reduction using PCA principal component analysis; The feature map after the preliminary dimensionality reduction by PCA principal component analysis is further compressed using a lightweight encoder to obtain a preliminary compressed feature map.

3. The method for reducing multimodal feature quantities of large models based on multi-level coding according to claim 2, characterized in that: Based on the preliminary compressed feature map, a multi-scale feature pyramid is constructed to generate features at different levels. The specific steps are as follows: Unify the size of the preliminary compressed feature map, and create a base layer feature map of the multi-scale feature pyramid based on the compressed feature map after the unified size; Use transposed convolution to upsample the base layer feature map to construct a high-resolution feature map; Apply maximum pooling to downsample the base layer feature map to construct a low-resolution feature map; Apply 1x1 convolution to the high-resolution feature map and the low-resolution feature map to adjust the number of channels, add the feature maps after the adjustment of the number of channels element by element, and obtain the fused multi-scale feature map; Upsample the fused multi-scale feature map and add it element by element with the corresponding position features of the higher-resolution feature map. Apply a 3x3 convolution layer to eliminate the aliasing effect and generate a multi-scale fused high-resolution feature map. The multi-scale fused high-resolution feature map of the current stage is used as the output of the current stage and the input of the next stage, gradually building a complete multi-scale feature pyramid; Through the multi-scale feature pyramid, feature maps of different resolutions are extracted and fused at each level to obtain different levels of features from low-level details to high-level semantics.

4. The method for reducing multimodal feature quantities of large models based on multi-level coding according to claim 3, characterized in that: The hierarchical attention mechanism is applied to features at different levels to obtain feature maps with local and global attention weights. The specific steps are as follows: Based on features at different levels, a local attention mechanism is designed, and a convolutional neural network is used to extract the importance weights of local areas and obtain feature maps with local attention weights. Based on features at different levels, a global attention mechanism is designed, and the self-attention mechanism is used to capture global dependencies and obtain the feature map of global attention weights. The feature map of local attention weights and the feature map of global attention weights are fused using a weighted summation method to generate a feature map with local and global attention weights.

5. The method for reducing multimodal feature quantities of large models based on multi-level coding according to claim 4, characterized in that: The multi-level attention optimized feature map is processed by image reconstruction to obtain the reconstructed feature representation and image. The specific steps are as follows: The multi-level attention optimization feature map is normalized and the spatial size is increased using a transposed convolutional layer to obtain the processed feature map; Use deep convolutional neural networks to extract encoders and decoders, and introduce skip connections to convert the processed Concatenate the features of the encoder stage and the decoder stage of the feature map to obtain a feature map with skip connections; Apply adaptive instance normalization to feature maps with skip connections to adjust the mean and variance of feature maps; The feature map processed by adaptive instance normalization is upsampled through multiple transposed convolutional layers to generate a feature map with the same resolution as the original input image, and the tanh activation function is used to obtain an image with the same spatial size as the original input. The upsampled and activated feature maps are integrated to obtain the reconstructed feature representation and image.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for reducing the multimodal feature quantity of a large model based on multi-level coding as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for reducing the multimodal feature quantity of a large model based on multi-level coding as described in any one of claims 1 to 5 are implemented.

Images