Method and system for linking convolutional neural network and visual representation network

Abstract

The method for linking the convolutional neural network and the visual representation network comprises the following steps: acquiring an input image, and performing feature extraction on the input image through the convolutional neural network to obtain multi-scale features with different scales; performing linear transformation, segmentation, flattening and recombination on the multi-scale features through hierarchical feature learning to obtain multi-level feature representation; through convolution and maximum pooling operation, aggregating key associated information from the multi-scale features to obtain aggregated features, and projecting the aggregated features to a target dimension through convolution operation to obtain multi-scale mark output; identifying the multi-level feature representation by adopting a multi-head self-attention model, and generating a self-attention feature representation; and performing image enhancement according to the self-attention feature representation and the multi-scale mark output to obtain an image block attention representation, and taking the image block attention representation as the input of a ViTs visual representation network. According to the design, the feature extraction capability of the convolutional neural network and the visual representation capability of ViTs can be effectively combined.

Description

Technical Field

[0001] The present invention relates to the field of image representation and image recognition, and in particular to a method and system for linking a convolutional neural network and a visual representation network. Background Art

[0002] In the field of image representation and image recognition, Convolutional Neural Networks (CNNs) and Vision Transformer (ViTs), as two mainstream technologies, have each demonstrated outstanding application potential and technical advantages. Convolutional neural networks play a vital role in many key links of image processing with their excellent feature extraction capabilities; while ViTs, with their advanced visual representation capabilities, bring new perspectives and effective solutions to image recognition tasks. However, despite the significant advantages of each of these two networks, due to their essential differences in output and input structures, it is difficult to effectively combine convolutional neural networks with ViTs to fully utilize their advantages in image representation. At the same time, the ViTs architecture also suffers from the lack of inductive bias and high dependence on large-scale training data sets in the visual representation process.

[0003] In view of the problems in the prior art that the ViTs architecture lacks inductive bias in visual representation, relies heavily on a large number of training data sets, and is difficult to effectively combine due to the differences in output and input structures between convolutional neural networks and ViTs, the present invention proposes an innovative method for linking convolutional neural networks with visual representation networks. This method cleverly combines the feature extraction capability of convolutional neural networks with the visual representation capability of ViTs, and introduces a scale attention mechanism. This mechanism can capture cross-scale feature dependencies, thereby enhancing the spatial perception and global feature perception capabilities of images, and thus improving the generalization and accuracy of image classification. Summary of the invention

[0004] The purpose of the present invention is to overcome the problem that the output and input structure differences between the convolutional neural network and the ViTs architecture in the prior art make it difficult to effectively combine the convolutional neural network with ViTs, and to provide a method and system for linking the convolutional neural network with the visual representation network.

[0005] In a first aspect, the present invention provides a method for linking a convolutional neural network with a visual representation network, comprising:

[0006] S1. Obtain an input image, extract features of the input image through a convolutional neural network, and obtain multi-scale features with different scales and feature depths;

[0007] S2, through hierarchical feature learning, multi-scale features are linearly transformed, segmented, flattened and reorganized to obtain a multi-level feature representation;

[0008] S3, aggregating key correlation information from the multi-scale features through convolution and maximum pooling operations to obtain aggregated features, and projecting the aggregated features to the target dimension through convolution operations to obtain multi-scale labeled outputs;

[0009] S4. Use a multi-head self-attention model to identify the multi-level feature representation and generate a self-attention feature representation; perform image enhancement based on the self-attention feature representation and the multi-scale labeled output to obtain an image block attention representation, and use the image block attention representation as the input of the ViTs visual representation network.

[0010] The S1 includes:

[0011] S11. Get an input image with a width of H, a height of W, and 3 channels (RGB)

[0012] S12. A convolutional neural network with four convolution kernels is used to extract hierarchical feature representations X with four different scales and feature depths from the input image. i :

[0013]

[0014] Where P i Indicates the specified spatial resolution, C i Represents the specified channel dimension.

[0015] The S2 includes:

[0016] S21. Represent the hierarchical features X i Transformed into an embedded representation with a fixed dimension D through a linear mapping layer:

[0017] X′ i =Linear(X i );

[0018]

[0019] In the formula, X′ i is the hierarchical feature representation X i The corresponding embedding representation, Linear(·) means that the feature is linearly transformed;

[0020] S22, embedding representation X′ i Divide into N non-overlapping image blocks and flatten the image blocks to obtain the flattened image representation X″ i :

[0021]

[0022] In the formula, Represents spatial dimension;

[0023] S23, concatenate the flattened image representations to obtain a multi-level feature representation X Σ :

[0024] X Σ =[X′0′‖X1″‖X′2′‖X′3′];

[0025] In the formula, ‖ represents splicing.

[0026] In S3, through convolution and maximum pooling operations, key associated information is aggregated from the multi-scale features to obtain aggregated features including:

[0027] s0=Pool max (Conv(X0));

[0028] s1=Pool max (Conv(X1));

[0029] s2=Pool max (X2);

[0030] s3=X3;

[0031] s Σ =[s0‖s1‖s2‖s3];

[0032] In the formula, S Σ is the aggregated feature; Conv(·) represents the convolution operation; Pool max (·) indicates the max pooling operation.

[0033] In S3, the aggregated features are projected to the target dimension through a convolution operation to obtain a multi-scale labeled output including:

[0034]

[0035] Where, T scale is a multi-scale labeled output; BN(·) represents a batch normalization operation; ReLU(·) represents a relu activation function, and the target dimension is D.

[0036] In S4, a multi-head self-attention model is used to identify the multi-level feature representation, and generating the self-attention feature representation includes:

[0037] X′ Σ =X Σ +MSA(Linear(X Σ));

[0038] In the formula, X′ Σ is the self-attention feature representation; MSA(·) represents the multi-head self-attention mechanism.

[0039] In S4, image enhancement is performed based on the self-attention feature representation and the multi-scale labeling output, and the image block attention representation is obtained, including:

[0040] Y=X′ Σ +FFN(Linear(X′ Σ ))+FFN(Linear(T scale ));

[0041] Where FFN(·) represents a multi-layer feedforward neural network; Y is the image block attention representation.

[0042] In a second aspect, the present invention provides a system for linking a convolutional neural network and a visual representation network, comprising:

[0043] A feature extraction module is used to obtain an input image, extract features from the input image through a convolutional neural network, and obtain multi-scale features with different scales and feature depths;

[0044] Hierarchical feature learning module, used to perform linear transformation, segmentation, flattening and reorganization of multi-scale features through hierarchical feature learning to obtain a multi-level feature representation;

[0045] A multi-scale label generation module, used to aggregate key associated information from the multi-scale features through convolution and maximum pooling operations to obtain aggregated features, and project the aggregated features to the target dimension through convolution operations to obtain multi-scale label output;

[0046] The bidirectional attention module is used to identify the multi-level feature representation using a multi-head self-attention model and generate a self-attention feature representation; perform image enhancement based on the self-attention feature representation and the multi-scale labeled output to obtain an image block attention representation, and use the image block attention representation as the input of the ViTs visual representation network.

[0047] In a third aspect, the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in any one of the above-mentioned methods of linking a convolutional neural network with a visual representation network when executing the computer program.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method of linking a convolutional neural network with a visual representation network described in any one of the above items are implemented.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. In the method of linking the convolutional neural network and the visual representation network, the backbone network of the convolutional neural network is used to generate a visual representation with a hierarchical structure. At the same time, the multi-level feature representation is further processed through hierarchical feature learning to obtain a multi-scale label sequence. The multi-scale label sequence is processed through lightweight convolution to obtain a multi-scale label output. Finally, the multi-level feature representation and multi-scale label output are processed through the attention model to obtain the image block attention representation. The image block attention representation is used as the input of the ViTs visual representation network, realizing the link between the convolutional neural network and the ViTs visual representation network, and effectively combining the feature extraction ability of the convolutional neural network with the advanced representation ability of the ViTs visual representation network, improving the generalization and accuracy of image classification. Therefore, this design effectively combines the feature extraction ability of the convolutional neural network with the advanced representation ability of the ViTs visual representation network, improving the generalization and accuracy of image classification.

[0051] 2. In the method of linking the convolutional neural network and the visual representation network, the multi-scale features are linearly transformed, segmented, flattened and reorganized through hierarchical feature learning to obtain a multi-level feature representation. The hierarchical feature learning step expands the data of the input image. Then, the key related information is aggregated through convolution and maximum pooling operations. The self-attention mechanism captures the dependencies between features at different levels and the local dependencies between image blocks, which enhances the model's understanding of details and spatial structures. The image block attention representation obtained through the above data augmentation and feature enhancement operations is input into the ViTs visual representation network, which can effectively alleviate the lack of inductive bias and dependence on a large amount of labeled data in the ViTs visual representation network. Therefore, the image data is augmented and enhanced in this design, which effectively alleviates the lack of inductive bias and dependence on a large amount of labeled data in the ViTs model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a method for linking a convolutional neural network and a visual representation network provided by an embodiment of the present invention.

[0053] Figure 2 It is a flowchart of feature extraction provided by an embodiment of the present invention.

[0054] Figure 3 It is a flowchart of hierarchical feature learning provided by an embodiment of the present invention.

[0055] Figure 4 It is another flowchart of hierarchical feature learning provided by an embodiment of the present invention.

[0056] Figure 5 It is a structural diagram of a system for linking a convolutional neural network and a visual representation network provided by an embodiment of the present invention.

[0057] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Among the related technologies, Convolutional Neural Networks (CNNs) and Vision Transformer (ViTs) are both mainstream image processing methods, and each has demonstrated strong application potential and technical advantages. However, due to their essential differences in output and input structures, it is difficult to effectively combine Convolutional Neural Networks with ViTs and comprehensively utilize the strengths of both in image representation.

[0060] In order to solve the technical problems existing in the related art, the embodiment of the present invention provides a method for linking a convolutional neural network with a visual representation network, see Figure 1 , Figure 1 It is a flow chart of a method for linking a convolutional neural network and a visual representation network provided in an embodiment of the present invention, and the method includes steps S1-S4.

[0061] S1. Obtain an input image, extract features of the input image through a convolutional neural network, and obtain multi-scale features with different scales and feature depths.

[0062] In this embodiment, for an input image, we first use a convolutional neural network as a feature extraction layer to extract hierarchical visual representations from the input image, which have different scales and feature depths. For details, see Figure 2 , Figure 2is a flow chart of feature extraction provided by an embodiment of the present invention, such as Figure 2 As shown, the consistency check provided in this embodiment includes steps S11-S12.

[0063] S11. Get an input image with a width of H, a height of W, and 3 channels (RGB)

[0064] S12. Through a convolutional neural network with four convolution kernels, four hierarchical feature representations X with different scales and feature depths are extracted from the input image. i :

[0065]

[0066] In the above formula, P i Indicates the specified spatial resolution, C i Indicates the specified channel dimension, Represents a three-dimensional real vector.

[0067] For an image with input width H and height W, a convolutional neural network is used to extract hierarchical feature representations with different scales and feature depths from different network stages. In order to maintain generality, the number of convolution kernels we use is 4, and after convolution, four hierarchical feature representations of different scales are output: X0, X1, X2, and X3.

[0068] S2. Through hierarchical feature learning, multi-scale features are linearly transformed, segmented, flattened and reorganized to obtain a multi-level feature representation.

[0069] In this embodiment, after obtaining multi-scale features with different scales and feature depths, we convert these features into multi-scale image block labels through linear projection and image block splicing, and finally use them as the input of the ViTs visual representation network to learn cross-scale local dependencies and cross-image block global dependencies, and convert the hierarchical features extracted at different stages of the convolutional neural network into an input form acceptable to the ViTs visual representation network. For details, please refer to Figure 3 , Figure 3 is a flow chart of hierarchical feature learning provided by an embodiment of the present invention, such as Figure 3 As shown, the hierarchical feature learning provided in this embodiment includes steps S21-S23.

[0070] S21. Represent the hierarchical features X i Transformed into an embedded representation with a fixed dimension D through a linear mapping layer:

[0071] X′ i =Linear(X i );

[0072]

[0073] In the formula, X′ i is the hierarchical feature representation X i The corresponding embedded representation, Linear(·) means that the feature is linearly transformed. In this step, the hierarchical feature representation is converted into an embedded representation with a fixed dimension D through the linear mapping layer. This embedded representation is a low-dimensional, compact vector that retains the important information in the original feature representation as much as possible while removing redundancy and noise.

[0074] S22, embedding representation X′ i The image blocks are divided into N non-overlapping image blocks. In this embodiment, the default value of N is 49 (7×7). The image blocks are flattened to obtain a flattened image representation X″ i :

[0075]

[0076] In the formula, Represents the spatial dimension of the tag. Since the input of Vits is represented by token, Represents the spatial dimension of the Vits input tokens.

[0077] Specifically, Figure 4 As shown, Figure 4 is another flowchart of hierarchical feature learning provided by an embodiment of the present invention. After obtaining multi-scale features, these features are linearly transformed, segmented, and flattened to obtain a flattened image representation X″ i .

[0078] S23, concatenate the flattened image representations corresponding to different scale levels to obtain a multi-level feature representation X Σ , and the obtained multi-level feature representation X Σ Output as a result of hierarchical feature learning:

[0079]

[0080] In the formula, ‖ represents splicing.

[0081] S3. Aggregate key correlation information from the multi-scale features through convolution and maximum pooling operations to obtain aggregated features, and project the aggregated features to the target dimension through lightweight convolution to obtain multi-scale labeled output.

[0082] In this embodiment, to enhance the image representation, we aggregate key correlation information from features of different scales through downsampling strategies such as maximum pooling and convolution operations. This process converts the initial hierarchical feature representation X i The spatial dimension of is normalized to N to maintain the same channel dimension. The normalized hierarchical feature representation X i Then concatenate on the channel dimension and project to the target dimension D through lightweight convolution, converting the hierarchical feature representation X i As the input of the subsequent multi-head self-attention model to capture the dependencies between features of different scales.

[0083] In this embodiment, through convolution and maximum pooling operations, key associated information is aggregated from the multi-scale features to obtain aggregated features including:

[0084] s0=Pool max (Conv(X0));

[0085] s1=Pool max (Conv(X1));

[0086] s2=Pool max (X2);

[0087] s3=X3;

[0088] s Σ =[s0‖s1‖s2‖s3];

[0089] In the formula, S Σ is the aggregated feature; Conv(·) represents the convolution operation; Pool max (·) indicates the max pooling operation.

[0090] In this embodiment, the aggregated features are projected to the target dimension through lightweight convolution to obtain a multi-scale labeled output including:

[0091]

[0092] Where, T scale is a multi-scale labeled output; BN(·) represents a batch normalization operation; ReLU(·) represents a relu activation function, and the target dimension is D.

[0093] S4. Use a multi-head self-attention model to identify the multi-level feature representation and generate a self-attention feature representation; perform image enhancement based on the self-attention feature representation and the multi-scale labeled output to obtain an image block attention representation, and use the image block attention representation as the input of the ViTs visual representation network.

[0094] In this embodiment, a multi-head self-attention (MSA) model is used to identify the multi-level feature representation, and generating the self-attention feature representation includes:

[0095] X′ Σ =X Σ +MSA(Linear(X Σ ));

[0096] In the formula, X′ Σ is the self-attention feature representation; MSA(·) represents the multi-head self-attention mechanism.

[0097] Image enhancement is performed based on self-attention feature representation and multi-scale labeling output, and the image block attention representation is obtained, including:

[0098] Y=X′ Σ +FFN(Linear(X′ Σ ))+FFN(Linear(T scale ));

[0099] Where FFN(·) represents a multi-layer feedforward neural network; Y is the image block attention representation.

[0100] In summary, the embodiment of the present invention provides a method for linking a convolutional neural network with a visual representation network, which first obtains the multi-scale features of the input image, then greatly augments the data of the input image through hierarchical feature learning, and then enhances the understanding of image details and spatial structure through a multi-head self-attention mechanism. The image block attention representation Y is obtained through the above data augmentation and feature enhancement operations, and the image block attention representation Y is used as the input of the ViTs visual representation network, which can effectively alleviate the lack of inductive bias and dependence on a large amount of labeled data in the ViTs model.

[0101] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a system linking a convolutional neural network and a visual representation network. The system linking a convolutional neural network and a visual representation network can be implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0102] See also Figure 5 , Figure 5 is a schematic diagram of a structure of a system for linking a convolutional neural network and a visual representation network provided by an embodiment of the present invention, such as Figure 5 As shown, the system for linking a convolutional neural network and a visual representation network provided by an embodiment of the present invention includes:

[0103] A feature extraction module is used to obtain an input image, and perform feature extraction on the input image through a convolutional neural network to obtain multi-scale features with different scales and feature depths. The feature extraction module is used to execute step S1, and S1 includes:

[0104] S11. Get an input image with a width of H and a height of W

[0105] S12. Through a convolutional neural network with four convolution kernels, hierarchical feature representations X0, X1, X2, and X3 with four different scales and feature depths are extracted from the input image:

[0106]

[0107] Where P i Indicates the specified spatial resolution, C i Represents the specified channel dimension.

[0108] A hierarchical feature learning module is used to perform linear transformation, segmentation, flattening and reorganization on multi-scale features through hierarchical feature learning to obtain a multi-level feature representation, wherein the hierarchical feature learning module is used to execute step S2, and S2 includes:

[0109] S21. Represent the hierarchical features X i Transformed into an embedded representation with a fixed dimension D through a linear mapping layer:

[0110] X′ i =Linear(X i );

[0111]

[0112] In the formula, X′ i is the hierarchical feature representation X i The corresponding embedding representation, Linear(·) means that the feature is linearly transformed;

[0113] S22, embedding representation X′ i Divide into N non-overlapping image blocks and flatten the image blocks to obtain the flattened image representation X″ i :

[0114]

[0115] In the formula, represents the spatial dimension of the marker;

[0116] S23, concatenate the flattened image representations to obtain a multi-level feature representation X Σ :

[0117] X Σ =[X″0‖X″1‖X″2‖X″3];

[0118] In the formula, ‖ represents splicing.

[0119] The multi-scale label generation module is used to aggregate key associated information from the multi-scale features through convolution and maximum pooling operations to obtain aggregated features, and project the aggregated features through lightweight convolution to obtain multi-scale label outputs. The multi-scale label generation module is used to execute step S3. In S3, the key associated information is aggregated from the multi-scale features through convolution and maximum pooling operations to obtain aggregated features, including:

[0120] s0=Pool max (Conv(X0));

[0121] s1=Pool max (Conv(X1));

[0122] s2=Pool max (X2);

[0123] s3=X3;

[0124] s Σ =[s0‖s1‖s2‖s3];

[0125] In the formula, S Σ is the aggregated feature; Conv(·) represents the convolution operation; Pool max (·) indicates the maximum pooling operation;

[0126] In S3, the aggregated features are projected to the target dimension through a convolution operation to obtain a multi-scale labeled output, and the obtained multi-scale labeled output includes:

[0127]

[0128] Where, T scale is a multi-scale labeled output; BN(·) represents a batch normalization operation; ReLU(·) represents a relu activation function, and the target dimension is D.

[0129] A bidirectional attention module is used to use a multi-head self-attention model to identify the multi-level feature representation and generate a self-attention feature representation; perform image enhancement based on the self-attention feature representation and the multi-scale labeled output to obtain an image block attention representation, and use the image block attention representation as an input of the ViTs visual representation network. The bidirectional attention module is used to execute step S4.

[0130] The bidirectional attention module includes a hierarchical attention module and an image attention module.

[0131] The hierarchical attention module is used to execute the step in S4, using a multi-head self-attention (MSA) model to identify the multi-level feature representation and generate a self-attention feature representation.

[0132] The adopting of a multi-head self-attention (MSA) model to identify the multi-level feature representation and generating the self-attention feature representation comprises:

[0133] X′ Σ =X Σ +MSA(Linear(X Σ ));

[0134] In the formula, X′ Σ is the self-attention feature representation; MSA(·) represents the multi-head self-attention mechanism.

[0135] The image attention module is used to execute the step in S4 of performing image enhancement according to the self-attention feature representation and the multi-scale labeling output to obtain the image block attention representation.

[0136] The image enhancement is performed according to the self-attention feature representation and the multi-scale label output to obtain the image block attention representation, which includes:

[0137] Y=X′ Σ +FFN(Linear(X′ Σ ))+FFN(Linear(T scale ));

[0138] Where FFN(·) represents a multi-layer feedforward neural network; Y is the image block attention representation.

[0139] From the above description of the system linking the convolutional neural network and the visual representation network, it can be seen that the present invention greatly augments the data of the input image through the hierarchical feature learning module; and enhances the understanding of image details and spatial structure through the multi-scale label generation module and the bidirectional attention module. The image block attention representation Y is obtained through the above data augmentation and feature enhancement operations. The image block attention representation Y can be directly used as the input of the ViTs model, which effectively alleviates the lack of inductive bias and dependence on a large amount of labeled data in the ViTs visual representation network.

[0140] Also, see Figure 6 , Figure 6 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present invention, such as Figure 6As shown, the electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the above-mentioned method of linking a convolutional neural network with a visual representation network when executing the computer program.

[0141] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, and when the instructions are executed by the processor, the steps in the method of linking a convolutional neural network and a visual representation network provided in the above embodiments can be implemented.

[0142] Generally speaking, the computer instructions for implementing the method of the present invention may be carried in any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media may include any computer-readable media, except for the signal itself that is temporarily propagating.

[0143] Computer-readable storage media, for example, may be, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with a system, device or device for executing instructions.

[0144] One or more programming languages ​​or their combinations can be used to write computer program codes for performing the operation of the present invention. These programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and also include conventional procedural programming languages, such as C language or similar programming languages. In particular, Python language suitable for neural network calculations and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through various types of networks, including a local area network (LAN) or a wide area network (WAN), or an Internet connection is performed through an Internet service provider.

[0145] Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for linking a convolutional neural network with a visual representation network, characterized in that: The method comprises: S1. Obtain an input image, extract features of the input image through a convolutional neural network, and obtain multi-scale features with different scales and feature depths; S2, through hierarchical feature learning, multi-scale features are linearly transformed, segmented, flattened and reorganized to obtain a multi-level feature representation; S3, aggregating key correlation information from the multi-scale features through convolution and maximum pooling operations to obtain aggregated features, and projecting the aggregated features to the target dimension through convolution operations to obtain multi-scale labeled outputs; S4. Use a multi-head self-attention model to identify the multi-level feature representation and generate a self-attention feature representation; perform image enhancement based on the self-attention feature representation and the multi-scale labeled output to obtain an image block attention representation, and use the image block attention representation as the input of the ViTs visual representation network.

2. The method of linking a convolutional neural network and a visual representation network according to claim 1, characterized in that: The S1 includes: S11, obtain the input image X with a width of H, a height of W, and a channel number of 3, S12. A convolutional neural network with four convolution kernels is used to extract hierarchical feature representations X with four different scales and feature depths from the input image. i : Where P i Indicates the specified spatial resolution, C i Represents the specified channel dimension.

3. The method of linking a convolutional neural network and a visual representation network according to claim 2, characterized in that: The S2 includes: S21. Represent the hierarchical features X i Transformed into an embedded representation with a fixed dimension D through a linear mapping layer: X′ i =Linear(X i ); In the formula, X′ i is the hierarchical feature representation X i The corresponding embedding representation, Linear(·) means that the feature is linearly transformed; S22, embedding representation X′ i Divide into N non-overlapping image blocks and flatten the image blocks to obtain the flattened image representation X″ i : Where P i ′2 Represents spatial dimension; S23, concatenate the flattened image representations to obtain a multi-level feature representation X Σ : X Σ =[X″0‖X″1‖X″2‖X″3]; In the formula, ‖ represents splicing.

4. The method of linking a convolutional neural network and a visual representation network according to claim 3, characterized in that: In S3, through convolution and maximum pooling operations, key associated information is aggregated from the multi-scale features to obtain aggregated features including: s0=Pool max (Conv(X0)); s1=Pool max (Conv(X1)); s2=Pool max (X2); s3=X3; <h2 style=";text-align:left;direction:ltr">s<h2 style=";text-align:left;direction:ltr"> Σ <h2 style=";text-align:left;direction:ltr"> =[s0‖s1‖s2‖s3]; In the formula, S Σ is the aggregated feature; Conv(·) represents the convolution operation; Pool max (·) indicates the max pooling operation.

5. The method of linking a convolutional neural network and a visual representation network according to claim 4, characterized in that: In S3, the aggregated features are projected to the target dimension through a convolution operation to obtain a multi-scale labeled output including: Where, T scale is a multi-scale labeled output; BN(·) represents a batch normalization operation; ReLU(·) represents a relu activation function, and the target dimension is D.

6. The method of linking a convolutional neural network and a visual representation network according to claim 5, characterized in that: In S4, a multi-head self-attention model is used to identify the multi-level feature representation, and generating the self-attention feature representation includes: X′ Σ =X Σ +MSA(Linear(X Σ )); In the formula, X′ Σ is the self-attention feature representation; MSA(·) represents the multi-head self-attention mechanism.

7. The method of linking a convolutional neural network and a visual representation network according to claim 6, characterized in that: In S4, image enhancement is performed based on the self-attention feature representation and the multi-scale labeling output, and the image block attention representation is obtained, including: Y=X′ Σ +FFN(Linear(X′ Σ ))+FFN(Linear(T scale )); Where FFN(·) represents a multi-layer feedforward neural network; Y is the image block attention representation.

8. A system linking a convolutional neural network with a visual representation network, including: A feature extraction module is used to obtain an input image, extract features from the input image through a convolutional neural network, and obtain multi-scale features with different scales and feature depths; Hierarchical feature learning module, used to perform linear transformation, segmentation, flattening and reorganization of multi-scale features through hierarchical feature learning to obtain a multi-level feature representation; A multi-scale label generation module, used to aggregate key correlation information from the multi-scale features through convolution and maximum pooling operations to obtain aggregated features, and project the aggregated features to the target dimension through convolution operations to obtain multi-scale label output; A bidirectional attention module, for identifying the multi-level feature representation using a multi-head self-attention model and generating a self-attention feature representation; Image enhancement is performed based on the self-attention feature representation and multi-scale labeled output to obtain the image block attention representation, which is used as the input of the ViTs visual representation network.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images