Image processing method, electronic device, storage medium, and program product

Abstract

The present application provides an image processing method that can be applied to the field of artificial intelligence technology, particularly to the field of digital image processing. The method includes: performing multi-scale sampling on an original image to obtain multiple sampled images of different scales; performing block processing on each sampled image to obtain a block sequence of each sampled image, wherein the blocks of the sampled images of different scales have the same scale; performing self-attention calculation on the block sequence of each sampled image to obtain an enhanced feature sequence of each sampled image; converting the enhanced feature sequence of each sampled image into a feature map of a preset scale to obtain multiple feature maps of the preset scale; and generating an enhanced image based on the multiple feature maps of the preset scale. The present application also provides an electronic device, a storage medium, and a program product.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, specifically to the field of digital image processing, and more specifically to an image processing method, electronic device, storage medium and program product. Background Art

[0002] In the fields of computer vision and image processing, effectively integrating feature information from diverse sources is a key challenge in improving algorithm performance. Due to the diverse size, shape, and texture of target objects, a single feature extraction method often fails to fully capture their key attributes. Related image processing methods struggle to fully capture and integrate contextual information at varying granularities when processing multi-scale information, resulting in incomplete feature representation. Summary of the Invention

[0003] In view of the above problems, the present application provides an image processing method, an electronic device, a storage medium and a program product.

[0004] According to one aspect of the present application, an image processing method is provided, comprising: performing multi-scale sampling on an original image to obtain a plurality of sampled images of different scales; performing block processing on each sampled image to obtain a block sequence of each sampled image, wherein the scales of the blocks of the sampled images of different scales are the same; performing self-attention calculation on the block sequence of each sampled image to obtain an enhanced feature sequence of each sampled image; converting the enhanced feature sequence of each sampled image into a feature map of a preset scale to obtain a plurality of feature maps of the preset scales; and generating an enhanced image based on the feature maps of the plurality of preset scales.

[0005] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0006] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0007] According to another aspect of the present application, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0009] Figure 1An application scenario diagram of the image processing method according to an embodiment of the present application is shown;

[0010] Figure 2 A flowchart of an image processing method according to an embodiment of the present application is shown;

[0011] Figure 3 A schematic diagram showing an image processing method according to another embodiment of the present application is shown;

[0012] Figure 4 A visual schematic diagram of an image processing method according to an embodiment of the present application is shown;

[0013] Figure 5 shows a structural block diagram of an image processing device according to an embodiment of the present application;

[0014] Figure 6 A block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0015] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0016] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0017] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0018] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0019] In the technical solution of this application, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0020] In the field of digital image processing, efficiently and comprehensively understanding image content is a core challenge for many advanced applications, such as computer vision, medical image analysis, and intelligent driving. The information contained in images is highly multi-scale. From subtle textures to macroscopic structures, features at different scales are crucial for fully capturing image semantics. However, existing image processing methods often struggle to effectively capture and integrate contextual information at different granularities when processing multi-scale information.

[0021] For example, while traditional convolutional neural networks (CNNs) excel in extracting local features, their fixed receptive fields limit their ability to capture global dependencies and long-range interactions in images. Simply expanding the receptive field by stacking more layers or using larger convolution kernels often increases computational costs or leads to information redundancy, and makes it difficult to adaptively focus on truly important areas in the image. Furthermore, while some multi-scale processing strategies, such as feature pyramid structures, can generate feature maps of varying resolutions, effectively aligning and fusing these heterogeneous features from different scales to avoid information loss or conflict remains an urgent problem. In practical applications, the size, texture details, and spatial distribution of objects in images are often variable, requiring the processing system to flexibly adapt to and extract key information at different scales.

[0022] In recent years, the attention mechanism has garnered widespread attention for its ability to model long-range dependencies, while hierarchical feature extraction strategies offer an effective approach for capturing multi-scale information. However, a key challenge in this field remains the design of a framework that organically combines multi-scale hierarchical feature extraction from images with global context modeling (via the attention mechanism) and ultimately achieves effective information fusion. This framework, which fully exploits the joint features of images at different levels of abstraction, addresses the challenges of image understanding in complex scenarios and significantly improves feature representation capabilities.

[0023] In related technologies, multi-scale feature extraction and fusion tasks for images often face the following key bottlenecks:

[0024] On the one hand, in the traditional hierarchical feature extraction process, features between different resolution levels often suffer from information redundancy or loss of key details. This makes it difficult for features from different layers to effectively complement each other during the subsequent fusion process, which in turn affects the model's accurate understanding and refined analysis of the overall semantics of the image. For example, a simple downsampling operation may cause the irreversible loss of key details in high-resolution images. However, related methods lack effective mechanisms to dynamically capture long-range dependencies and filter highly discriminative features when extracting features within a layer. As a result, they are unable to effectively suppress the interference of redundant information on the final image representation, and it is difficult to ensure that the model can focus on salient areas at different scales.

[0025] On the other hand, related image processing methods often have difficulty effectively modeling the hierarchical spatial associations of images when integrating features from different resolution levels, resulting in the gradual attenuation of cross-scale semantic information during the fusion process, making the final image representation lack contextual consistency and robustness. Specifically, although some methods have attempted to introduce attention mechanisms or Transformer structures, their tokenization strategies between different resolution levels are usually not unified or adaptive, and fail to achieve consistent and fixed sizes of tokens generated by all levels in the spatial dimension. This makes it difficult for intra-layer feature extraction to fully capture long-distance dependencies at a uniform granularity, and limits the model's ability to discriminate complex textures and structural relationships.

[0026] In addition, related fusion methods, such as simple feature concatenation or global average pooling, fail to fully consider the unique contributions and interactions of features at each layer, resulting in information redundancy or dilution of key information, failing to achieve the complementary advantages of features at different scales, and unable to achieve refined alignment and nonlinear correlation mining of multi-resolution features through pixel-level addition. This makes it difficult to bridge the "gap" between information at different scales, ultimately limiting the depth and robustness of the overall image information representation, making it difficult to achieve optimal performance in complex image processing tasks.

[0027] Figure 1 The figure shows an application scenario of the image processing method according to an embodiment of the present application.

[0028] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0029] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0030] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0031] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0032] It should be noted that the image processing method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present application can generally be set in the server 105. The image processing method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0033] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0034] Figure 2 A flowchart of an image processing method according to an embodiment of the present application is shown.

[0035] like Figure 2As shown, the image processing method of this embodiment includes operations S210 to S250, and the image processing method can be executed by at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105.

[0036] In operation S210 , multi-scale sampling is performed on the original image to obtain a plurality of sampled images of different scales.

[0037] According to an embodiment of the present application, the scale of the image may be the resolution of the image. Multi-scale sampling may include upsampling and downsampling, which are not limited here. Multi-scale sampling is preferably downsampling. In order to obtain sampled images of different scales, the original image may be downsampled based on different step sizes to obtain sampled images of different scales. In this case, the step size of each downsampling may be, for example, 2, 4, 6, or 2, 5, 7, etc., and may not be limited thereto. The original image may also be downsampled to obtain a downsampled image, and then the downsampled image may be further downsampled to obtain the next layer of downsampled images, and so on, to obtain sampled images of different scales. In this case, the step size of each downsampling layer may be the same or different.

[0038] For example, let the original image be X0, and its scale is H×W×C. Where H is the height of the original image, W is the width of the original image, and C is the number of channels of the original image. The step size of each downsampling layer is 2, so the sampled image X after downsampling at the kth layer is k It can be determined based on the following formula (1).

[0039] X k = Downsample(X k-1 ) = (X k-1 ) 2, k = 1, 2, ..., N (1)

[0040] In formula (1), Downsample(·) represents the downsampling algorithm, (·) indicates that downsampling uses a bilinear interpolation algorithm. 2 is a downsampling operation with a step size of 2. Through this operation, a total of N layers of downsampled images X1, X2,..., X N , where X N is the smallest downsampled image with a spatial resolution of H / 2 N × W / 2 N .

[0041] Based on this method, we can perform multi-scale downsampling on the original image, gradually reducing the spatial resolution and generating feature pyramids of different scales. Each layer of the feature pyramid represents a sampled image at a specific scale. The sampled image can be considered a low-resolution version of the original image, while still preserving the image's visual structure.

[0042] In operation S220 , each sampled image is subjected to block processing to obtain a block sequence of each sampled image, wherein the blocks of sampled images of different scales have the same scale.

[0043] According to embodiments of the present application, the block segmentation method used for performing the block segmentation process is not limited to various block segmentation methods, as long as the scales of the blocks obtained from the sampled images of different scales are all the same. After each sampled image is segmented to obtain one or more blocks, a block sequence of the sampled image can be obtained by sequentially concatenating all the blocks of each sampled image.

[0044] In operation S230 , self-attention calculation is performed on the block sequence of each sampled image to obtain an enhanced feature sequence of each sampled image.

[0045] According to an embodiment of the present application, the enhanced feature sequence can represent abstract, attention-enhanced image features that no longer have the visual structure of the image.

[0046] In operation S240 , the enhanced feature sequences of the respective sampled images are converted into feature maps of a preset scale, thereby obtaining a plurality of feature maps of the preset scale.

[0047] According to the embodiments of the present application, the preset scale can be customized based on business needs and is not limited here. By reorganizing the enhanced feature sequence back to its original spatial structure, a feature map can be obtained. The feature map can represent the model's deep understanding of the image and is used for subsequent fusion.

[0048] In operation S250 , an enhanced image is generated based on a plurality of feature maps of preset scales.

[0049] According to an embodiment of the present application, corresponding to this operation, multiple feature maps of preset scales may be first fused to obtain fused features, and then the fused features may be decoded to obtain an enhanced image fused with multi-scale features of the original image.

[0050] Through the above-mentioned embodiments of the present application, by maintaining the consistency of the scales of the blocks obtained by segmenting images of different scales, the spatial receptive fields represented by the blocks on the original image are made consistent, so that after the features of different scale layers are converted to the preset scale, their information can be "aligned" in space, which ultimately helps to improve the fusion effect.

[0051] Figure 3A schematic diagram of an image processing method according to another embodiment of the present application is shown.

[0052] like Figure 3 As shown, for an original image 310 with a scale of H×W×C, the original image is downsampled layer by layer with a preset step size to obtain multi-layer sampled images, so that the sampled images of different layers have different scales. The step size can be 2 n , where n is a positive integer. For example, by performing a layered downsampling operation with a step size of 2, a first downsampled image 311 with a scale of H / 2×W / 2×C, a second downsampled image 312 with a scale of H / 4×W / 4×C, and a third downsampled image 313 with a scale of H / 8×W / 8×C can be obtained.

[0053] Next, a blocking operation based on the aforementioned adaptive blocking strategy is performed on each downsampled image, and a first image block sequence 321 , a second image block sequence 322 , a third image block sequence 323 , etc. can be obtained accordingly.

[0054] According to one embodiment of the present application, the so-called adaptive blocking strategy refers to that, for each sampled image, the sampled image is uniformly grid-blocked according to the number of layers in which the sampled image is located, to obtain a plurality of blocks, so that the scales of the blocks of sampled images of different scales are the same. For example, the number of blocks and the block scale of the sampled image can be determined according to the number of layers in which the sampled image is located and the scale of the sampled image; the sampled image is uniformly grid-blocked according to the number of blocks and the block scale to obtain a plurality of blocks, such as Tokens. The scales of the plurality of blocks are all the block scales, and the number of the plurality of blocks is the number of blocks. Based on the plurality of blocks, a block sequence of the sampled image, such as a Token sequence, can be generated.

[0055] For example, for the N layers of downsampled images X1, X2, ..., X N For the convenience of description, X N Considered as layer 1 (smallest image layer), X N-1 is considered as the second layer, and so on, X1 is considered as the Nth layer (the largest downsampled image layer). Then the jth layer downsampled image can correspond to the X in the Nth layer downsampled image. N-j+1 . Assume that the spatial resolution of the j-th layer sampling image is H j ×W j = (H / 2 N-j+1 )×(W / 2 N-j+1 ). The j-th layer feature map can be divided into a 2 j ×2 j The grid of M j = 4 jToken. The scale of each Token is (H j / 2 j )×(W j / 2 j )×C. It is worth noting that through this block strategy, the tokens generated by all layers have the same scale in the spatial dimension, that is, (H / 2 N+1 )×(W / 2 N+1 )×C. All M in the jth layer j Tokens are reorganized into a Token sequence T j , Token sequence T j The dimension is M j ×(P T 2 ×C), where P T =H / 2 N+1 or W / 2 N+1 , represents the spatial side length of Token.

[0056] Through the above-mentioned embodiments of the present application, by introducing a preset step size and combining the number of layers and scales of the sampled images, an adaptive blocking strategy for sampled images of different scales is determined, so that the scales of the blocks of sampled images of each scale are the same blocking scale, which can offset the sampling multiples between sampled images of different scales, thereby ensuring that the image blocks ultimately generated for the sampled images of each scale have the same scale in the spatial dimension.

[0057] Next, continue to refer to Figure 3 For each image block sequence, a self-attention calculation operation based on the aforementioned self-attention calculation method is performed, and a first enhanced feature sequence 331, a second enhanced feature sequence 332, a third enhanced feature sequence 333, etc. can be correspondingly obtained. For example, a multi-head self-attention (MHA) calculation can be combined to achieve hierarchical attention feature extraction, and a self-attention (SA) calculation can also be combined to achieve hierarchical attention feature extraction, which is not limited here.

[0058] Taking MHA calculation as an example, based on the above embodiment, the Token sequence T generated by each layer of downsampled image is j Multi-head self-attention calculations can be performed independently.

[0059] Let the j-th layer Token sequence T j The corresponding query, key, and value matrices are .

[0060] The calculation principle of single-head attention output is shown in formula (4).

[0061] (4)

[0062] In formula (4), is the dimension of the key vector, Representation matrix The transpose of .

[0063] Based on the single-head attention calculation principle shown in formula (4), after parallel calculation of multiple attention heads, the outputs of each head can be spliced ​​and linearly projected in combination with formula (5) to obtain the j-th layer Token sequence T j The enhanced feature sequence T output after self-attention calculation j out .

[0064] T j out = Concat(head1, ..., head h ) W o (5)

[0065] Where h is the number of attention heads, W o is the output projection matrix.

[0066] By dividing the sampled images of different scales into blocks of the same scale based on the aforementioned method, the same self-attention module can be reused for all layers during the self-attention calculation process, which is conducive to simplifying the model architecture and reducing the number of parameters.

[0067] Next, the enhanced feature sequences of each sampled image can be converted into an initial feature map. The scale of the initial feature map is the same as the scale of the sampled image used to obtain the initial feature map. The initial feature map of each sampled image is then resampled to obtain a feature map of a preset scale for each sampled image.

[0068] According to an embodiment of the present application, the preset scale may be a scale of a sampled image at a certain location, or a scale customized by a user, which is not limited here.

[0069] Based on the above embodiment, the enhanced feature sequence T output by self-attention of each layer of Token sequence can be j out Reorganize back to its original image space structure to form the processed initial feature map F j processed , whose scale is the same as the j-th layer downsampled image X N-j+1 Same, that is (H / 2 N-j+1 )×(W / 2 N-j+1 )×C. Then, the initial feature map F of each layer can be scaled according to the preset scale.j processed Resampling is performed, for example, by using a bilinear interpolation method to restore the scale of the initial feature map of each layer to a preset scale, so as to obtain a feature map of the preset scale of each sampled image.

[0070] According to an embodiment of the present application, the preset scale may be the scale of the original image.

[0071] In this case, based on the above embodiment, since the sampled images of each layer are obtained by downsampling the original image, corresponding to this scene, the initial feature map F after each layer is processed can be j processed Upsampling is performed, for example, bilinear interpolation can be used to restore its scale to the original image scale H×W×C. Assume that the upsampling operation is (·), then the feature map after upsampling at layer j is F j upsampled = (F j processed ).

[0072] For example, continue to refer to Figure 3 , a reorganization operation can be performed on each enhanced feature sequence to reorganize the result back to the original image space, and correspondingly, a first initial feature map 341 with a scale of H / 2×W / 2×C, a second initial feature map 342 with a scale of H / 4×W / 4×C, a third initial feature map 343 with a scale of H / 8×W / 8×C, etc. An upsampling operation can be performed on each initial feature map to the scale of the original image 310, and correspondingly, a first feature map 351, a second feature map 352, a third feature map 353, etc., all with a scale of H×W×C, can be obtained.

[0073] By determining the preset scale as the scale of the original image and restoring the feature map based on the scale of the original image, it is beneficial to more accurately represent the features of the feature map in the original image space, thereby improving the accuracy of the subsequent enhanced image.

[0074] Next, multiple feature maps of preset scales may be pixel-wise added to obtain an enhanced feature map, and then the enhanced feature map may be decoded to obtain an enhanced image.

[0075] Based on the above embodiment, taking the preset scale as the scale of the original image as an example, formula (6) can be combined to upsample all N layers of initial feature maps to the feature map F1 of the original resolution. upsampled , F2 upsampled , ..., F N upsampledPixel-level addition generates a fusion result R containing multi-scale attention fusion information. The dimensions of R are H × W × C. This fusion result can be used as the enhanced image output or as the input for subsequent tasks (such as image recognition, segmentation, and reconstruction), providing richer and more robust feature representations for downstream applications.

[0076] (6)

[0077] According to an embodiment of the present application, the enhanced image determined based on the fusion result R may be fused with local features of the original image at different scales.

[0078] For example, continue to refer to Figure 3 By performing pixel-level fusion and decoding on the first feature map 351, the second feature map 352, the third feature map 353, etc., an enhanced image 360 ​​with a scale of H×W×C can be obtained.

[0079] Through the above-mentioned embodiments of the present application, it is possible to effectively utilize layered downsampling to obtain multi-scale information of the image, and through adaptive blocking and self-attention mechanisms, capture local features of the image at different resolution levels, and finally through pixel-level fusion, organically combine the fine features and macro features of each scale, thereby significantly improving the feature expression ability and processing effect of the image.

[0080] According to an embodiment of the present application, in the process of executing the above-mentioned image processing method, the original image can also be processed into blocks to obtain a block sequence of the original image, wherein the scale of the blocks of the original image is the same as the scale of the blocks of each sampled image. Self-attention calculation is performed on the block sequence of the original image to obtain an enhanced feature sequence of the original image. The enhanced feature sequence of the original image is converted into a feature map of a preset scale. The above-mentioned process of performing pixel-level addition on a plurality of feature maps of preset scales to obtain an enhanced feature map can also be expressed as: performing pixel-level addition on the feature map of the preset scale of the original image and the feature maps of the preset scale of the plurality of sampled images to obtain an enhanced feature map.

[0081] It should be noted that the specific implementation methods of segmenting the original image, calculating self-attention, and restoring the original image can refer to the description of the implementation methods of segmenting the sampled image, calculating self-attention, and restoring the original image in the aforementioned embodiment, which will not be repeated here.

[0082] In the process of pixel-level fusion, the feature map of the original image and the feature maps of each sampled image can be combined and added at the pixel level. The obtained enhanced image can include both the global features and local features of the original image.

[0083] Through the above-mentioned embodiments of the present application, it is possible to effectively utilize layered downsampling to obtain multi-scale information of the image, and through adaptive blocking and self-attention mechanisms, capture local and global context dependencies at different resolution levels, and finally through pixel-level fusion, organically combine the fine features and macro features of each scale, thereby significantly improving the feature expression ability and processing effect of the image.

[0084] Figure 4 A visual schematic diagram of an image processing method according to an embodiment of the present application is shown.

[0085] like Figure 4 As shown, for the original image 400, after the layered downsampling operation, for example, three downsampled images 410 can be obtained. After each downsampled image is divided into blocks, three block sequences 420 can be obtained. After self-attention calculation is performed on each block sequence, three enhanced feature sequences 430 can be obtained. According to the mapping space with the same scale as the scale of the downsampled images corresponding to the three enhanced feature sequences 430, the three enhanced feature sequences 430 can be reorganized back to their original image space structure to obtain three initial feature maps 440. Each initial feature map is upsampled to obtain three feature maps 450. The three feature maps 450 are fused at the pixel level to obtain enhanced features 460. The enhanced feature map 460 is decoded to obtain an enhanced image ( Figure 4 not shown).

[0086] Through the above-mentioned embodiments of the present application, an image processing method based on multi-scale hierarchical attention fusion is realized. By performing hierarchical downsampling on the input image to obtain a multi-scale view, an adaptive blocking strategy is used to generate blocks at each layer and a self-attention mechanism is applied to capture long-distance dependencies within the layer. Finally, the processed features of each layer are upsampled to the original resolution and pixel-level fusion is performed. This method can effectively extract and fuse the multi-scale features of images at different resolutions, fully explore the joint features of images at different resolutions, enhance the image representation ability and information utilization efficiency, thereby meeting the challenges of multi-scale information processing and significantly improving the feature expression ability of images. It has important theoretical and practical significance.

[0087] According to an embodiment of the present application, the original image may include a first image and a second image. The first image may be an image generated based on detection data of a first modality of the object. The second image may be an image generated based on detection data of a second modality of the object.

[0088] For example, the first modality may include spectroscopy and the second modality may include sonar. Thus, the first image may include a spectral image, and the second image may include a sonar image. After acquiring a spectral image and a sonar image of the same object, the spectral image and the sonar image may be processed separately based on operations S210 through S240 described above to obtain spectral feature maps at various preset scales for the spectral image and sonar feature maps at various preset scales for the sonar image. The spectral feature maps at various preset scales and the sonar feature maps at various preset scales may then be fused to obtain an enhanced image that incorporates multimodal information, including spectral and sonar information.

[0089] It should be noted that, for implementation details of the image processing method in this embodiment, reference may be made to the description in the aforementioned embodiments, which will not be repeated here.

[0090] According to the embodiments of the present application, through the collaborative design of the hierarchical attention mechanism and cross-modal convolutional fusion, multi-scale complementary feature extraction of spectral images and sonar images can be achieved, significantly improving the classification accuracy of complex scenes. The hierarchical attention module dynamically calculates feature sequences for different resolution levels, which can effectively suppress noise interference and focus on cross-modal key areas, so that the model can still maintain robust feature representation capabilities under low signal-to-noise ratio conditions. By modeling nonlinear associations through local receptive fields, the complementary information of spectral reflectivity and sonar echo intensity is fully explored. The joint optimization of the two can avoid the problems of feature over-smoothing or local information loss in traditional methods, while ensuring computational efficiency, and effectively improve the accuracy of multi-source data classification.

[0091] In addition, the architectural design based on layered downsampling and attention feature superposition breaks through the reliance of traditional multimodal fusion methods on single-resolution features and achieves full-level integration of multi-granularity semantic information. By generating a multi-scale feature pyramid through layer-by-layer downsampling and combining it with the process of backsampling to restore resolution, it can not only preserve the detailed texture of high-resolution data but also capture the global contextual associations of low-resolution layers. The attention superposition strategy splices feature maps of different levels in the channel dimension to construct a high-dimensional feature matrix with joint spatial-feature expression. It can effectively improve the model's detection sensitivity for small target features in tasks such as farmland edge recognition and underwater terrain classification, and alleviate the problems of false detection and missed detection caused by the semantic gap of heterogeneous data.

[0092] According to the embodiments of this application, the multi-scale hierarchical attention fusion framework proposed based on the above image processing method has the core idea of ​​effectively integrating local and global information at different granularities. This framework can be further expanded to process more complex and multi-dimensional visual data formats.

[0093] According to an embodiment of the present application, the original image may also include a voxel image determined based on three-dimensional volume data. The three-dimensional volume data may include, but is not limited to, data of a three-dimensional volume formed by stacking multiple video frames along a time axis. The voxel image may include, but is not limited to, at least one of the following: three-dimensional scanning data used in industrial non-destructive testing, computed tomography (CT) and magnetic resonance imaging (MRI) used in medical imaging.

[0094] In the case where the original image is a voxel image, the above-mentioned image processing method can be performed as follows: multi-scale sampling is performed on the original voxel image to obtain a plurality of sampled voxel images of different scales. Each sampled voxel image is subjected to block processing to obtain a voxel block sequence of each sampled voxel image, wherein the scales of the voxel blocks of the sampled voxel images of different scales are the same. Self-attention calculation is performed on the voxel block sequence of each sampled voxel image to obtain a voxel enhanced feature sequence of each sampled voxel image. The voxel enhanced feature sequence of each sampled voxel image is converted into a voxel feature map of a preset scale to obtain a plurality of voxel feature maps of the preset scales. An enhanced voxel image is generated based on the plurality of voxel feature maps of the preset scales.

[0095] It should be noted that the above-mentioned process of processing the original voxel image to generate the enhanced voxel image has the same or corresponding implementation method as the process of processing the original image to generate the enhanced image in the aforementioned embodiment. Regarding the specific implementation method of processing the original voxel image to generate the enhanced voxel image in this embodiment, please refer to the aforementioned embodiment and will not be repeated here.

[0096] Through the above-mentioned embodiments of the present application, by performing multi-scale voxel sampling on the voxel image, generating voxel image blocks and calculating attention at the voxel level of different voxel resolutions, and finally realizing the fusion of multi-scale three-dimensional features, the accuracy of tasks such as lesion recognition and defect detection can be effectively improved.

[0097] When the original image is 3D volume data generated by stacking multiple video frames along the time axis, the 3D volume data may include not only the width, height, and channel information of the video frames, but also time information. The image processing method for this can be the same as the aforementioned method for processing the original voxel image to generate the enhanced voxel image, and will not be further described here.

[0098] Through the above-mentioned embodiments of the present application, by incorporating the time dimension into the consideration of layered sampling of video images, forming a spatiotemporal pyramid, and extracting video image blocks at different spatiotemporal scales and applying self-attention, the dynamic changes and temporal associations in the video can be effectively captured, and better applications can be achieved in fields such as behavior recognition and event detection.

[0099] Based on the above concept, this application proposes an innovative image processing method based on multi-scale hierarchical attention fusion, which aims to overcome the bottlenecks existing in existing technologies when processing multi-scale information such as images. The key points of its core technology are: First, by layering downsampling the original image, a multi-resolution feature pyramid is systematically constructed to capture the spatial information of the image at different granularities. Secondly, an innovative adaptive blocking strategy is introduced: for each layer of downsampled feature map, starting from the layer with the smallest scale (layer 1), it is divided into 4 layers. j Tokens of the same size. This design ensures that the tokens generated at different resolution levels have a uniform and fixed scale in the spatial dimension, providing standardized input for subsequent attention calculations. Next, the independent token sequence of each layer is input into the multi-head self-attention mechanism for processing, thereby effectively capturing long-distance dependencies and global context information within their respective resolution levels, and adaptively screening out highly discriminative features. Finally, all feature maps of each layer that have been processed by attention are upsampled to the resolution of the original image and deeply fused by pixel-level addition. This fusion method can organically combine fine details from different scales with macro-context information to form a comprehensive and robust image feature representation.

[0100] Through the above technical solutions, this application effectively solves the problems of information loss, insufficient context association and low fusion efficiency in multi-scale feature fusion of traditional methods, and significantly improves the feature representation ability of images and the ability to understand complex scenes.

[0101] According to the embodiments of the present application, considering the challenges of high data acquisition costs or incomplete data in practical applications, it is possible to explore combining the above-mentioned processing method of the present application with advanced machine learning paradigms to improve its generalization ability and robustness. For example, it can be combined with self-supervised learning (SSL) or contrastive learning (CL), using a large amount of unlabeled multi-scale data for pre-training. By designing cross-scale or hierarchical self-supervised tasks, such as predicting missing tokens and reconstructing high-resolution features, the algorithm learns powerful feature representations, and then fine-tunes with a small amount of labeled data to cope with small sample learning scenarios. In addition, it is also possible to explore combining with the knowledge distillation (KD) model to compress large and complex multi-scale fusion models into smaller and more efficient models for deployment on resource-constrained edge devices.

[0102] Figure 5 The figure shows a structural block diagram of an image processing device according to an embodiment of the present application.

[0103] like Figure 5 As shown, the image processing device 500 includes a sampling module 510, a sampling image blocking module 520, a sampling image self-attention calculation module 530, a sampling image conversion module 540 and an image enhancement module 550.

[0104] The sampling module 510 is used to perform multi-scale sampling on the original image to obtain a plurality of sampled images of different scales.

[0105] The sampled image blocking module 520 is configured to perform blocking processing on each sampled image to obtain a block sequence of each sampled image, wherein the blocks of sampled images of different scales have the same scale.

[0106] The sampled image self-attention calculation module 530 is used to perform self-attention calculation on the block sequence of each sampled image to obtain an enhanced feature sequence of each sampled image.

[0107] The sampling image conversion module 540 is used to convert the enhanced feature sequence of each sampling image into a feature map of a preset scale to obtain a plurality of feature maps of the preset scale.

[0108] The image enhancement module 550 is configured to generate an enhanced image based on a plurality of feature maps of preset scales.

[0109] According to an embodiment of the present application, the sampling module includes a downsampling submodule.

[0110] The downsampling submodule is used to downsample the original image layer by layer with a preset step size to obtain multi-layer sampled images, so that the sampled images of different layers have different scales.

[0111] According to an embodiment of the present application, the sampling image blocking module includes a blocking submodule and a blocking sequence generation submodule.

[0112] The blocking submodule is used to uniformly block the sampled image in a grid according to the number of layers where the sampled image is located, to obtain multiple blocks, so that the blocks of sampled images of different scales have the same scale.

[0113] The block sequence generation submodule is used to generate a block sequence of the sampled image according to a plurality of blocks.

[0114] According to an embodiment of the present application, the blocking submodule includes a blocking information determination unit and a blocking unit.

[0115] The block information determining unit is used to determine the number of blocks and the block scale of the sampled image according to the number of layers where the sampled image is located and the scale of the sampled image.

[0116] The blocking unit is used to uniformly grid-block the sampled image according to the number of blocks and the block scale to obtain multiple blocks, the scales of the multiple blocks are all the block scales, and the number of the multiple blocks is the block number.

[0117] According to an embodiment of the present application, the sampled image conversion module includes an initial conversion submodule and a resampling submodule.

[0118] The initial conversion submodule is used to convert the enhanced feature sequence of each sampled image into an initial feature map, wherein the scale of the initial feature map is the same as the scale of the sampled image used to obtain the initial feature map.

[0119] The resampling submodule is used to resample the initial feature map of each sampled image to obtain a feature map of a preset scale for each sampled image.

[0120] According to an embodiment of the present application, the image enhancement module includes a pixel-level addition submodule and a decoding submodule.

[0121] The pixel-level addition submodule is used to perform pixel-level addition on multiple feature maps of preset scales to obtain an enhanced feature map.

[0122] The decoding submodule is used to decode the enhanced feature map to obtain an enhanced image.

[0123] According to an embodiment of the present application, the image processing device also includes an original image segmentation module, an original image self-attention calculation module and an original image restoration module.

[0124] The original image block module is used to perform block processing on the original image to obtain a block sequence of the original image, wherein the scale of the blocks of the original image is the same as the scale of the blocks of each sampled image.

[0125] The original image self-attention calculation module is used to perform self-attention calculation on the block sequence of the original image to obtain the enhanced feature sequence of the original image.

[0126] The original image conversion module is used to convert the enhanced feature sequence of the original image into a feature map of a preset scale.

[0127] According to an embodiment of the present application, the pixel-level addition submodule includes a pixel-level addition unit.

[0128] The pixel-level adding unit is used to perform pixel-level addition on the feature map of the preset scale of the original image and the feature maps of the preset scales of multiple sampled images to obtain an enhanced feature map.

[0129] According to an embodiment of the present application, any multiple modules among the sampling module 510, the sampled image segmentation module 520, the sampled image self-attention calculation module 530, the sampled image conversion module 540, and the image enhancement module 550 can be combined into a single module, or any one of them can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present application, at least one of the sampling module 510, the sampled image segmentation module 520, the sampled image self-attention calculation module 530, the sampled image conversion module 540, and the image enhancement module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware by any other reasonable means of integrating or packaging circuits, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the sampling module 510, the sampling image segmentation module 520, the sampling image self-attention calculation module 530, the sampling image conversion module 540 and the image enhancement module 550 can be at least partially implemented as a computer program module, which can perform the corresponding function when it is executed.

[0130] Figure 6 A block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of the present application is shown.

[0131] like Figure 6 As shown, an electronic device 600 according to an embodiment of the present application includes a processor 601, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0132] Various programs and data required for the operation of the electronic device 600 are stored in the RAM 603. The processor 601, ROM 602, and RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than the ROM 602 and RAM 603. The processor 601 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0133] According to an embodiment of the present application, electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to bus 604. Electronic device 600 may also include one or more of the following components connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or modem. Communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. Removable media 611, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 610 as needed, so that computer programs read from the removable media can be installed into storage section 608 as needed.

[0134] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the image processing method according to the embodiments of this application.

[0135] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0136] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the image processing method provided in the embodiments of the present application.

[0137] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the computer program is executed by the processor 601. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0138] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0139] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0140] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0141] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Downsampling the original image layer by layer with a preset step size to obtain multi-layer sampled images, so that the sampled images of different layers have different scales, where the scale is the resolution; For each sampled image, the number of blocks and the block scale of the sampled image are determined according to the layer number of the sampled image and the scale of the sampled image; Performing uniform grid division of the sampled image according to the number of blocks and the block scale to obtain a plurality of blocks, so that the scales of the blocks of the sampled images of different scales are the same, and the scales of the plurality of blocks are all the block scales; Sequentially splicing multiple blocks obtained from each sampled image to obtain a block sequence of each sampled image; Using the same self-attention module, self-attention calculation is performed on the block sequence of each sampled image to obtain the enhanced feature sequence of each sampled image; Converting the enhanced feature sequence of each sampled image into a feature map of a preset scale to obtain a plurality of feature maps of the preset scale; Perform pixel-level addition on multiple feature maps of preset scales to obtain an enhanced feature map; The enhanced feature map is decoded to obtain an enhanced image.

2. The method according to claim 1, characterized in that The preset step size is 2 n , where n is a positive integer.

3. The method according to claim 1, characterized in that The converting of the enhanced feature sequence of each sampled image into a feature map of a preset scale comprises: Converting the enhanced feature sequence of each sampled image into an initial feature map, wherein the scale of the initial feature map is the same as the scale of the sampled image used to obtain the initial feature map; The initial feature map of each sampled image is resampled to obtain a feature map of a preset scale of each sampled image.

4. The method according to claim 1, wherein The preset scale is the scale of the original image.

5. The method according to claim 4, characterized in that The method further comprises: Performing block processing on the original image to obtain a block sequence of the original image, wherein the scale of the blocks of the original image is the same as the scale of the blocks of each sampled image; Perform self-attention calculation on the block sequence of the original image to obtain the enhanced feature sequence of the original image; Convert the enhanced feature sequence of the original image into a feature map of a preset scale; The performing pixel-level addition on the feature maps of the preset scale to obtain the enhanced feature map includes: performing pixel-level addition on the feature map of the preset scale of the original image and the feature maps of the preset scale of the multiple sampled images to obtain the enhanced feature map.

6. The method according to claim 1, characterized in that The original image includes a first image and a second image, where the first image is an image generated based on detection data of a first modality of the object, and the second image is an image generated based on detection data of a second modality of the object.

7. The method according to claim 1, characterized in that The self-attention calculation is a multi-head self-attention calculation.

8. The method according to claim 1, characterized in that The original image includes a voxel image determined based on three-dimensional volume data.

9. The method according to claim 8, characterized in that The three-dimensional volume data includes data of a three-dimensional volume formed by stacking multiple video frames along a time axis.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.

Images