Image processing method, related device, equipment and storage medium

Abstract

The invention discloses an image processing method, a related device, equipment and a storage medium. The method comprises the steps of performing division processing on a to-be-processed image according to n scale levels to obtain a window sequence corresponding to each scale level; performing feature embedding processing on each window sequence to obtain a window embedding sequence corresponding to each window sequence; for the jth window in the window sequence corresponding to the ith scale level, obtaining a multi-scale prefix set according to the window embedding sequence corresponding to the ith scale level; for the jth window in the window sequence corresponding to the ith scale level, attention weighting calculation is carried out on the multi-scale prefix set, and multi-scale attention features are obtained; and generating an image processing result according to the multi-scale attention feature corresponding to each window in each scale level. According to the method, the feature representation quality and the calculation efficiency can be improved, and the dependence of the model on a large amount of labeled data is reduced by utilizing multi-scale feature expression.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image processing method, related apparatus, device and storage medium. BACKGROUND

[0002] For a long time, convolutional neural network (CNN) dominates computer vision tasks, and visual geometry group network (VGG) and residual network (ResNet) architectures are particularly typical. With outstanding feature extraction capability and parameter sharing mechanism, CNN has achieved remarkable results in image classification, object detection and other tasks, but also has the defects of limited receptive field and weak global relationship modeling capability.

[0003] At present, a vision transformer (ViT) model is proposed, and its core idea is to regard an image as a sequence of visual words. That is, first, the input image is divided into fixed-size non-overlapping patches, then each patch is flattened into a one-dimensional vector and converted into an embedding vector through linear projection, and finally, the embedding vectors are encoded.

[0004] However, in the process of research and practice of the current technology, the inventors found that at least the following problems exist in the current scheme. Compared with CNN, the ViT model does not incorporate the image-specific inductive bias (for example, local correlation, spatial invariance), resulting in high dependence on training data. In the scene of limited data samples, the model has insufficient generalization ability, and it is difficult to adapt to various image data stably, which limits the application effect in low data volume scenes. SUMMARY

[0005] The embodiments of the present application provide an image processing method, related apparatus, device and storage medium, which can improve feature representation quality and computing efficiency while reducing the dependence of the model on a large amount of labeled data by using multi-scale feature expression, thereby effectively alleviating the problem of insufficient model generalization ability in low data volume scenes, and further improving the adaptability of the model to various image data.

[0006] Therefore, the present application provides, in one aspect, an image processing method, comprising:

[0007] The image to be processed is divided and processed according to n scale levels, to obtain a window sequence corresponding to each scale level, wherein the window sequence includes a plurality of windows belonging to the corresponding scale level, and n is an integer greater than 1;

[0008] The feature embedding processing is performed on each window sequence to obtain a window embedding sequence corresponding to each window sequence, wherein the window embedding sequence includes window embedding vectors corresponding to each window belonging to the corresponding scale level;

[0009] For the jth window in the window sequence corresponding to the ith scale level, a multi-scale prefix set is obtained according to the window embedding sequence corresponding to the ith scale level, wherein the multi-scale prefix set includes a prefix sequence corresponding to the jth window from the ith scale level to the nth scale level, and the prefix sequence includes window embedding vectors corresponding to windows before the jth window, i and j are integers greater than or equal to 1;

[0010] For the jth window in the window sequence corresponding to the ith scale level, the multi-scale prefix set is subjected to attention weighting calculation to obtain a multi-scale attention feature.

[0011] According to the multi-scale attention feature corresponding to each window in each scale level, an image processing result is generated.

[0012] Another aspect of the present application provides an image processing device, comprising:

[0013] The processing module is configured to divide the to-be-processed image according to n scale levels respectively to obtain a window sequence corresponding to each scale level, wherein the window sequence includes a plurality of windows belonging to the corresponding scale level, and n is an integer greater than 1;

[0014] The processing module is further configured to perform feature embedding processing on each window sequence to obtain a window embedding sequence corresponding to each window sequence, wherein the window embedding sequence includes window embedding vectors corresponding to each window belonging to the corresponding scale level;

[0015] The acquisition module is configured to, for the jth window in the window sequence corresponding to the ith scale level, obtain a multi-scale prefix set according to the window embedding sequence corresponding to the ith scale level, wherein the multi-scale prefix set includes a prefix sequence corresponding to the jth window from the ith scale level to the nth scale level, and the prefix sequence includes window embedding vectors corresponding to windows before the jth window, i and j are integers greater than or equal to 1;

[0016] The calculation module is configured to, for the jth window in the window sequence corresponding to the ith scale level, perform attention weighting calculation on the multi-scale prefix set to obtain a multi-scale attention feature.

[0017] The generation module is configured to generate an image processing result according to the multi-scale attention feature corresponding to each window in each scale level.

[0018] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0019] The processing module is specifically configured to determine n scale factors according to n scale levels, where each scale level corresponds to a scale factor.

[0020] n window sizes are determined according to the n scale factors, where each scale factor corresponds to a window size.

[0021] The image to be processed is windowed according to the n window sizes respectively, to obtain a window sequence corresponding to each scale level.

[0022] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0023] The processing module is specifically configured to, for the j th window in the window sequence corresponding to the i th scale level, perform embedding coding on pixel values corresponding to all pixel points in the window, to obtain a pixel projection vector.

[0024] For the j th window in the window sequence corresponding to the i th scale level, position coding is performed on pixel values corresponding to all pixel points in the window, to obtain a position coding vector.

[0025] For the j th window in the window sequence corresponding to the i th scale level, the pixel projection vector and the position coding vector corresponding to the window are added dimension by dimension, to obtain a window embedding vector.

[0026] The window embedding vectors corresponding to all windows in the same window sequence are taken as a window embedding sequence corresponding to the window sequence.

[0027] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0028] The processing module is specifically configured to, for the j th window in the window sequence corresponding to the i th scale level, generate a fusion basis vector corresponding to the window according to a scale level corresponding to the window and spatial position information of the window, where the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0029] For the j th window in the window sequence corresponding to the i th scale level, a position coding vector corresponding to the window is output based on the fusion basis vector corresponding to the window through a fully connected network.

[0030] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0031] The processing module is specifically configured to extract a vector corresponding to the jth row in the spatial coding matrix corresponding to the i th scale level as a spatial coding vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial coding matrix is a numerical matrix that can be optimized by model training parameters.

[0032] The processing module is specifically configured to extract a vector corresponding to the jth row in the spatial coding matrix corresponding to the i th scale level as a spatial coding vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial coding matrix is a numerical matrix that can be optimized by model training parameters.

[0033] The processing module is specifically configured to extract a vector corresponding to the jth row in the spatial coding matrix corresponding to the i th scale level as a spatial coding vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial coding matrix is a numerical matrix that can be optimized by model training parameters.

[0034] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0035] The processing module is specifically configured to perform sine-cosine encoding on spatial position information corresponding to the window to obtain a spatial sine-cosine vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0036] The processing module is specifically configured to perform sine-cosine encoding on spatial position information corresponding to the window to obtain a spatial sine-cosine vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0037] The processing module is specifically configured to perform sine-cosine encoding on spatial position information corresponding to the window to obtain a spatial sine-cosine vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0038] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0039] The processing module is specifically configured to perform sine-cosine encoding on spatial position information corresponding to the window to obtain a spatial sine-cosine vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0040] The processing module is specifically configured to perform sine-cosine encoding on spatial position information corresponding to the window to obtain a spatial sine-cosine vector for the jth window in the window sequence corresponding to the i th scale level, where the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0041] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0042] the acquisition module is specifically configured to, for the jth window in the window sequence corresponding to the ith scale level, in a case where (j-1) is less than or equal to the prefix truncation threshold, taking the window embedding vectors corresponding to the (j-1) windows as the prefix sequence, where the (j-1) windows are all the windows before the jth window in the window sequence;

[0043] for the jth window in the window sequence corresponding to the ith scale level, in a case where (j-1) is greater than the prefix truncation threshold, taking the window embedding vectors corresponding to the most recent τ windows before the jth window as the prefix sequence, where τ is an integer greater than or equal to 1, and τ represents the prefix truncation threshold;

[0044] for the jth window in the window sequence corresponding to the ith scale level, according to the prefix sequence corresponding to the window, taking the prefix sequences from the ith scale level to the n th scale level as the multi-scale prefix set.

[0045] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0046] The acquisition module is further configured to acquire a task estimated time consumption corresponding to the image processing task.

[0047] The processing module is further configured to, in a case where the task estimated time consumption is less than or equal to a delay threshold, take a first preset value as the prefix truncation threshold.

[0048] The processing module is further configured to, in a case where the task estimated time consumption is greater than the delay threshold, take a second preset value as the prefix truncation threshold, where the second preset value is greater than the first preset value.

[0049] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0050] The acquisition module is specifically configured to calculate a dynamic truncation threshold corresponding to the ith scale level according to the number of windows included in the window sequence corresponding to the ith scale level.

[0051] for the jth window in the window sequence corresponding to the ith scale level, in a case where (j-1) is less than or equal to the dynamic truncation threshold corresponding to the ith scale level, taking the window embedding vectors corresponding to the (j-1) windows as the prefix sequence, where the (j-1) windows are all the windows before the jth window in the window sequence;

[0052] for the jth window in the window sequence corresponding to the ith scale level, if (j-1) is greater than the dynamic truncation threshold corresponding to the ith scale level, a window embedding vector corresponding to the (j-Li)th window before the jth window is taken as the prefix sequence, where Li is an integer greater than or equal to 1, and Li represents the dynamic truncation threshold corresponding to the ith scale level;

[0053] for the jth window in the window sequence corresponding to the ith scale level, the prefix sequence from the ith scale level to the nth scale level is taken as the multi-scale prefix set according to the prefix sequence corresponding to the window.

[0054] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0055] The calculation module is specifically configured to, for the jth window in the window sequence corresponding to the ith scale level, construct a query vector corresponding to the window according to a window embedding vector corresponding to the window;

[0056] for the jth window in the window sequence corresponding to the ith scale level, construct a key matrix and a value matrix corresponding to the window according to the multi-scale prefix set corresponding to the window;

[0057] for the jth window in the window sequence corresponding to the ith scale level, perform attention calculation on the query vector, the key matrix and the value matrix corresponding to the window to obtain an attention feature;

[0058] for the jth window in the window sequence corresponding to the ith scale level, perform weighted calculation on the attention features of the window from the ith scale level to the nth scale level to obtain a multi-scale attention feature corresponding to the window.

[0059] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0060] The generation module is specifically configured to determine an image global feature according to the multi-scale attention feature corresponding to each window in each scale level;

[0061] obtain a category probability vector through a classifier based on the image global feature, where the category probability vector includes a probability value of the to-be-processed image belonging to each category;

[0062] determine a category with the maximum probability value in the category probability vector as the classification result of the to-be-processed image.

[0063] In a possible design, in another implementation manner of another aspect of the embodiment of the present application,

[0064] The generating module is specifically configured to determine, for the jth window in the window sequence corresponding to the ith scale level, a candidate box coordinate and a target probability vector corresponding to the window according to the multi-scale attention feature corresponding to the window, where the candidate box coordinate is used to indicate a candidate box.

[0065] The non-maximum suppression operation is performed on all the candidate boxes according to the candidate box coordinate and the target probability vector corresponding to each window in each scale level, to obtain a target detection result.

[0066] In a possible design, in another implementation manner of the other aspect of the embodiment of the present application,

[0067] The generating module is specifically configured to concatenate the multi-scale attention feature corresponding to each window in each scale level to obtain a multi-scale fusion feature.

[0068] The multi-scale fusion feature is up-sampled to obtain a target feature map, where the target feature map has the same resolution as the to-be-processed image.

[0069] Pixel-wise class prediction is performed on the target feature map to obtain a semantic segmentation result of the to-be-processed image.

[0070] The present application provides a computer device in another aspect, comprising a memory and a processor, the memory stores a computer program, and the processor implements the method of the above aspects when executing the computer program.

[0071] The present application provides a computer readable storage medium in another aspect, which stores a computer program, and the computer program is executed by a processor to implement the method of the above aspects.

[0072] The present application provides a computer program product in another aspect, comprising a computer program, and the computer program is executed by a processor to implement the method of the above aspects.

[0073] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0074] In the embodiments of the present application, a method for image processing is provided. First, the to-be-processed image is divided according to a plurality of scale levels to obtain a window sequence corresponding to each scale level. Then, feature embedding processing is performed on each window sequence to obtain a window embedding sequence corresponding to each window sequence. Based on this, a multi-scale prefix set corresponding to each window under each scale level is constructed. Next, attention weighting calculation is performed according to the multi-scale prefix set to obtain a multi-scale attention feature corresponding to each window. Finally, based on the multi-scale attention feature corresponding to each window, an image processing result is generated.

[0075] In this way, on the one hand, by fusing window embedding vectors at different scale levels, the balance between local details and global context can be achieved, the problem of insufficient representation ability of a single scale level can be solved, and the richness of features can be improved. On the other hand, the current window is used for attention weighted calculation of the multi-scale prefix set, and the multi-scale prefix set only focuses on the windows (i.e., historical windows) that appear before the current window, so that the selection of historical information can be optimized, the interference of subsequent information can be avoided, and the feature representation quality and the calculation efficiency can be considered.

[0076] In summary, while improving the feature representation quality and the calculation efficiency, the multi-scale feature representation is used to reduce the dependence of the model on a large amount of labeled data. Thus, the problem of insufficient model generalization ability in a low-data scenario can be effectively alleviated, and the adaptability of the model to various image data can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 An example diagram applied to an image classification scenario in an embodiment of the present application;

[0078] Figure 2 An example diagram applied to a target detection scenario in an embodiment of the present application;

[0079] Figure 3 An example diagram applied to a semantic segmentation scenario in an embodiment of the present application;

[0080] Figure 4 An example diagram of an implementation environment of an image processing method in an embodiment of the present application;

[0081] Figure 5 Another example diagram of an implementation environment of an image processing method in an embodiment of the present application;

[0082] Figure 6 An example flow diagram of an image processing method in an embodiment of the present application;

[0083] Figure 7 An example diagram of implementing image processing based on a visual transformation model in an embodiment of the present application;

[0084] Figure 8 An example diagram of dividing an image based on multiple scale levels in an embodiment of the present application;

[0085] Figure 9 An example diagram of constructing a multi-scale prefix set in an embodiment of the present application;

[0086] Figure 10 Another example diagram of constructing a multi-scale prefix set in an embodiment of the present application;

[0087] Figure 11A structural schematic diagram of a visual transformation model in an embodiment of the present application;

[0088] Figure 12 A schematic diagram of an image processing apparatus in an embodiment of the present application;

[0089] Figure 13 A structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The embodiment of the present application provides a method for image processing, a related device, equipment and a storage medium, which can improve feature representation quality and computing efficiency, utilize multi-scale feature expression, reduce dependence of a model on a large amount of labeled data, thereby effectively alleviating the problem of insufficient model generalization ability in a low data volume scene, and further improving the adaptability of the model to various image data.

[0091] The terms "first", "second", "third", "fourth" and the like in the specification and claims of the present application and in the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily have to be described in a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented, for example, in an order other than that illustrated or described herein. In addition, the terms "include" and "correspond to" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0092] It can be understood that in the specific embodiments of the present application, the related data of the image to be processed and the like need to obtain user permission or consent when applied to specific products or technologies. That is, before collecting the related data of the user, the user can be prompted to collect his / her related data through a prompt interface, a prompt pop-up window or a prompt voice, and the like. Only after obtaining the user's permission or consent, the step of collecting the user's related data is started. In other words, all the user data collected by the present application is collected under the condition that the user agrees to authorize, and the collection, use and processing of the related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0093] With the rapid development of artificial intelligence (AI) and computer vision (CV) technologies, image processing techniques have become a core means for mining the deep value of visual data. Through precise extraction and intelligent analysis of image features, it can optimize the execution efficiency of tasks and improve the accuracy of decision-making. Currently, a ViT model is proposed, the core idea of which is to regard an image as a sequence of visual words. First, the input image is divided into fixed-size non-overlapping patches, then each patch is flattened into a one-dimensional vector, and finally the embedding vector is converted through linear projection. Finally, the embedding vectors are added to the position encoding to form a sequence, which is input to the encoder for processing.

[0094] The advantage of the ViT model is its strong global relationship modeling capability, which can capture the dependency between distant pixels in an image. However, the ViT model faces a trade-off between computational efficiency and representation ability when processing high-resolution images. The ViT model uses a fixed-size window to divide the image, which leads to a dilemma: a large window loses detailed information, and a small window causes the computational complexity to grow quadratically.

[0095] Based on this, in the embodiments of the present application, a multi-scale vision transformer (MS-ViT) model based on a multi-scale patch attention mechanism is proposed. By using multiple windows of different scale levels to divide the image at the same time, and realizing cross-scale information flow in the attention calculation process. That is, when calculating the attention of the current position, not only the context information of the same scale level is considered, but also the global information provided by the larger scale level window is included, so as to effectively balance the detailed features and global context information while maintaining the computational efficiency. Thus, it breaks the limitation of traditional single-scale window and provides a solution to the efficiency bottleneck of transformer models in visual tasks. Further, it expands the application possibility of transformer models in high-resolution visual tasks, provides a feasible solution for fields such as medical images that have high requirements on precision and efficiency, and promotes the development of visual and language multi-modal large models, supporting higher quality visual understanding and generation.

[0096] Before introducing the specific method of the present application, the application scenario of the present application is exemplarily described. It should be understood that the following application scenarios are only for illustration and are not limited to this.

[0097] (1) Image classification scenario;

[0098] In the image classification scenario, the core of the task is to extract the global semantic and overall layout features of the image, and the local irrelevant details are easy to interfere with the classification result. Based on the method provided in the application, a large scale factor can be configured to focus on the aggregation of global features, thereby reducing the calculation cost while enhancing the accuracy of the classification judgment.

[0099] Illustratively, taking the example of distinguishing whether an image is an indoor environment (e.g., a living room, an office, etc.) or an outdoor environment (e.g., a park, a street, etc.), 3 scale levels can be configured, and a set of scale factors is defined as {16, 32, 64}, that is, each corresponding window size is 16x16, 32x32, and 64x64. Among them, the global features of the image are captured based on the larger window size, and the key components of the image are captured based on the smaller window size.

[0100] For ease of understanding, please refer to Figure 1 , Figure 1 is an example of an image classification scenario applied in the embodiments of the application, as shown in Figure 1 (A) figure, based on the larger window size, the spatial relationship between the wall and the ceiling in the image is captured, and based on the smaller window size, the sofa in the image is captured, thereby it can be identified as an indoor environment. As shown in Figure 1 (B) figure, based on the larger window size, the spatial relationship between the sky and the ground in the image is captured, and based on the smaller window size, the trees in the image are captured, thereby it can be identified as an outdoor environment.

[0101] (2) target detection scenario;

[0102] In the target detection scenario, the core of the task is to achieve accurate positioning of small targets. Based on the method provided in the application, windows across scale levels are configured to cover the target details and global structural features, thereby providing reliable support for high-precision target detection.

[0103] Illustratively, taking the example of lesion detection on medical images, 9 scale levels can be configured, and a set of scale factors is defined as {1, 2, 4, 8, 16, 32, 64, 128, 256}, that is, each corresponding window size is 1x1, 2x2, 4x4, 8x8, 16x16, 32x32, 64x64, 128x128, and 256x256. Among them, the spatial relationship of the anatomical structure is captured based on the larger window size, the small nodule area is focused on based on the smaller window size, and the density difference and edge morphology of the lesion are captured. The intermediate window size is used to realize the association between the lesion and the anatomical structure.

[0104] For ease of understanding, please refer to Figure 2 , Figure 2For one of the application examples applied to the target detection scene, as shown in Figure 2 Fig. (A) in the middle (A) is a medical image of the lung. After target detection is performed on the medical image, the target detection result as shown in Fig. (B) in the middle (B) is obtained. Wherein, 201 is used to indicate the position of one lesion, and 202 is used to indicate the position of another lesion. Figure 2

[0105] (3) Semantic segmentation scene;

[0106] In the semantic segmentation scene, accurate pixel-level classification highly depends on the capture of subtle texture features. Based on the method provided in the present application, multiple scale levels corresponding windows are configured, the small size windows are used to mine the detailed features in the image, and the large size windows are used to provide global semantic constraints, so as to realize the collaborative optimization of detailed and global features.

[0107] Exemplarily, taking the task of segmenting a bird image from an image as an example, 6 scale levels can be configured, and a set of scale factors is defined as {4, 8, 16, 32, 64, 128}, that is, each corresponding window size is 4x4, 8x8, 16x16, 32x32, 64x64 and 128x128. Among them, the overall shape and posture of the bird are captured based on the larger window size, which provides global semantic constraints for detailed segmentation, and the detailed features such as the stripe distribution of the feather, the boundary of the iris and the pupil, etc. are focused based on the smaller window size.

[0108] For ease of understanding, please refer to Figure 3 , Figure 3 For one of the application examples applied to the semantic segmentation scene, as shown in Figure 3 Fig. (A) in the middle (A) is an original image. After semantic segmentation is performed on the original image, the semantic segmentation result as shown in Fig. (B) in the middle (B) is obtained, wherein all the white pixel points are used to constitute the bird area, and all the black pixel points are used to constitute the non-bird area. Figure 3

[0109] It should be noted that the above application scenarios are only examples, and the image processing method provided in the present embodiment can also be applied to various scenes, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, etc.

[0110] The method provided in the present application can be applied to different implementation environments, which will be introduced respectively in combination with the drawings.

[0111] Implementation environment one;

[0112] The method provided in the present application can be applied to Figure 4 ​​The illustrated implementation environment includes at least the terminal 401. The terminal 401 involved in the present application includes but is not limited to a mobile phone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device, a virtual reality device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc. The client is deployed on the terminal 401, and the client can run on the terminal 401 in the form of a browser, or run on the terminal 401 in the form of an independent application (applicant, APP) or a mini-program, etc.

[0113] In combination with the above implementation environment, in step A1, the terminal 401 performs division processing on the to-be-processed image according to n scale levels respectively, to obtain a window sequence corresponding to each scale level. In step A2, the terminal 401 performs feature embedding processing on each window sequence, to obtain a window embedding sequence corresponding to each window sequence. In step A3, the terminal 401 obtains a multi-scale prefix set corresponding to each window under each scale level. In step A4, the terminal 401 performs attention weighting calculation based on the multi-scale prefix set corresponding to each window under each scale level, to obtain a multi-scale attention feature. In step A5, the terminal 401 generates an image processing result according to the multi-scale attention feature corresponding to each window in each scale level.

[0114] Implementation environment two;

[0115] The method provided by the present application can be applied to Figure 5 The illustrated implementation environment includes the terminal 501 and the server 502, and the terminal 501 and the server 502 can communicate with each other through the network 503. The network 503 uses standard communication technologies and / or protocols, and is usually the Internet, but can also be any network, including but not limited to any combination of Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, a private network, or a virtual private network. In some embodiments, the above-mentioned data communication technologies can be replaced or supplemented by customized or dedicated data communication technologies.

[0116] The terminal 501 can refer to the description of the terminal 401 in the foregoing embodiments, and details are not described herein.

[0117] The server 502 involved in the present application can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and AI platforms. When the number of servers is multiple, the multiple servers can be composed into a blockchain, and the servers are nodes on the blockchain. The data involved in the present application can be saved on the blockchain, for example, scale levels, model parameters, etc.

[0118] In combination with the above implementation environment, in step B1, the terminal 501 responds to the upload instruction for the to-be-processed image. In step B2, the terminal 501 sends the to-be-processed image to the server 502 through the network 503. In step B3, the server 502 performs division processing on the to-be-processed image according to n scale levels, to obtain a window sequence corresponding to each scale level. In step B4, the server 502 performs feature embedding processing on each window sequence, to obtain a window embedding sequence corresponding to each window sequence. In step B5, the server 502 obtains a multi-scale prefix set corresponding to each window under each scale level. In step B6, the server 502 performs attention weighting calculation based on the multi-scale prefix set corresponding to each window under each scale level, to obtain a multi-scale attention feature. In step B7, the server 502 generates an image processing result according to the multi-scale attention feature corresponding to each window in each scale level. In step B8, the server 502 sends the image processing result to the terminal 501 through the network 503. In step B9, the terminal 501 displays the image processing result.

[0119] In combination with the above introduction, the image processing method in the present application will be introduced as follows, please refer to Figure 6 The image processing method in the embodiment of the present application can be independently completed by the terminal, independently completed by the server, or completed by the terminal and the server in cooperation. The method of the present application includes the following steps.

[0120] S601, performing division processing on the to-be-processed image according to n scale levels, to obtain a window sequence corresponding to each scale level, wherein the window sequence includes multiple windows belonging to the corresponding scale level, and n is an integer greater than 1;

[0121] In one or more embodiments, first, a to-be-processed image is obtained. Then, division processing is performed on the to-be-processed image according to n scale levels, thereby obtaining n window sequences. Each window sequence includes multiple windows belonging to the corresponding scale level.

[0122] Specifically, n is an integer greater than 1, that is, the image to be processed needs to be divided and processed according to multiple different scale levels, each scale level corresponds to a window sequence, and each window sequence includes windows arranged in a certain order (for example, from left to right, and then from top to bottom). Wherein the window sequence corresponding to the i-th scale level is denoted as , and the j-th window in the window sequence corresponding to the i-th scale level is denoted as .

[0123] S602, performing feature embedding processing on each window sequence to obtain a window embedding sequence corresponding to each window sequence, wherein the window embedding sequence includes window embedding vectors corresponding to each window belonging to the corresponding scale level;

[0124] In one or more embodiments, each window in each window sequence needs to be processed for feature embedding respectively, thereby obtaining a window embedding vector corresponding to each window.

[0125] Specifically, taking the j-th window in the window sequence corresponding to the i-th scale level as an example, on the one hand, a linear transformation needs to be performed on all pixel points in the window to obtain a pixel projection vector. On the other hand, a position encoding needs to be performed on the window to obtain a position encoding vector. It should be noted that the pixel projection vector and the position encoding vector have the same dimension, so the window embedding vector can be obtained by adding dimension by dimension. Wherein the pixel projection vector of the j-th window in the window sequence corresponding to the i-th scale level is denoted as , the position encoding vector is denoted as , and the window embedding vector is denoted as . The window embedding vectors of all windows under the same scale level constitute a window embedding sequence corresponding to the window sequence, denoted as .

[0126] S603, for the j-th window in the window sequence corresponding to the i-th scale level, according to the window embedding sequence corresponding to the i-th scale level, a multi-scale prefix set is obtained, wherein the multi-scale prefix set includes a prefix sequence corresponding to the j-th window from the i-th scale level to the n-th scale level, and the prefix sequence includes window embedding vectors corresponding to windows before the j-th window, i and j are integers greater than or equal to 1;

[0127] In one or more embodiments, a corresponding prefix sequence is constructed for each window in each window sequence. Thus, based on the prefix sequence corresponding to the window, a corresponding multi-scale prefix set is constructed.

[0128] Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, the prefix sequence only contains the window embedding vectors corresponding to the windows before the jth window in the ith scale level. The prefix sequence for the jth window from the ith scale level to the nth scale level constitutes a multi-scale prefix set. Wherein, the prefix sequence of the jth window in the window sequence corresponding to the ith scale level is denoted as , and the multi-scale prefix set is denoted as .

[0129] S604, for the jth window in the window sequence corresponding to the ith scale level, attention weighted calculation is performed on the multi-scale prefix set to obtain a multi-scale attention feature;

[0130] In one or more embodiments, attention weighted calculation is performed on each window in each window sequence, that is, attention calculation is performed on the multi-scale prefix set corresponding to each window to obtain an attention feature. Then, weighted calculation is performed on the attention features in the corresponding scale level range of each window to obtain a multi-scale attention feature.

[0131] Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, attention calculation is performed on the multi-scale prefix set corresponding to the jth window based on an attention mechanism to obtain an attention feature. The attention mechanism refers to a computer mechanism in a neural network that allows the model to selectively focus on a specific part of the input data, which is realized by the interaction of a query vector, a key matrix, and a value matrix in an encoder. Weighted calculation is performed on the attention features in the corresponding scale level range (i.e., from the ith scale level to the nth scale level) of each window to obtain a multi-scale attention feature. Wherein, the attention feature of the jth window in the window sequence corresponding to the ith scale level is denoted as , and the multi-scale attention feature is denoted as .

[0132] S605, according to the multi-scale attention feature corresponding to each window in each scale level, an image processing result is generated.

[0133] In one or more embodiments, based on the multi-scale attention feature corresponding to each window in each scale level, an image processing result of a corresponding task is output.

[0134] Exemplarily, in an image classification task, high-precision image category recognition can be provided, which is suitable for scenarios that require attention to both global semantics and local details.

[0135] Exemplarily, in the target detection and segmentation task, different scales of targets can be detected simultaneously through the multi-scale feature representation, which is suitable for object detection and fine segmentation in complex scenes.

[0136] Exemplarily, in the image generation and editing task, the autoregressive (i.e., a process of predicting the next element of a sequence based on the generated part of the sequence, in the present application, it refers to processing the window sequence of each scale level) characteristic is utilized to support high-quality image generation and editing applications, such as image completion, style transfer, etc.

[0137] Exemplarily, in the video understanding task, by extending to the spatiotemporal dimension, video analysis and understanding tasks such as action recognition, video summarization, etc. are supported.

[0138] Specifically, for ease of understanding, please refer to Figure 7 , Figure 7 For an example of implementing image processing based on the visual transformation model in the embodiments of the present application, as shown in the figure, the MS-ViT model provided by the present application includes an image preprocessing module, a multi-scale block editor, a multi-scale attention calculation module, a multi-scale information fusion module, and a task-specific output layer. The image preprocessing module is used for performing size normalization, pixel value standardization, noise removal, and other preprocessing on the image to be processed. The multi-scale block editor includes n block units, and each block unit is used to block sample the image to be processed according to the corresponding scale level to generate the corresponding window sequence.

[0139] The multi-scale attention calculation module includes a same-scale attention unit, a cross-scale attention unit, and a prefix management unit. The same-scale attention unit is used to perform attention calculation on the windows of the same scale level to capture the local correlation information of the same scale level. The cross-scale attention unit is used to perform attention calculation on the windows of different scale levels to fuse local details and global context. The prefix management unit is used to construct a corresponding prefix sequence for each window.

[0140] The multi-scale information fusion module is used to perform weighted calculation on the attention features of each window in the corresponding scale level range to obtain multi-scale attention features with multi-scale information. The task-specific output layer is used to decode the multi-scale attention features according to the downstream task (e.g., target detection, image classification) to output the final image processing result.

[0141] In the embodiments of the present application, a method for image processing is provided. In the above manner, on the one hand, by fusing window embedding vectors at different scale levels, the balance between local details and global context can be achieved, the problem of insufficient representation ability of a single scale level can be solved, and thus the richness of features can be improved. On the other hand, the multi-scale prefix set only focuses on the windows (i.e., historical windows) that appear before the current window, and thus the selection of historical information can be optimized, the interference of subsequent information can be avoided, and the feature representation quality and the calculation efficiency can be considered.

[0142] The MS-ViT model provided in the present application performs the above steps S601 to S605. Based on this, in the scenario of limited data, the MS-ViT model can exhibit stronger generalization ability. That is, the adaptability to different resolution inputs is better, the resolution mismatch problem is alleviated, and compared with the ViT model, the transfer learning performance of the MS-ViT model is improved by about 10%, and less data is required to adapt to new tasks.

[0143] Optionally, in the above Figure 6 Based on one or more embodiments corresponding thereto, in another optional embodiment of the embodiments of the present application, the to-be-processed image is divided according to n scale levels respectively to obtain a window sequence corresponding to each scale level, which can specifically include:

[0144] n scale factors are determined according to the n scale levels, wherein each scale level corresponds to a scale factor;

[0145] n window sizes are determined according to the n scale factors, wherein each scale factor corresponds to a window size;

[0146] The to-be-processed image is divided into windows according to the n window sizes respectively to obtain a window sequence corresponding to each scale level.

[0147] In one or more embodiments, a way of dividing the to-be-processed image according to different scale levels is introduced. As known from the foregoing embodiments, the present application uses a multi-scale partitioning mechanism to divide the to-be-processed image, for example, 3 to 10 window sizes of different sizes are used to cover the to-be-processed image from fine granularity to coarse granularity.

[0148] It should be noted that in the present application, multi-scale partitioning refers to using windows of different sizes to divide the same image to form representations of multiple resolutions.

[0149] Specifically, let the to-be-processed image be denoted as wherein, represents the height of the to-be-processed image, denotes the width of the image to be processed, denotes the number of channels of the image to be processed. A set of scale factors is defined, denoted as , and, . Wherein n denotes the total number of scale levels, denotes the scale factor corresponding to the nth scale level. For each scale factor, the image to be processed is divided into windows (usually non-overlapping windows) of size to form a grid, and the windows in the grid are sorted to obtain a window sequence. Wherein, denotes the scale factor corresponding to the ith scale level.

[0150] For ease of understanding, please refer to Figure 8 , Figure 8 is an example of dividing an image based on multiple scale levels in the embodiments of the present application, as shown in the figure, taking the size of the image to be processed as 4x4 as an example, the image to be processed is divided based on 3 scale levels, and the scale factors corresponding to the 3 scale levels are . Based on this, the image to be processed is divided into a window sequence including 16 windows with a window size of 1x1, a window sequence including 4 windows with a window size of 2x2, and a window sequence including 1 window with a window size of 4x4.

[0151] It can be seen that for each scale level, the number of windows in the corresponding window sequence can be calculated as follows:

[0152] Equation (1)

[0153] Wherein, denotes the number of windows corresponding to the ith scale level. denotes the ceiling function. denotes the height of the image to be processed, denotes the width of the image to be processed. denotes the scale factor corresponding to the ith scale level.

[0154] It should be noted that in the present application, the window refers to the basic unit of dividing the image. In actual application, increasing the number of scale levels can improve the representation ability of the image, but the amount of calculation and memory occupation will also increase. Reducing the number of scale levels can reduce the representation ability of the image, but the amount of calculation will be reduced, thereby improving the processing speed. Therefore, a reasonable number of scale levels can be used according to the actual task, which is not limited here.

[0155] Secondly, in the embodiments of the present application, a way of dividing the to-be-processed image according to different scale levels is provided. Through the above way, the to-be-processed image is divided by using different window sizes, and image features of different granularities can be obtained, that is, a small-size window can capture local details in the image, and a large-size window can effectively cover global information of the image, so that the MS-ViT model can capture fine-grained local features and global semantic information at the same time. In a complex scene with multi-scale targets, compared with the ViT model, the detection accuracy of the MS-ViT model can be improved by 15% to 20%, thereby being beneficial to improving the image detail preservation and reconstruction quality.

[0156] Optionally, in the above Figure 6 On the basis of one or more embodiments, in another optional embodiment provided by the embodiments of the present application, the feature embedding processing is performed on each window sequence to obtain a window embedding sequence corresponding to each window sequence, which can specifically include:

[0157] For the jth window in the window sequence corresponding to the ith scale level, the pixel values corresponding to all pixel points in the window are embedded and encoded to obtain a pixel projection vector;

[0158] For the jth window in the window sequence corresponding to the ith scale level, the pixel values corresponding to all pixel points in the window are positionally encoded to obtain a position encoding vector;

[0159] For the jth window in the window sequence corresponding to the ith scale level, the pixel projection vector and the position encoding vector corresponding to the window are added dimension by dimension to obtain a window embedding vector;

[0160] The window embedding vectors corresponding to all windows in the same window sequence are taken as the window embedding sequence corresponding to the window sequence.

[0161] In one or more embodiments, a way of constructing a window embedding vector is introduced. As known from the foregoing embodiments, for each window under each scale level, feature embedding processing is respectively performed, and thus the window embedding vector corresponding to each window is obtained. The window embedding vectors corresponding to all windows under the ith scale level are arranged into a window embedding sequence according to a predefined spatial order (for example, from left to right, from top to bottom), that is, wherein, represents the window embedding vector corresponding to the jth window under the ith scale level.

[0162] ​Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, on the one hand, the pixel values corresponding to all the pixel points in the window are mapped from a pixel space to a fixed dimension (for example, 128 dimensions) embedding space through a linear transformation (for example, matrix multiplication), to obtain a pixel projection vector corresponding to the window, denoted as .

[0163] On the other hand, a position identifier needs to be added to the window, that is, the scale level to which the window belongs and the spatial position of the window in the image to be processed are marked, to output a position encoding vector with the same dimension as the pixel projection vector, denoted as .

[0164] Based on this, the window embedding vector corresponding to each window can be calculated in the following manner:

[0165] ; formula (2)

[0166] wherein, represents the window embedding vector corresponding to the jth window under the ith scale level. represents the jth window under the ith scale level. represents the pixel projection vector corresponding to the jth window under the ith scale level. represents the position encoding vector corresponding to the jth window under the ith scale level.

[0167] It can be seen that adding the pixel projection vector and the position encoding vector of the same window dimension by dimension not only retains the visual features of the window, but also retains the scale level and spatial position information of the window.

[0168] Secondly, in the embodiment of the application, a way of constructing a window embedding vector is provided. Through the above-mentioned way, the windows under multiple scale levels are encoded, the visual features and position features of the windows are fused, and different windows are given unique feature identifiers. Therefore, the feature discrimination degree can be improved, the modeling capability of the MS-ViT model for the spatial structure of the image can be strengthened, and thus the efficiency and accuracy of task processing can be improved.

[0169] Optionally, in the above Figure 6 On the basis of one or more embodiments corresponding to the above, in another optional embodiment provided by the embodiment of the application, for the jth window in the window sequence corresponding to the ith scale level, the pixel values corresponding to all the pixel points in the window are position encoded to obtain a position encoding vector, which can specifically include:

[0170] For the jth window in the window sequence corresponding to the ith scale level, a fusion base vector corresponding to the window is generated according to the scale level corresponding to the window and spatial position information of the window, wherein the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0171] For the jth window in the window sequence corresponding to the ith scale level, a fusion base vector corresponding to the window is generated according to the scale level corresponding to the window and spatial position information of the window, wherein the spatial position information represents two-dimensional coordinates of the window in the image to be processed.

[0172] In one or more embodiments, a way of constructing a position encoding vector is introduced. As known from the foregoing embodiments, for each window under each scale level, a corresponding position encoding vector needs to be constructed.

[0173] Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, the scale level corresponding to the window is denoted as i, and the spatial position information is denoted as (x, y), wherein x represents the number of rows of the window in the grid under the corresponding scale level, and y represents the number of columns of the window in the grid under the corresponding scale level. Based on this, the scale level and the spatial position information are spliced to obtain a fusion base vector, denoted as V, and V = [i, x, y].

[0174] Next, the fusion base vector is input into a fully connected network, and a corresponding position encoding vector is output through the fully connected network, and the position encoding vector and the pixel projection vector have the same dimension, for example, both are 128 dimensions. That is, the fully connected network can calculate the position encoding vector in the following way:

[0175] Equation (3)

[0176] wherein, represents the position encoding vector corresponding to the jth window under the ith scale level. represents a weight matrix of the fully connected network. represents the fusion base vector. represents a bias vector.

[0177] Again, in the embodiments of the present application, a way of constructing a position encoding vector is provided. Through the above way, the scale level corresponding to the window and the spatial position information can be directly linearly transformed based on the fully connected network. Therefore, not only has high computational efficiency, but also because the weights of the fully connected network can be trained and optimized, it can adapt to the feature requirements of different tasks, thereby improving the effect of feature encoding.

[0178] Optionally, in the above Figure 6According to one or more embodiments, another optional embodiment provided by the embodiment of the present application is that, for the jth window in the window sequence corresponding to the ith scale level, the pixel values corresponding to all pixel points in the window are positionally encoded to obtain a position encoding vector, which can specifically include:

[0179] For the jth window in the window sequence corresponding to the ith scale level, a vector corresponding to the jth row in the spatial encoding matrix corresponding to the ith scale level is extracted as a spatial encoding vector, wherein the spatial encoding matrix is a numerical matrix whose parameters can be optimized through model training.

[0180] For the jth window in the window sequence corresponding to the ith scale level, a scale encoding vector corresponding to the ith scale level is obtained, wherein the scale encoding vector is a numerical vector whose parameters can be optimized through model training.

[0181] For the jth window in the window sequence corresponding to the ith scale level, a position encoding vector corresponding to the window is generated according to the spatial encoding vector and the scale encoding vector corresponding to the window.

[0182] In one or more embodiments, another way of constructing a position encoding vector is introduced. As known from the foregoing embodiments, for each window under each scale level, a corresponding position encoding vector needs to be constructed. In order to facilitate understanding, the following will take 3 scale levels, scale factors {4, 8, 16}, and the size of the image to be processed as 256x256 as an example for introduction, that is, the scale factor corresponding to the scale level 1 is 4, the scale factor corresponding to the scale level 2 is 8, and the scale factor corresponding to the scale level 3 is 16.

[0183] Specifically, when the scale factor is equal to 4, the image to be processed is divided into 4096 windows, and a learnable spatial encoding matrix is configured for this scale level, denoted as When the scale factor is equal to 8, the image to be processed is divided into 1024 windows, and a learnable spatial encoding matrix is configured for this scale level, denoted as When the scale factor is equal to 16, the image to be processed is divided into 256 windows, and a learnable spatial encoding matrix is configured for this scale level, denoted as .

[0184] In addition, a corresponding scale encoding vector is configured for each scale level, that is, the scale encoding vector configured for the scale level 1 is denoted as , the scale encoding vector configured for the scale level 2 is denoted as , and the scale encoding vector configured for the scale level 3 is denoted as . Wherein d represents the dimension of the position encoding vector, for example, 128.

[0185] Based on this, taking the jth window in the window sequence corresponding to the ith scale level as an example, the vector corresponding to the jth row in the spatial encoding matrix corresponding to the ith scale level is extracted as the spatial encoding vector corresponding to the window. And the scale encoding vector corresponding to the ith scale level is called. Finally, the position encoding vector is calculated in the following way:

[0186] Equation (4)

[0187] Wherein, represents the position encoding vector corresponding to the jth window under the ith scale level. represents the spatial encoding vector corresponding to the jth window under the ith scale level. represents the scale encoding vector corresponding to the ith scale level.

[0188] It should be noted that the spatial encoding matrix and the scale encoding vector involved in the present application can be optimized through MS-ViT model training.

[0189] Again, in the embodiments of the present application, another way of constructing the position encoding vector is provided. Through the above-mentioned way, the scale level and the spatial position information are respectively encoded by learnable parameters, which can better represent their respective features. Based on this, the fusion of scale level and spatial position information can be realized through vector addition operation, which has low computational complexity, thereby facilitating the improvement of the inference efficiency of the MS-ViT model.

[0190] Optionally, in the above Figure 6 Based on one or more embodiments corresponding thereto, in another optional embodiment provided by the embodiments of the present application, for the jth window in the window sequence corresponding to the ith scale level, the pixel values corresponding to all pixel points in the window are position encoded to obtain a position encoding vector, which can specifically include:

[0191] For the jth window in the window sequence corresponding to the ith scale level, the spatial position information corresponding to the window is sine-cosine encoded to obtain a spatial sine-cosine vector, wherein the spatial position information represents the two-dimensional coordinates of the window in the image to be processed.

[0192] For the jth window in the window sequence corresponding to the ith scale level, the scale level corresponding to the window is sine-cosine encoded to obtain a scale sine-cosine vector.

[0193] For the jth window in the window sequence corresponding to the ith scale level, the position encoding vector corresponding to the window is generated according to the spatial sine-cosine vector and the scale sine-cosine vector corresponding to the window.

[0194] In one or more embodiments, another way of constructing the position encoding vector is introduced. As known from the foregoing embodiments, for each window at each scale level, a corresponding position encoding vector needs to be constructed. Wherein, the scale level corresponding to each window is denoted as i, and the spatial position information is denoted as (x, y), x represents the number of rows of the window in the grid at the corresponding scale level, and y represents the number of columns of the window in the grid at the corresponding scale level.

[0195] Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, the spatial position information or the scale level can be sine-cosine encoded in the following way:

[0196] Equation (5)

[0197] Equation (6)

[0198] wherein, represents the input identifier, i.e., the spatial position information or the scale level. represents the dimension of the position encoding vector, for example, 128. represents the dimension index.

[0199] Exemplarily, for the jth window at the ith scale level as spatial position information, the x value in the spatial position information is substituted into in Equation (5), and the y value in the spatial position information is substituted into in Equation (6), thereby obtaining the sine value corresponding to the even dimension index (for example, 0, 2, …, 126), and the cosine value corresponding to the odd dimension index (for example, 1, 3, …, 127). After splicing the respective sine values and cosine values, the spatial sine-cosine vector is obtained, denoted as .

[0200] Exemplarily, for the jth window at the ith scale level as spatial position information, the scale level is substituted into in Equation (5) and in Equation (6), thereby obtaining the cosine value corresponding to the odd dimension index (for example, 1, 3, …, 127). After splicing the respective sine values and cosine values, the scale sine-cosine vector is obtained, denoted as .

[0201] Finally, the position encoding vector is calculated in the following way:

[0202] Equation (7)

[0203] wherein, denotes a position encoding vector corresponding to the jth window at the ith scale level. denotes a spatial sine-cosine vector corresponding to the jth window at the ith scale level. denotes a scale sine-cosine vector corresponding to the ith scale level.

[0204] Again, in the embodiments of the present application, another way of constructing a position encoding vector is provided. Through the above-mentioned way, the sine-cosine rule is adapted to any scale level and spatial position information, and therefore has strong generalization and stability.

[0205] Optionally, in the above-mentioned Figure 6 On the basis of one or more embodiments, in another optional embodiment provided by the embodiments of the present application, for the jth window in the window sequence corresponding to the ith scale level, a multi-scale prefix set is obtained according to the window embedding sequence corresponding to the ith scale level, which can specifically include:

[0206] For the jth window in the window sequence corresponding to the ith scale level, the window embedding vectors of the (j-1) windows are taken as the prefix sequence, wherein the (j-1) windows are all the windows before the jth window in the window sequence.

[0207] For the jth window in the window sequence corresponding to the ith scale level, the prefix sequence from the ith scale level to the nth scale level is taken as the multi-scale prefix set according to the prefix sequence corresponding to the window.

[0208] In one or more embodiments, a way of constructing a multi-scale prefix set is introduced. As known from the foregoing embodiments, in the causal attention calculation, the historical sequence visible to the current window is the prefix sequence, and each window in the present application has a corresponding prefix sequence. Among them, the causal attention is a mechanism for limiting the calculation range of attention, and each window can only focus on the window before it, ensuring the unidirectionality of information flow, which can be used for autoregressive models.

[0209] Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, the prefix sequence corresponding to the window can be constructed in the following way, that is:

[0210] ; formula (8)

[0211] wherein, denotes a prefix sequence corresponding to the jth window at the ith scale level. denotes a window embedding vector corresponding to the first window at the ith scale level. denotes the window embedding vector corresponding to the (j-1)-th window at the i-th scale level.

[0212] Based on this, the following method can be used to construct the multi-scale prefix set corresponding to the window, that is:

[0213] Equation (9)

[0214] wherein, denotes the multi-scale prefix set corresponding to the j-th window at the i-th scale level. denotes the prefix sequence corresponding to the j-th window at the i-th scale level. denotes the prefix sequence corresponding to the j-th window at the n-th scale level.

[0215] It can be seen that the multi-scale prefix set contains the prefix sequence of the i-th scale level and all larger scale levels, thereby enabling the small scale window to access the global context information provided by the larger scale window. That is, the effective receptive field of the model is expanded through the multi-scale prefix mechanism, so that each scale level considers not only the historical information of the current scale, but also the historical information of all larger scale windows when calculating attention.

[0216] Secondly, in the embodiments of the present application, a method for constructing a multi-scale prefix set is provided. By the above method, at the same scale level, the window embedding vectors corresponding to all historical windows are taken as the prefix sequence, which can maximize the preservation of the time sequence correlation information, thereby enabling the MS-ViT model to capture long time sequence dependencies more accurately.

[0217] Optionally, in the above Figure 6 Based on one or more embodiments corresponding thereto, in another optional embodiment provided by the embodiments of the present application, for the j-th window in the window sequence corresponding to the i-th scale level, a multi-scale prefix set is obtained according to the window embedding sequence corresponding to the i-th scale level, which can specifically include:

[0218] For the j-th window in the window sequence corresponding to the i-th scale level, in the case that (j-1) is less than or equal to the prefix truncation threshold, the window embedding vectors corresponding to the (j-1) windows are taken as the prefix sequence, wherein the (j-1) windows are all the windows before the j-th window in the window sequence;

[0219] For the j-th window in the window sequence corresponding to the i-th scale level, in the case that (j-1) is greater than the prefix truncation threshold, the window embedding vectors corresponding to the most recent τ windows before the j-th window are taken as the prefix sequence, wherein τ is an integer greater than or equal to 1, and τ represents the prefix truncation threshold;

[0220] For the j-th window in the window sequence corresponding to the i-th scale level, the prefix sequence from the i-th scale level to the n-th scale level is used as a multi-scale prefix set based on the prefix sequence corresponding to the window.

[0221] In one or more embodiments, another method for constructing a multi-scale prefix set is introduced. As can be seen from the foregoing embodiments, the window sequence corresponding to the i-th scale level is denoted as... ,and, , This represents the number of windows at the i-th scale level. When calculating the attention features for the j-th window in the window sequence, the causality principle is followed, meaning only windows preceding the j-th window are considered.

[0222] Specifically, taking the division of the image to be processed based on three scale levels as an example, for easier understanding, please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram illustrating the construction of a multi-scale prefix set in an embodiment of this application. As shown in the figure, for the j-th window at scale level 1, its corresponding original prefix sequence is denoted as... ,and, For the j'-th window at scale level 2, its corresponding original prefix sequence is denoted as... ,and, For the j''-th window at scale level 3, its corresponding original prefix sequence is denoted as... ,and, .

[0223] To control computational complexity, a prefix truncation threshold is introduced to process the original prefix sequence, thereby limiting the maximum length of each prefix sequence. Taking the j-th window in the window sequence corresponding to the i-th scale level as an example, the corresponding prefix sequence can be extracted from the original prefix sequence in the following way:

[0224] ;Formula (10)

[0225] in, This represents the prefix sequence corresponding to the j-th window at the i-th scale level. Represents the i-th scale level. The window embedding vector corresponding to each window. This represents the window embedding vector corresponding to the (j-1)th window at the i-th scale level. This indicates the prefix truncation threshold.

[0226] visible, From the first The prefix sequence is composed of the window embedding vectors corresponding to the first window to the jth window. Therefore, the length of the prefix sequence is less than or equal to the prefix truncation threshold. That is, if (j-1) is less than or equal to the prefix truncation threshold, the window embedding vectors corresponding to the (j-1) windows are taken as the prefix sequence. If (j-1) is greater than the prefix truncation threshold, truncation is needed, that is, the τ windows closest to the jth window are truncated, and the window embedding vectors corresponding to the τ windows are taken as the prefix sequence.

[0227] Please continue to refer to Figure 9 For the jth window under the scale level 1, the corresponding prefix sequence is denoted as , and For the j'th window under the scale level 2, the corresponding prefix sequence is denoted as , and For the j''th window under the scale level 3, the corresponding prefix sequence is denoted as , and .

[0228] Based on this, for the jth window under the ith scale level, the prefix sequences containing the ith scale level and larger scale levels (that is, from the scale level i+1 to the scale level n) are combined to form the corresponding multi-scale prefix set, so that the window of a small scale can access the global context information provided by the larger scale window, that is:

[0229] Equation (11)

[0230] wherein, denotes the multi-scale prefix set corresponding to the jth window under the ith scale level. denotes the prefix sequence corresponding to the jth window under the ith scale level. denotes the prefix sequence corresponding to the jth window under the n-th scale level.

[0231] It should be noted that on the edge device, the MS-ViT model inference speed can be further improved. After quantization of the MS-ViT model, the performance loss is small, and therefore, it is more suitable for resource-constrained scenarios. In addition, the energy efficiency of the MS-ViT model deployed on the mobile end is also improved.

[0232] Secondly, in the embodiments of the present application, another way of constructing a multi-scale prefix set is provided. Through the above-mentioned way, under the same scale level, part of the window embedding vectors corresponding to the historical windows are taken as the prefix sequence according to the prefix truncation threshold. In this way, the computational complexity can be controlled to wherein, denotes the total number of windows, represents a prefix truncation threshold. Compared with the ViT model, the MS-ViT model can improve the speed of processing high-resolution images (for example, 1024x1024) by 30% to 50% and improve the memory usage efficiency by about 40% under the same hardware conditions, so as to better support large-scale training.

[0233] Optionally, in the above Figure 6 Based on one or more embodiments corresponding, another optional embodiment provided by the embodiment of the application can further include:

[0234] Obtaining a task estimated time consumption corresponding to the image processing task;

[0235] In a case where the task estimated time consumption is less than or equal to the delay threshold, a first preset value is taken as the prefix truncation threshold;

[0236] In a case where the task estimated time consumption is greater than the delay threshold, a second preset value is taken as the prefix truncation threshold, wherein the second preset value is greater than the first preset value.

[0237] In one or more embodiments, a manner of supporting custom prefix truncation threshold is introduced. As known from the foregoing embodiments, the prefix truncation threshold is a parameter for controlling the computational complexity and limiting the length of the historical information considered, and in the present application, the prefix truncation threshold is set as a tunable hyperparameter.

[0238] Specifically, the prefix truncation threshold can limit the maximum number of historical windows considered by each window when calculating attention. For example, the prefix truncation threshold is set to 10, that is, each window considers at most the previous 10 windows. For another example, the prefix truncation threshold is set to 50, that is, each window considers at most the previous 50 windows. For another example, the prefix truncation threshold is set to ∞, that is, each window considers all previous windows (i.e., no truncation).

[0239] In the present application, the prefix truncation threshold can be set according to the task estimated time consumption of different image processing tasks. Generally, for image processing tasks with shorter task estimated time consumption, a smaller prefix truncation threshold can be used. For image processing tasks with longer task estimated time consumption, a larger prefix truncation threshold can be used.

[0240] Based on this, the image processing task is determined to correspond to a task estimated time consumption. The task estimated time consumption can be an average time consumption of the image processing task within a period of time, or an expected time consumption provided by a business party. For example, in a real-time video processing task, the corresponding task estimated time consumption can be 5 milliseconds, and for example, in a high-precision image generation task, the corresponding task estimated time consumption can be 20 milliseconds. If the task estimated time consumption is less than or equal to a delay threshold (for example, 10 milliseconds), a first preset value (for example, 8) is taken as a prefix truncation threshold (that is, τ = 8). If the task estimated time consumption is greater than the delay threshold, a second preset value (for example, 128) is taken as the prefix truncation threshold (that is, τ = 128).

[0241] It can be seen that the larger the prefix truncation threshold, the larger the context range, the higher the calculation complexity and the higher the generation quality. The smaller the prefix truncation threshold, the smaller the context range, the lower the calculation complexity, and the faster the inference speed.

[0242] Again, in the embodiments of the present application, a way of supporting custom prefix truncation threshold is provided. Through the above way, the prefix truncation threshold can be adjusted on demand, on the one hand, to avoid wasting of calculation resources caused by redundant information, and on the other hand, to avoid loss of key information. Thus, the processing efficiency and the sequence representation integrity can be effectively balanced, thereby improving the applicability and generalization ability of the MS-ViT model.

[0243] Optionally, in the above Figure 6 Based on one or more embodiments corresponding thereto, in another optional embodiment provided by the embodiments of the present application, for the jth window in the window sequence corresponding to the ith scale level, a multi-scale prefix set is obtained according to the window embedding sequence corresponding to the ith scale level, which can specifically include:

[0244] According to the number of windows included in the window sequence corresponding to the ith scale level, a dynamic truncation threshold corresponding to the ith scale level is calculated;

[0245] For the jth window in the window sequence corresponding to the ith scale level, in the case that (j-1) is less than or equal to the dynamic truncation threshold corresponding to the ith scale level, the window embedding vector corresponding to the (j-1) windows is taken as the prefix sequence, wherein the (j-1) windows are all the windows before the jth window in the window sequence;

[0246] For the jth window in the window sequence corresponding to the ith scale level, in the case that (j-1) is greater than the dynamic truncation threshold corresponding to the ith scale level, the window embedding vector corresponding to the last Li windows before the jth window is taken as the prefix sequence, wherein Li is an integer greater than or equal to 1, and Li represents the dynamic truncation threshold corresponding to the ith scale level;

[0247] For the jth window in the window sequence corresponding to the ith scale level, the prefix sequence from the ith scale level to the nth scale level is taken as the multi-scale prefix set according to the prefix sequence corresponding to the window.

[0248] In one or more embodiments, another way of constructing a multi-scale prefix set is introduced. As known from the foregoing embodiments, the window sequence corresponding to the ith scale level is denoted as , and , denotes the number of windows corresponding to the ith scale level. For the jth window in the window sequence, the causal principle is followed in calculating the attention feature, that is, only the windows before the jth window are considered.

[0249] Specifically, taking the division of the image to be processed based on 3 scale levels as an example, for ease of understanding, please refer to Figure 10 , Figure 10 Fig. 3 is another schematic diagram of constructing a multi-scale prefix set in the embodiments of the present application, as shown in the figure, for the jth window under the scale level 1, the original prefix sequence corresponding thereto is denoted as , and . For the j'th window under the scale level 2, the original prefix sequence corresponding thereto is denoted as , and . For the j''th window under the scale level 3, the original prefix sequence corresponding thereto is denoted as , and .

[0250] In order to control the computational complexity, the dynamic truncation threshold corresponding to each scale level is calculated, and the original prefix sequence is processed based on the dynamic truncation threshold, so as to limit the maximum length of each prefix sequence. Taking the window sequence corresponding to the ith scale level as an example, the dynamic truncation threshold can be calculated in the following way:

[0251] Equation (12)

[0252] wherein denotes the dynamic truncation threshold corresponding to the ith scale level. denotes the floor function. denotes the truncation ratio, and . denotes the number of windows corresponding to the ith scale level.

[0253] Taking the jth window in the window sequence corresponding to the ith scale level as an example, the corresponding prefix sequence can be taken out from the original prefix sequence in the following way:

[0254] Equation (13)

[0255] wherein, represents the prefix sequence corresponding to the jth window at the ith scale level. represents the window embedding vector corresponding to the jth window at the ith scale level. represents the window embedding vector corresponding to the jth window at the ith scale level. represents the window embedding vector corresponding to the jth window at the ith scale level. represents the dynamic truncation threshold corresponding to the ith scale level.

[0256] It can be seen that, is the prefix sequence composed of the window embedding vectors corresponding to the jth window and the jth window. Therefore, the length of the prefix sequence is less than or equal to the dynamic truncation threshold at the corresponding scale level. That is, if (j-1) is less than or equal to the dynamic truncation threshold, the window embedding vector corresponding to the (j-1)th window is taken as the prefix sequence. If (j-1) is greater than the dynamic truncation threshold, truncation is needed, that is, the Li windows closest to the jth window are truncated, and the window embedding vectors corresponding to the Li windows are taken as the prefix sequence.

[0257] Please continue to refer to Figure 10 , for the jth window at the scale level 1, the corresponding prefix sequence is denoted as , and . For the j'th window at the scale level 2, the corresponding prefix sequence is denoted as , and . For the j''th window at the scale level 3, the corresponding prefix sequence is denoted as , and .

[0258] Based on this, for the jth window at the ith scale level, the prefix sequence of the ith scale level and larger scale levels (i.e., from scale level i+1 to scale level n) are combined to form a corresponding multi-scale prefix set, so that the window of a small scale can access the global context information provided by the larger scale window, that is, please refer to the aforementioned formula (11), which will not be repeated here.

[0259] Secondly, in the embodiment of the application, another way of constructing a multi-scale prefix set is provided. Through the above-mentioned way, the dynamic truncation threshold can be calculated in combination with the length of the window sequence, and the flexibility of threshold selection is improved. Therefore, not only can the prefix information be sufficient, but also information redundancy can be avoided.

[0260] Optionally, in the above-mentioned Figure 6According to one or more embodiments, in another optional embodiment provided by the embodiments of the present application, for the jth window in the window sequence corresponding to the ith scale level, the multi-scale prefix set is calculated by attention weighting to obtain a multi-scale attention feature, which can specifically include:

[0261] For the jth window in the window sequence corresponding to the ith scale level, a query vector corresponding to the window is constructed according to the window embedding vector corresponding to the window.

[0262] For the jth window in the window sequence corresponding to the ith scale level, a key matrix and a value matrix corresponding to the window are constructed according to the multi-scale prefix set corresponding to the window.

[0263] For the jth window in the window sequence corresponding to the ith scale level, the query vector, the key matrix and the value matrix corresponding to the window are calculated by attention to obtain an attention feature.

[0264] For the jth window in the window sequence corresponding to the ith scale level, the attention features of the window from the ith scale level to the nth scale level are calculated by weighting to obtain a multi-scale attention feature corresponding to the window.

[0265] In one or more embodiments, a way of calculating the multi-scale prefix set by attention weighting is introduced. As known from the foregoing embodiments, the single-head attention mechanism or the multi-head attention mechanism can be used for attention weighting calculation. Among them, the multi-head attention is used to perform multiple times in parallel for the attention mechanism, each time using a different projection matrix, and then combining the results to capture the feature relationship of different subspaces.

[0266] Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, the attention feature can be calculated in the following way:

[0267] Equation (14)

[0268] Wherein, represents the attention feature of the jth window under the ith scale level. represents the query vector of the jth window under the ith scale level, which is converted from the window embedding vector of the jth window under the ith scale level (for example, by linear transformation). represents the key matrix converted based on . represents the value matrix converted based on . represents the multi-scale prefix set corresponding to the jth window under the ith scale level. a scaling factor representing an attention dimension.

[0269] Based on this, the multi-scale attention feature corresponding to the window can be calculated in the following manner:

[0270] Equation (15)

[0271] wherein, represents the multi-scale attention feature of the jth window at the ith scale level. represents a trainable weight parameter, and satisfies . represents the attention feature of the jth window at the kth scale level, and k ranges from i to n.

[0272] It should be noted that in actual application, the number of layers and the hidden dimension can also be set based on the model deployment location. For example, when the MS-ViT model is deployed on a mobile terminal, the processing resources are limited, and the accuracy needs to be sacrificed for speed, so the number of layers can be configured as 12 and the hidden dimension can be configured as 768. For another example, when the MS-ViT model is deployed on a cloud service, the best performance can be pursued, so the number of layers can be configured as 24 and the hidden dimension can be configured as 2048.

[0273] Secondly, in the embodiments of the present application, a manner of attention weighted calculation on the multi-scale prefix set is provided. Through the above manner, the effective fusion of information of different scales is realized based on attention weighted average, that is, the multi-scale fusion is realized through the attention mechanism. Therefore, the information exchange and fusion between different scales are realized, so as to more effectively balance the capture of local details and global context.

[0274] Optionally, in the above Figure 6 Based on one or more embodiments described above, in another optional embodiment provided by the embodiments of the present application, an image processing result is generated according to the multi-scale attention feature corresponding to each window in each scale level, which can specifically include:

[0275] Determine an image global feature according to the multi-scale attention feature corresponding to each window in each scale level.

[0276] Obtain a category probability vector through a classifier based on the image global feature, wherein the category probability vector includes a probability value of the to-be-processed image belonging to each category.

[0277] Determine the category with the maximum probability value in the category probability vector as the classification result of the to-be-processed image.

[0278] In one or more embodiments, a manner of implementing image classification is introduced. As known from the foregoing embodiments, after obtaining the multi-scale attention features corresponding to each window in each scale level, all the multi-scale attention features can be aggregated based on global average pooling.

[0279] Specifically, taking the jth window in the window sequence corresponding to the ith scale level as an example, the image global feature can be calculated in the following manner:

[0280] Equation (16)

[0281] wherein, denotes the image global feature. n denotes the total number of scale levels. denotes the number of corresponding windows under the ith scale level. denotes the multi-scale attention feature of the jth window under the ith scale level.

[0282] Based on this, the image global feature is taken as the input of a classifier (for example, a fully connected network), and the classifier outputs a category probability vector in the following manner:

[0283] Equation (17)

[0284] wherein, denotes the category probability vector. denotes the weight matrix of the classifier. denotes the image global feature. denotes the bias vector of the classifier.

[0285] It can be understood that the category probability vector includes the probability value of the to-be-processed image belonging to each category. Thus, the category corresponding to the maximum probability value is taken as the classification result of the to-be-processed image.

[0286] Secondly, in the embodiments of the present application, a manner of implementing image classification is provided. Through the above manner, based on the aggregation of the attention features of multi-scale windows, the fine-grained local features and global semantic information can be captured at the same time, and in a complex scene with multi-scale targets, the detection accuracy of the MS-ViT model can be improved by 15% to 20% compared with the ViT model. The accuracy of the MS-ViT model on fine-grained classification tasks (for example, bird identification, vehicle type identification) is improved by 3% to 5%. Based on this, under the condition that the model parameter amount is reduced by 20%, the MS-ViT model can still maintain good classification performance.

[0287] Optionally, in the above Figure 6Based on one or more corresponding embodiments, in another optional embodiment provided by this application, image processing results are generated according to the multi-scale attention features corresponding to each window in each scale level, which may specifically include:

[0288] For the j-th window in the window sequence corresponding to the i-th scale level, the candidate box coordinates and target probability vector corresponding to the window are determined based on the multi-scale attention features corresponding to the window. The candidate box coordinates are used to indicate a candidate box.

[0289] Based on the candidate box coordinates and target probability vector corresponding to each window at each scale level, non-maximum suppression is performed on all candidate boxes to obtain the target detection results.

[0290] In one or more embodiments, a method for implementing object detection is described. As can be seen from the foregoing embodiments, after obtaining the multi-scale attention features corresponding to each window at each scale level, it is necessary to execute the bounding box regression branch and the category classification branch respectively.

[0291] Specifically, taking the j-th window in the window sequence corresponding to the i-th scale level as an example, the candidate box coordinates can be calculated in the bounding box regression branch as follows:

[0292] ;Formula (18)

[0293] in, This represents the coordinates of the candidate box corresponding to the j-th window at the i-th scale level. For example, the candidate box coordinates include the x-coordinate of the center, y-coordinate of the center, the height of the box, and the width of the box. This represents the weight matrix corresponding to the bounding box regression branch. This represents the bias vector corresponding to the bounding box regression branch. This represents the multi-scale attention feature of the j-th window at the i-th scale level.

[0294] In the category classification branch, the target probability vector can be calculated as follows:

[0295] ;Formula (19)

[0296] in, This represents the target probability vector. This represents the weight matrix of the category classification branches. The bias vector represents the category classification branch. This represents the multi-scale attention feature of the j-th window at the i-th scale level.

[0297] Based on this, candidate boxes for all scale levels can be obtained according to the coordinates of the candidate boxes corresponding to each window at each scale level. Based on the target probability vector corresponding to each window at each scale level, the candidate boxes for all scale levels are sorted in descending order of class probability (i.e., the maximum value in the target probability vector). Next, non-maximum suppression (NMS) is performed, and an intersection-overlap ratio (IoU) threshold (e.g., 0.5) is set to remove candidate boxes whose overlap exceeds the IoU threshold, retaining the candidate box with the highest confidence as the target detection result.

[0298] Secondly, this application provides a method for object detection. Through this method, based on multi-scale attention feature matching, the problem of missed detection of small objects in single-scale detection is solved. Compared to the ViT model, the MS-ViT model improves the average precision (AP) by 2% to 3% on the common objects in context (COCO) dataset, especially in small object detection. When processing high-resolution medical images, the lesion detection rate is improved by approximately 10%.

[0299] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, image processing results are generated according to the multi-scale attention features corresponding to each window in each scale level, which may specifically include:

[0300] The multi-scale attention features corresponding to each window at each scale level are concatenated to obtain multi-scale fused features.

[0301] Upsampling is performed on the multi-scale fusion features to obtain the target feature map, wherein the target feature map has the same resolution as the image to be processed;

[0302] Pixel-by-pixel category prediction is performed on the target feature map to obtain the semantic segmentation result of the image to be processed.

[0303] In one or more embodiments, a method for implementing object detection is described. As can be seen from the foregoing embodiments, after obtaining the multi-scale attention features corresponding to each window at each scale level, all multi-scale attention features can be concatenated to obtain multi-scale fused features.

[0304] Specifically, multi-scale fusion features can be calculated in the following way:

[0305] ;Formula (20)

[0306] in, This represents multi-scale fusion features. n represents the total number of scale levels. This indicates the splicing of tensor dimensions. This represents the convolution corresponding to the i-th scale level. This represents the multi-scale attention features for all windows at the i-th scale level (i.e., The result after aggregation.

[0307] Based on this, progressive upsampling can be performed using transposed convolution, i.e.:

[0308] ;Formula (21)

[0309] in, Represents the target feature map. This represents transposed convolution. This represents the multi-scale fusion feature.

[0310] Since the target feature map and the image to be processed have the same resolution, category prediction can be performed on each pixel in the target feature map, that is:

[0311] ;Formula (22)

[0312] in, The class probability vector of pixel (x,y) is used to select the class corresponding to the maximum probability as the segmentation label of the pixel. The labels of all pixels constitute the final semantic segmentation result (i.e., the segmentation mask). This represents the weight matrix for the segmentation task. This represents the bias vector for the segmentation task. This represents the feature vector of pixel (x, y) in the target feature map.

[0313] Secondly, this application provides a method for object detection. Through this method, based on the fusion and upsampling of multi-scale attention features, both global context and local detail information are considered, resulting in smoother semantic segmentation boundary boundaries, thereby improving instance segmentation quality. Compared to the ViT model, the MS-ViT model improves boundary accuracy by approximately 5%. Furthermore, the Fréchet inception distance (FID) score of the generated image is improved by 20% to 30%. In image inpainting tasks, detail fidelity and contextual consistency are significantly improved, helping to support the generation of higher resolution and better quality images.

[0314] Based on the above introduction, the structure of the MS-ViT model will be described below. Please refer to [link / reference]. Figure 11 , Figure 11The figure shows a schematic diagram of the structure of the visual transformation model in this application embodiment. As shown in the figure, the MS-ViT model includes an input layer for receiving and preprocessing the input image to be processed, a multi-scale embedding layer for implementing multi-scale window segmentation and embedding generation, a multi-scale transformer module (including a multi-scale prefix attention layer, a feedforward network layer, layer normalization, and residual connections), and an output layer designed according to the specific task.

[0315] The input to the MS-ViT model includes the image to be processed. Scale factor set Prefix truncation threshold The number of layers in a multi-scale transformer module .

[0316] The MS-ViT model outputs image processing results for a specific task.

[0317] In summary, the pseudocode description of the core algorithm of the MS-ViT model is as follows:

[0318] As can be seen, the flow of information in this application is mainly reflected in two dimensions: intra-sequence information flow (i.e., within each scale level, information flows unidirectionally from front to back along the sequence, which conforms to the principle of causality) and cross-scale information flow (i.e., information flows from large-scale window to small-scale window, realizing the context transfer from global to local).

[0319] Multi-scale information fusion can be achieved through spatial correspondence mapping (i.e., establishing spatial correspondence between windows of different scales so that information can flow correctly), attention weighting mechanism (i.e., adaptively adjusting the importance of information at different scales through learnable weight parameters), and hierarchical information integration (i.e., as the network depth increases, information is gradually fused and refined at various scale levels).

[0320] The image processing apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 12 , Figure 12 This is a schematic diagram of one embodiment of the image processing apparatus in this application. The image processing apparatus 120 includes:

[0321] The processing module 1201 is used to divide the image to be processed according to n scale levels to obtain a window sequence corresponding to each scale level. The window sequence includes multiple windows belonging to the corresponding scale level, and n is an integer greater than 1.

[0322] The processing module 1201 is also used to perform feature embedding processing on each window sequence to obtain a window embedding sequence corresponding to each window sequence, wherein the window embedding sequence includes window embedding vectors corresponding to each window belonging to the corresponding scale level.

[0323] The acquisition module 1202 is used to acquire a multi-scale prefix set for the j-th window in the window sequence corresponding to the i-th scale level, based on the window embedding sequence corresponding to the i-th scale level. The multi-scale prefix set includes the prefix sequence corresponding to the j-th window from the i-th scale level to the n-th scale level. The prefix sequence includes the window embedding vector corresponding to the windows before the j-th window. i and j are both integers greater than or equal to 1.

[0324] The calculation module 1203 is used to perform attention weighting calculation on the multi-scale prefix set for the j-th window in the window sequence corresponding to the i-th scale level, so as to obtain the multi-scale attention features;

[0325] The generation module 1204 is used to generate image processing results based on the multi-scale attention features corresponding to each window in each scale level.

[0326] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0327] The processing module 1201 is specifically used to determine n scale factors based on n scale levels, where each scale level corresponds to one scale factor;

[0328] n window sizes are determined based on n scale factors, where each scale factor corresponds to one window size;

[0329] The image to be processed is divided into windows according to n window sizes to obtain the window sequence corresponding to each scale level.

[0330] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0331] The processing module 1201 is specifically used to embed and encode the pixel values ​​corresponding to all pixels in the window of the window sequence corresponding to the i-th scale level to obtain the pixel projection vector.

[0332] For the j-th window in the window sequence corresponding to the i-th scale level, perform position encoding on the pixel values ​​corresponding to all pixels in the window to obtain the position encoding vector;

[0333] For the j-th window in the window sequence corresponding to the i-th scale level, the pixel projection vector and the position encoding vector corresponding to the window are added dimension by dimension to obtain the window embedding vector;

[0334] The window embedding vectors corresponding to all windows in the same window sequence are used as the window embedding sequence corresponding to the window sequence.

[0335] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0336] The processing module 1201 is specifically used to generate a fusion basis vector corresponding to the window for the j-th window in the window sequence corresponding to the i-th scale level, based on the scale level and spatial location information of the window, wherein the spatial location information represents the two-dimensional coordinates of the window in the image to be processed.

[0337] For the j-th window in the window sequence corresponding to the i-th scale level, the position encoding vector corresponding to the window is output through a fully connected network based on the fusion base vector corresponding to the window.

[0338] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0339] The processing module 1201 is specifically used to extract the vector corresponding to the jth row from the spatial encoding matrix corresponding to the i-th scale level as the spatial encoding vector for the j-th window in the window sequence corresponding to the i-th scale level. The spatial encoding matrix is ​​a numerical matrix whose parameters can be optimized through model training.

[0340] For the j-th window in the window sequence corresponding to the i-th scale level, obtain the scale encoding vector corresponding to the i-th scale level, where the scale encoding vector is a numerical vector whose parameters can be optimized through model training;

[0341] For the j-th window in the window sequence corresponding to the i-th scale level, generate the positional encoding vector corresponding to the window based on the spatial encoding vector and scale encoding vector corresponding to the window.

[0342] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0343] The processing module 1201 is specifically used to perform sine and cosine encoding on the spatial location information of the j-th window in the window sequence corresponding to the i-th scale level, so as to obtain a spatial sine and cosine vector, wherein the spatial location information represents the two-dimensional coordinates of the window in the image to be processed.

[0344] For the j-th window in the window sequence corresponding to the i-th scale level, perform sine and cosine encoding on the scale level corresponding to the window to obtain the scale sine and cosine vector;

[0345] For the j-th window in the window sequence corresponding to the i-th scale level, generate the position encoding vector corresponding to the window based on the spatial sine and cosine vectors and the scale sine and cosine vectors corresponding to the window.

[0346] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0347] The acquisition module 1202 is specifically used to take the window embedding vectors corresponding to the (j-1) windows as a prefix sequence for the j-th window in the window sequence corresponding to the i-th scale level, where the (j-1) windows are all the windows in the window sequence before the j-th window;

[0348] For the j-th window in the window sequence corresponding to the i-th scale level, the prefix sequence from the i-th scale level to the n-th scale level is used as a multi-scale prefix set based on the prefix sequence corresponding to the window.

[0349] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0350] The acquisition module 1202 is specifically used to, for the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is less than or equal to the prefix truncation threshold, take the window embedding vector corresponding to (j-1) windows as the prefix sequence, where (j-1) windows are all windows in the window sequence before the j-th window;

[0351] For the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is greater than the prefix truncation threshold, the window embedding vectors corresponding to the τ nearest windows before the j-th window are used as the prefix sequence, where τ is an integer greater than or equal to 1, and τ represents the prefix truncation threshold.

[0352] For the j-th window in the window sequence corresponding to the i-th scale level, the prefix sequence from the i-th scale level to the n-th scale level is used as a multi-scale prefix set based on the prefix sequence corresponding to the window.

[0353] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0354] The acquisition module 1202 is also used to acquire the estimated time of the image processing task.

[0355] The processing module 1201 is also used to use the first preset value as the prefix truncation threshold when the estimated task time is less than or equal to the delay threshold.

[0356] The processing module 1201 is further configured to use a second preset value as a prefix truncation threshold when the estimated task time is greater than the delay threshold, wherein the second preset value is greater than the first preset value.

[0357] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0358] The acquisition module 1202 is specifically used to calculate the dynamic truncation threshold corresponding to the i-th scale level based on the number of windows included in the window sequence corresponding to the i-th scale level.

[0359] For the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is less than or equal to the dynamic truncation threshold corresponding to the i-th scale level, the window embedding vectors corresponding to (j-1) windows are used as a prefix sequence, where (j-1) windows are all windows in the window sequence before the j-th window;

[0360] For the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is greater than the dynamic truncation threshold corresponding to the i-th scale level, the window embedding vectors corresponding to the Li nearest windows before the j-th window are used as the prefix sequence, where Li is an integer greater than or equal to 1, and Li represents the dynamic truncation threshold corresponding to the i-th scale level.

[0361] For the j-th window in the window sequence corresponding to the i-th scale level, the prefix sequence from the i-th scale level to the n-th scale level is used as a multi-scale prefix set based on the prefix sequence corresponding to the window.

[0362] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0363] The calculation module 1203 is specifically used to construct the query vector corresponding to the window for the j-th window in the window sequence corresponding to the i-th scale level, based on the window embedding vector corresponding to the window.

[0364] For the j-th window in the window sequence corresponding to the i-th scale level, construct the key matrix and value matrix corresponding to the window based on the multi-scale prefix set corresponding to the window;

[0365] For the j-th window in the window sequence corresponding to the i-th scale level, attention is calculated on the query vector, key matrix, and value matrix corresponding to the window to obtain attention features;

[0366] For the j-th window in the window sequence corresponding to the i-th scale level, the attention features of the window from the i-th scale level to the n-th scale level are weighted and calculated to obtain the multi-scale attention features corresponding to the window.

[0367] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0368] The generation module 1204 is specifically used to determine the global features of the image based on the multi-scale attention features corresponding to each window in each scale level;

[0369] Based on global image features, a classifier is used to obtain a category probability vector, which includes the probability value of the image to be processed belonging to each category.

[0370] The category with the highest probability value in the category probability vector is determined as the classification result of the image to be processed.

[0371] Optionally, in the above Figure 12 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0372] The generation module 1204 is specifically used to determine the candidate box coordinates and target probability vector of the j-th window in the window sequence corresponding to the i-th scale level, based on the multi-scale attention features corresponding to the window. The candidate box coordinates are used to indicate a candidate box.

[0373] Based on the candidate box coordinates and target probability vector corresponding to each window at each scale level, non-maximum suppression is performed on all candidate boxes to obtain the target detection results.

[0374] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the image processing apparatus 120 provided in this application,

[0375] The generation module 1204 is specifically used to concatenate the multi-scale attention features corresponding to each window at each scale level to obtain multi-scale fusion features;

[0376] Upsampling is performed on the multi-scale fusion features to obtain the target feature map, wherein the target feature map has the same resolution as the image to be processed;

[0377] Pixel-by-pixel category prediction is performed on the target feature map to obtain the semantic segmentation result of the image to be processed.

[0378] Figure 13 This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 1300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1322 (e.g., one or more processors) and a memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device. Furthermore, the CPU 1322 may be configured to communicate with the storage media 1330 and execute the series of instruction operations in the storage media 1330 on the computer device 1300.

[0379] Computer device 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems 1341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0380] The steps performed by the computer device in the above embodiments can be based on this ​ The computer device structure shown.

[0381] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0382] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0383] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0384] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0385] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0386] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0387] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0388] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a server or terminal device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0389] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: The image to be processed is divided into n scale levels to obtain a window sequence corresponding to each scale level. The window sequence includes multiple windows belonging to the corresponding scale level, and n is an integer greater than 1. Each window sequence is subjected to feature embedding processing to obtain a window embedding sequence corresponding to each window sequence, wherein the window embedding sequence includes window embedding vectors corresponding to each window belonging to the corresponding scale level; For the j-th window in the window sequence corresponding to the i-th scale level, a multi-scale prefix set is obtained based on the window embedding sequence corresponding to the i-th scale level. The multi-scale prefix set includes prefix sequences corresponding to the j-th window from the i-th scale level to the n-th scale level. The prefix sequences include window embedding vectors corresponding to windows preceding the j-th window. i and j are both integers greater than or equal to 1. For the j-th window in the window sequence corresponding to the i-th scale level, attention weighting is performed on the multi-scale prefix set to obtain the multi-scale attention features; Image processing results are generated based on the multi-scale attention features corresponding to each window at each scale level.

2. The method according to claim 1, characterized in that, The process of dividing the image to be processed according to n scale levels to obtain a window sequence corresponding to each scale level includes: Based on the n scale levels, n scale factors are determined, where each scale level corresponds to one scale factor; n window sizes are determined based on the n scale factors, where each scale factor corresponds to one window size; The image to be processed is divided into windows according to the n window sizes to obtain the window sequence corresponding to each scale level.

3. The method according to claim 1 or 2, characterized in that, The step of performing feature embedding processing on each window sequence to obtain a window embedding sequence corresponding to each window sequence includes: For the j-th window in the window sequence corresponding to the i-th scale level, embedding encoding is performed on the pixel values ​​corresponding to all pixels in the window to obtain the pixel projection vector; For the j-th window in the window sequence corresponding to the i-th scale level, perform position encoding on the pixel values ​​corresponding to all pixels in the window to obtain the position encoding vector; For the j-th window in the window sequence corresponding to the i-th scale level, the pixel projection vector and the position encoding vector corresponding to the window are added dimension by dimension to obtain the window embedding vector; The window embedding vectors corresponding to all windows in the same window sequence are used as the window embedding sequence corresponding to the window sequence.

4. The method according to claim 3, characterized in that, For the j-th window in the window sequence corresponding to the i-th scale level, position encoding is performed on the pixel values ​​corresponding to all pixels within the window to obtain a position encoding vector, including: For the j-th window in the window sequence corresponding to the i-th scale level, a fusion basis vector corresponding to the window is generated according to the scale level and spatial location information of the window, wherein the spatial location information represents the two-dimensional coordinates of the window in the image to be processed; For the j-th window in the window sequence corresponding to the i-th scale level, the position encoding vector corresponding to the window is output through a fully connected network based on the fusion base vector corresponding to the window.

5. The method according to claim 3, characterized in that, For the j-th window in the window sequence corresponding to the i-th scale level, position encoding is performed on the pixel values ​​corresponding to all pixels within the window to obtain a position encoding vector, including: For the j-th window in the window sequence corresponding to the i-th scale level, extract the vector corresponding to the j-th row from the spatial encoding matrix corresponding to the i-th scale level as the spatial encoding vector, wherein the spatial encoding matrix is ​​a numerical matrix whose parameters can be optimized through model training; For the j-th window in the window sequence corresponding to the i-th scale level, obtain the scale encoding vector corresponding to the i-th scale level, wherein the scale encoding vector is a numerical vector whose parameters can be optimized through model training; For the j-th window in the window sequence corresponding to the i-th scale level, the position encoding vector corresponding to the window is generated based on the spatial encoding vector and the scale encoding vector corresponding to the window.

6. The method according to claim 3, characterized in that, For the j-th window in the window sequence corresponding to the i-th scale level, position encoding is performed on the pixel values ​​corresponding to all pixels within the window to obtain a position encoding vector, including: For the j-th window in the window sequence corresponding to the i-th scale level, the spatial location information corresponding to the window is sine and cosine encoded to obtain a spatial sine and cosine vector, wherein the spatial location information represents the two-dimensional coordinates of the window in the image to be processed; For the j-th window in the window sequence corresponding to the i-th scale level, perform sine and cosine encoding on the scale level corresponding to the window to obtain the scale sine and cosine vector; For the j-th window in the window sequence corresponding to the i-th scale level, the position encoding vector corresponding to the window is generated based on the spatial sine and cosine vectors and the scale sine and cosine vectors corresponding to the window.

7. The method according to any one of claims 1 to 6, characterized in that, For the j-th window in the window sequence corresponding to the i-th scale level, a multi-scale prefix set is obtained based on the window embedding sequence corresponding to the i-th scale level, including: For the j-th window in the window sequence corresponding to the i-th scale level, the window embedding vectors corresponding to (j-1) windows are used as a prefix sequence, where the (j-1) windows are all windows in the window sequence before the j-th window; For the j-th window in the window sequence corresponding to the i-th scale level, the prefix sequence from the i-th scale level to the n-th scale level is used as a multi-scale prefix set based on the prefix sequence corresponding to the window.

8. The method according to any one of claims 1 to 6, characterized in that, For the j-th window in the window sequence corresponding to the i-th scale level, a multi-scale prefix set is obtained based on the window embedding sequence corresponding to the i-th scale level, including: For the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is less than or equal to the prefix truncation threshold, the window embedding vectors corresponding to (j-1) windows are used as the prefix sequence, where the (j-1) windows are all windows in the window sequence before the j-th window; For the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is greater than the prefix truncation threshold, the window embedding vectors corresponding to the τ nearest windows before the j-th window are used as the prefix sequence, where τ is an integer greater than or equal to 1, and τ represents the prefix truncation threshold. For the j-th window in the window sequence corresponding to the i-th scale level, the prefix sequence from the i-th scale level to the n-th scale level is used as a multi-scale prefix set based on the prefix sequence corresponding to the window.

9. The method according to claim 8, characterized in that, The method further includes: Obtain the estimated time consumption of the image processing task; If the estimated time of the task is less than or equal to the delay threshold, the first preset value is used as the prefix truncation threshold. If the estimated time of the task is greater than the delay threshold, a second preset value is used as the prefix truncation threshold, wherein the second preset value is greater than the first preset value.

10. The method according to any one of claims 1 to 6, characterized in that, For the j-th window in the window sequence corresponding to the i-th scale level, a multi-scale prefix set is obtained based on the window embedding sequence corresponding to the i-th scale level, including: The dynamic truncation threshold corresponding to the i-th scale level is calculated based on the number of windows included in the window sequence corresponding to the i-th scale level. For the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is less than or equal to the dynamic truncation threshold corresponding to the i-th scale level, the window embedding vectors corresponding to (j-1) windows are used as a prefix sequence, where the (j-1) windows are all windows in the window sequence before the j-th window; For the j-th window in the window sequence corresponding to the i-th scale level, if (j-1) is greater than the dynamic truncation threshold corresponding to the i-th scale level, the window embedding vectors corresponding to the Li nearest windows before the j-th window are used as a prefix sequence, where Li is an integer greater than or equal to 1, and Li represents the dynamic truncation threshold corresponding to the i-th scale level. For the j-th window in the window sequence corresponding to the i-th scale level, the prefix sequence from the i-th scale level to the n-th scale level is used as a multi-scale prefix set based on the prefix sequence corresponding to the window.

11. The method according to any one of claims 1 to 10, characterized in that, For the j-th window in the window sequence corresponding to the i-th scale level, attention weighting is performed on the multi-scale prefix set to obtain multi-scale attention features, including: For the j-th window in the window sequence corresponding to the i-th scale level, construct the query vector corresponding to the window based on the window embedding vector corresponding to the window; For the j-th window in the window sequence corresponding to the i-th scale level, construct the key matrix and value matrix corresponding to the window based on the multi-scale prefix set corresponding to the window; For the j-th window in the window sequence corresponding to the i-th scale level, attention is calculated on the query vector, key matrix, and value matrix corresponding to the window to obtain attention features; For the j-th window in the window sequence corresponding to the i-th scale level, the attention features of the window from the i-th scale level to the n-th scale level are weighted and calculated to obtain the multi-scale attention features corresponding to the window.

12. The method according to any one of claims 1 to 11, characterized in that, The step of generating image processing results based on the multi-scale attention features corresponding to each window at each scale level includes: Based on the multi-scale attention features corresponding to each window in each scale level, determine the global features of the image; Based on the global features of the image, a classifier is used to obtain a category probability vector, wherein the category probability vector includes the probability value of the image to be processed belonging to each category; The category with the highest probability value in the category probability vector is determined as the classification result of the image to be processed.

13. The method according to any one of claims 1 to 11, characterized in that, The step of generating image processing results based on the multi-scale attention features corresponding to each window at each scale level includes: For the j-th window in the window sequence corresponding to the i-th scale level, the candidate box coordinates and target probability vector corresponding to the window are determined according to the multi-scale attention features corresponding to the window, wherein the candidate box coordinates are used to indicate a candidate box; Based on the candidate box coordinates and target probability vector corresponding to each window in each scale level, non-maximum suppression is performed on all candidate boxes to obtain the target detection result.

14. The method according to any one of claims 1 to 11, characterized in that, The step of generating image processing results based on the multi-scale attention features corresponding to each window at each scale level includes: The multi-scale attention features corresponding to each window in each scale level are concatenated to obtain multi-scale fusion features; The multi-scale fusion features are upsampled to obtain a target feature map, wherein the target feature map has the same resolution as the image to be processed; Pixel-by-pixel category prediction is performed on the target feature map to obtain the semantic segmentation result of the image to be processed.

15. An image processing apparatus, characterized in that, include: The processing module is used to divide the image to be processed according to n scale levels to obtain a window sequence corresponding to each scale level, wherein the window sequence includes multiple windows belonging to the corresponding scale level, and n is an integer greater than 1; The processing module is further configured to perform feature embedding processing on each window sequence to obtain a window embedding sequence corresponding to each window sequence, wherein the window embedding sequence includes window embedding vectors corresponding to each window belonging to the corresponding scale level. The acquisition module is used to acquire a multi-scale prefix set for the j-th window in the window sequence corresponding to the i-th scale level, based on the window embedding sequence corresponding to the i-th scale level. The multi-scale prefix set includes prefix sequences corresponding to the j-th window from the i-th scale level to the n-th scale level. The prefix sequences include window embedding vectors corresponding to windows preceding the j-th window. Both i and j are integers greater than or equal to 1. The computation module is used to perform attention weighting calculation on the multi-scale prefix set for the j-th window in the window sequence corresponding to the i-th scale level, so as to obtain the multi-scale attention features. The generation module is used to generate image processing results based on the multi-scale attention features corresponding to each window in each scale level.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the image processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the image processing method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the image processing method according to any one of claims 1 to 14.

Images