Multi-modal remote sensing large model reasoning acceleration method, device and equipment based on cooperation of mixed attention and domestic mercuric chloride processor, and storage medium

Abstract

The embodiment of the invention provides a multi-modal remote sensing large model reasoning acceleration method, device and equipment based on cooperation of mixed attention and a domestic mercuric chloride processor, and a storage medium. The method comprises the following steps: acquiring a to-be-processed image and a processing task; according to a preset window size and an overlapping band size, the to-be-processed image is split to obtain split images, and an overlapping region corresponding to the overlapping band size exists between the adjacent split images; according to a pre-trained image block recognition model, feature information and connection information corresponding to the split image are extracted, and an in-window dense attention network model in the image block recognition model is used for capturing local fine context information of the split image; the inter-window block sparse attention network model is used for determining cross-block connection and global sparse context information of the split image; and executing the processing task according to the overlapping region, the feature information and the connection information to obtain an execution result, thereby improving the reasoning acceleration efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, device, and storage medium for accelerating multimodal remote sensing large model inference based on hybrid attention and collaboration with a domestically produced Ascend processor. Background Technology

[0002] With the rapid development of remote sensing technology, the spatial resolution and coverage of remote sensing images are increasing day by day, and images with resolutions of tens of thousands of pixels or even higher have become the norm. When performing intelligent interpretation tasks such as target detection, semantic segmentation, or change detection on such images, the large size leads to increased computational load and high requirements for hardware equipment, resulting in low processing efficiency for remote sensing images. Summary of the Invention

[0003] The purpose of some embodiments of this application is to provide a method, apparatus, device, and storage medium for accelerating inference of multimodal remote sensing large models based on hybrid attention and collaboration with a domestic Ascend processor. Through the technical solutions of the embodiments of this application, an image to be processed and a processing task are acquired, wherein the resolution of the image to be processed is greater than a preset value; the image to be processed is split according to a preset window size and an overlap band size to obtain split images, and adjacent split images have an overlap region corresponding to the overlap band size; feature information and connection information corresponding to the split images are extracted according to a pre-trained patch recognition model, wherein the pre-trained patch recognition model includes at least an in-window dense attention network model and an inter-window sparse attention network model, wherein the in-window dense attention network model is used to capture local fine context information of the split images, and the inter-window sparse attention network model is used to determine cross-block connections and global sparse connections of the split images. The following information describes how, based on the overlapping region, the feature information, and the connection information, the processing task is executed to obtain the execution result corresponding to the image to be processed. In this embodiment, a high-resolution image is split into multiple images by sliding a window, and the split images have overlapping regions. The split images are then input into a patch recognition model. Specifically, the dense attention network model within the window is used to capture the local fine context information of the split images, and the sparse attention network model between the windows is used to determine the cross-block connections and global sparse context information of the split images. At the same time, the information of the overlapping region is stored. In this way, in subsequent calculations, the stored information of the overlapping region, as well as the key information and connection information identified by the patch recognition model, can be directly called to execute the processing task, such as object detection, semantic segmentation, or change detection. This improves the inference acceleration efficiency of multimodal remote sensing large models based on hybrid attention and the collaboration of the domestic Ascend processor.

[0004] Firstly, some embodiments of this application provide a method for accelerating inference of large multimodal remote sensing models based on hybrid attention and collaboration with a domestically produced Ascend processor, including: Acquire the image to be processed and the processing task, wherein the resolution of the image to be processed is greater than a preset value; The image to be processed is split according to the preset window size and the overlap band size to obtain split images, and adjacent split images have an overlap area corresponding to the overlap band size; Based on a pre-trained tile recognition model, feature information and connection information corresponding to the split image are extracted. The pre-trained tile recognition model includes at least an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image. Based on the overlapping region, the feature information, and the connection information, the processing task is executed to obtain the execution result corresponding to the image to be processed.

[0005] Some embodiments of this application perform window sliding splitting on high-resolution images, with overlapping areas between the split images. The split images are then input into a patch recognition model. Specifically, the dense attention network model within the window is used to capture the local fine context information of the split images, while the sparse attention network model between the windows is used to determine the cross-block connections and global sparse context information of the split images. Simultaneously, the information of the overlapping areas is stored. In this way, during subsequent calculations, the stored information of the overlapping areas, as well as the key information and connection information identified by the patch recognition model, can be directly called to perform the processing tasks, such as object detection, semantic segmentation, or change detection. This improves the inference acceleration efficiency of multimodal remote sensing large models based on hybrid attention and the collaboration of the domestic Ascend processor.

[0006] Optionally, the step of splitting the image to be processed according to a preset window size and an overlap band size to obtain split images, wherein adjacent split images have an overlap region corresponding to the overlap band size, includes: Based on the first size information, pyramid level, and target density information of the image to be processed, determine the second size information of the split image and the third size information of the overlapping region; Based on the second size information and the third size information, the image to be processed is split to obtain the split images, and a spatial index and a sideband mask are generated for each split image.

[0007] Some embodiments of this application divide the entire image into a series of tiles with overlapping boundaries and generate a spatial index and sideband mask for each tile, providing structured information for subsequent sparse attention and caching mechanisms.

[0008] Optionally, the tile recognition model is obtained in the following manner: Collect image sample data; Based on the image sample data, a neural network model is trained to obtain the image patch recognition model. The backbone network of the neural network model is a Transformer network, and the neural network model includes an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image.

[0009] Some embodiments of this application avoid simultaneous loading of all image features by using sliding window segmentation, sparse attention, and KV caching, significantly reducing peak memory usage and improving inference efficiency for ultra-high resolution images.

[0010] Optionally, the method further includes: The overlapping regions in the split image are cached according to the key and value vectors.

[0011] Optionally, caching the overlapping regions in the split image according to key-value vectors includes: For each overlapping region, the content of the overlapping region is cached based on the key boundary information, value boundary information, and position offset mapping information.

[0012] In some embodiments of this application, a dedicated key and value vector cache is established for the edge regions of processed tiles. When calculating the current tile, relevant key-value features are retrieved and reused from the cache of its adjacent and already processed tiles based on the positional offset mapping relationship. This mechanism avoids redundant calculation of overlapping areas and enhances the feature coherence between tiles.

[0013] Secondly, some embodiments of this application provide a multimodal remote sensing large model inference acceleration device based on hybrid attention and collaboration with a domestically produced Ascend processor, including: The acquisition module is used to acquire the image to be processed and the processing task, wherein the resolution of the image to be processed is greater than a preset value; The splitting module is used to split the image to be processed according to a preset window size and an overlap band size to obtain split images, and adjacent split images have an overlap area corresponding to the overlap band size; The recognition module is used to extract feature information and connection information corresponding to the split image based on a pre-trained tile recognition model. The pre-trained tile recognition model includes at least an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image. The execution module is used to execute the processing task based on the overlapping region, the feature information, and the connection information, and obtain the execution result corresponding to the image to be processed.

[0014] Some embodiments of this application perform window sliding splitting on high-resolution images, with overlapping areas between the split images. The split images are then input into a patch recognition model. Specifically, the dense attention network model within the window is used to capture the local fine context information of the split images, while the sparse attention network model between the windows is used to determine the cross-block connections and global sparse context information of the split images. Simultaneously, the information of the overlapping areas is stored. In this way, during subsequent calculations, the stored information of the overlapping areas, as well as the key information and connection information identified by the patch recognition model, can be directly called to perform the processing tasks, such as object detection, semantic segmentation, or change detection. This improves the inference acceleration efficiency of multimodal remote sensing large models based on hybrid attention and the collaboration of the domestic Ascend processor.

[0015] Optionally, the splitting module is used for: Based on the first size information, pyramid level, and target density information of the image to be processed, determine the second size information of the split image and the third size information of the overlapping region; Based on the second size information and the third size information, the image to be processed is split to obtain the split images, and a spatial index and a sideband mask are generated for each split image.

[0016] Some embodiments of this application divide the entire image into a series of tiles with overlapping boundaries and generate a spatial index and sideband mask for each tile, providing structured information for subsequent sparse attention and caching mechanisms.

[0017] Optionally, the device further includes a training module, the training module being used for: Collect image sample data; Based on the image sample data, a neural network model is trained to obtain the image patch recognition model. The backbone network of the neural network model is a Transformer network, and the neural network model includes an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image.

[0018] Some embodiments of this application avoid simultaneous loading of all image features by using sliding window segmentation, sparse attention, and KV caching, significantly reducing peak memory usage and improving inference efficiency for ultra-high resolution images.

[0019] Optionally, the execution module is configured to: The overlapping regions in the split image are cached according to the key and value vectors.

[0020] Optionally, the execution module is configured to: For each overlapping region, the content of the overlapping region is cached based on the key boundary information, value boundary information, and position offset mapping information.

[0021] In some embodiments of this application, a dedicated key and value vector cache is established for the edge regions of processed tiles. When calculating the current tile, relevant key-value features are retrieved and reused from the cache of its adjacent and already processed tiles based on the positional offset mapping relationship. This mechanism avoids redundant calculation of overlapping areas and enhances the feature coherence between tiles.

[0022] Thirdly, some embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the multimodal remote sensing large model inference acceleration method based on hybrid attention and collaboration with the domestic Ascend processor as described in any embodiment of the first aspect.

[0023] Optionally, the electronic device includes a fusion operator, which is achieved by fusioning QK... T It is obtained by fusing Scale, Softmax, Dropout, and multiplication by V.

[0024] Optionally, the electronic device includes a tile-level double buffering mechanism.

[0025] Fourthly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the multimodal remote sensing large model inference acceleration method based on hybrid attention and collaboration with a domestic Ascend processor as described in any embodiment of the first aspect.

[0026] Fifthly, some embodiments of this application provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the multimodal remote sensing large model inference acceleration method based on hybrid attention and collaboration with the domestic Ascend processor as described in any embodiment of the first aspect. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating a method for accelerating inference of a large multimodal remote sensing model based on hybrid attention and collaboration with a domestic Ascend processor, provided for an embodiment of this application; Figure 2 A flowchart illustrating another method for accelerating inference of a multimodal remote sensing large model based on the collaboration of hybrid attention and domestic Ascend processor, provided for an embodiment of this application; Figure 3 This application provides an embodiment of the block sparse-sliding window hybrid attention mechanism. Figure 4 A schematic diagram of the structure of a multimodal remote sensing large model inference acceleration device based on hybrid attention and collaboration with a domestic Ascend processor, provided for embodiments of this application; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] With the rapid development of aerial remote sensing technology, the spatial resolution and coverage of remote sensing images are increasing daily, and images with resolutions of tens of thousands of pixels or even higher have become commonplace. When performing intelligent interpretation tasks such as object detection, semantic segmentation, or change detection on these images, deep learning models based on the Transformer architecture face significant challenges: High computational complexity: Standard self-attention mechanisms have... The time and space complexity is given by N, where N is the number of pixels or tokens. For large images, the value of N is enormous, causing the computational load and memory usage to increase exponentially, far exceeding the capacity of existing hardware.

[0032] Peak memory limit: Full-map inference requires loading a huge feature map into video memory at once, resulting in extremely high peak video memory usage, which is difficult to meet even for high-end accelerator cards.

[0033] Loss of contextual information and boundary effects: To avoid the above problems, the industry generally adopts the sliding window (Tiling) strategy, which divides a large image into several sub-tiles for independent processing. However, this method severs the contextual connections between tiles, leading to problems such as missed detections, false detections, or discontinuous segmentation of targets in the boundary areas due to incomplete information, which in particular affects the recall rate of small targets.

[0034] Insufficient hardware utilization: While existing sparse attention or multi-scale strategies reduce theoretical complexity at the algorithm level, they often lack deep adaptation to specific hardware architectures (such as the AI ​​Core of the Ascend processor and the TBE operator development environment) in engineering deployment. General-purpose operator implementations cannot fully utilize the hardware's computation and storage layers, causing scheduling overhead and data transfer latency between operators to become new performance bottlenecks, limiting overall throughput.

[0035] Caching mechanisms are not well-suited: Key-value caching (KV caching) technology, widely used in natural language processing, is mainly used to accelerate the generation of long sequences in autoregressive models. However, if it is simply applied to two-dimensional images, it lacks a targeted design for the spatial adjacency relationships and location information of tiles, making it difficult to efficiently reuse the features of adjacent tiles.

[0036] In summary, there is an urgent need for an innovative solution that can balance global context and local precision while deeply collaborating with dedicated AI processor architectures such as Ascend, in order to systematically solve problems such as high memory usage, high computational complexity, and boundary effects in remote sensing large-scale image inference while ensuring model accuracy.

[0037] In view of this, some embodiments of this application provide a method for accelerating inference of a multimodal remote sensing large model based on hybrid attention and collaboration with a domestic Ascend processor. The method includes: acquiring an image to be processed and a processing task, wherein the resolution of the image to be processed is greater than a preset value; splitting the image to be processed according to a preset window size and an overlap band size to obtain split images, wherein adjacent split images have an overlap region corresponding to the overlap band size; extracting feature information and connection information corresponding to the split images according to a pre-trained patch recognition model, wherein the pre-trained patch recognition model includes at least an in-window dense attention network model and an inter-window block sparse attention network model, the in-window dense attention network model being used to capture local fine context information of the split images, and the inter-window block sparse attention network model being used to determine cross-block connections and global sparse context information of the split images; and extracting feature information and connection information corresponding to the split images according to the overlap band size. The domain, the feature information, and the connection information are used to perform processing tasks and obtain the execution results corresponding to the image to be processed. In this embodiment, a high-resolution image is split into windows with overlapping areas. The split images are then input into a tile recognition model. Specifically, the dense attention network model within the window is used to capture the local fine context information of the split images, and the sparse attention network model between the windows is used to determine the cross-block connections and global sparse context information of the split images. At the same time, the information of the overlapping areas is stored. In this way, in subsequent calculations, the stored information of the overlapping areas, as well as the key information and connection information identified by the tile recognition model, can be directly called to perform processing tasks, such as object detection, semantic segmentation, or change detection. This improves the inference acceleration efficiency of multimodal remote sensing large models based on hybrid attention and the collaboration of the domestic Ascend processor.

[0038] like Figure 1 As shown, embodiments of this application provide a method for accelerating inference of large multimodal remote sensing models based on hybrid attention and collaboration with a domestically developed Ascend processor. This method includes: S101. Obtain the image to be processed and the processing task. The resolution of the image to be processed is greater than a preset value. Specifically, the terminal device acquires an image to be processed, which may be a remote sensing image. The processing task includes at least object detection, semantic segmentation, or change detection. The terminal device is equipped with an Ascend processor.

[0039] S102. Based on the preset window size and overlap band size, the image to be processed is split to obtain split images, and adjacent split images have an overlapping area corresponding to the overlap band size. Specifically, the terminal device is equipped with an adaptive segmentation and overlapping sliding window strategy, that is, the window size of the sliding window is adaptively determined based on the size of the input remote sensing image (i.e., the image to be processed), the pyramid level, and the estimated target density. That is, the preset window size and the overlapping area size. The entire image is divided into a series of tiles with overlapping boundaries, i.e., split images. Spatial indexes and sideband masks are generated for each split image to provide structured information for subsequent sparse attention and caching mechanisms.

[0040] S103. Based on the pre-trained tile recognition model, extract the feature information and connection information corresponding to the split image. The pre-trained tile recognition model includes at least an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image.

[0041] Specifically, a tile recognition model is pre-built on the terminal device. This tile recognition model includes dense attention calculation within the window. Within each independent tile (window), standard dense self-attention calculation is performed on the query, key, and value vectors to ensure that the pixels within the window can fully interact and capture fine contextual information in the local area. This contextual information is a multi-dimensional matrix, i.e., tensor information.

[0042] This tile recognition model is trained based on a large model of hybrid attention and multimodal remote sensing data. It also includes inter-window sparse attention computation, defining a structured sparse connection topology (e.g., ring topology or cross topology) such that the query vector of each tile only undergoes attention computation with the key / value vectors of a few topologically adjacent tiles. By using a predefined sparse index, it avoids brute-force scoring globally, reducing the complexity of inter-window information interaction from... Reduced to approximately linear , where k is the number of neighbors in a sparse connection.

[0043] The terminal device inputs the segmented image into a pre-trained tile recognition model to extract the feature information and connection information corresponding to the segmented image. In other words, it can identify key information in the segmented image, such as the content of target detection, and can also identify the identifiers and number of other segmented images connected to a certain segmented image.

[0044] S104. Based on the overlapping region, the feature information, and the connection information, perform the processing task to obtain the execution result corresponding to the image to be processed.

[0045] Specifically, when performing processing tasks, the terminal device prioritizes acquiring the content of the already stored overlapping regions. This eliminates the need to process the original image, thus accelerating the efficiency of multimodal remote sensing large model inference based on hybrid attention and the collaboration of the domestic Ascend processor. Then, based on the connection information, it acquires other split images connected to the current split image and processes them until all split images have been processed, obtaining the execution result corresponding to the image to be processed.

[0046] This application provides a method and systematic implementation for accelerating remote sensing large-scale image inference based on a hybrid attention approach of block sparsity and sliding window. Through a collaborative design of "dense within the window, structured sparsity between windows, cross-block feature cache reuse, and Ascend end-operator fusion and asynchronous pipelined processing," it reduces peak memory usage, shortens end-to-end latency, and improves throughput while maintaining accuracy, thus enhancing the processing performance of boundary targets. It profoundly utilizes the "similarity gap" characteristic of attention scores. That is, spatially adjacent or content-related tiles have significantly higher attention scores than unrelated tiles. This invention uses structured adjacency relationships (ring / cross topology) to approximate the global Top-k sparse selection and strengthens this adjacency advantage through a KV caching mechanism, physically avoiding the huge overhead of global scoring. Combined with the fusion computing capabilities of Ascend hardware, the theoretical complexity advantage is effectively transformed into practical performance gains.

[0047] Some embodiments of this application perform window sliding splitting on high-resolution images, with overlapping areas between the split images. The split images are then input into a patch recognition model. Specifically, an intra-window dense attention network model is used to capture the local fine context information of the split images, while an inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split images. At the same time, the information of the overlapping areas is stored. In subsequent calculations, the stored information of the overlapping areas, as well as the key information and connection information identified by the patch recognition model, can be directly called to perform processing tasks, such as object detection, semantic segmentation, or change detection. This improves the inference acceleration efficiency of multimodal remote sensing large models based on hybrid attention and the collaboration of the domestic Ascend processor.

[0048] Another embodiment of this application further supplements the description of the multimodal remote sensing large model inference acceleration method based on hybrid attention and the collaboration of domestic Ascend processors provided in the above embodiments.

[0049] like Figure 2 As shown, it includes: S1: Adaptive block segmentation and overlapping sliding window; S2 window-based dense attention computation determines whether cross-block interaction is needed. If so, execute S3: inter-window block sparse attention computation; If not, execute S4: cross-block KV cache and position alignment; S5: Ascend operator fusion optimization; S6: Multi-block well travel and boundary constraints; S7: System-level pipelining and memory management; Optionally, the image to be processed is split according to a preset window size and an overlap band size to obtain split images, wherein adjacent split images have an overlap region corresponding to the overlap band size, including: Based on the first size information, pyramid level and target density information of the image to be processed, determine the second size information of the split image and the third size information of the overlapping region; Based on the second and third size information, the image to be processed is split into split images, and a spatial index and sideband mask are generated for each split image.

[0050] Specifically, such as Figure 3 As shown, the embodiments of this application employ the following method to perform adaptive segmentation and overlap parameters: Input the image to be processed, i.e., the large image size. Set up a basic window Overlapping zone Pixel; The window overlap band size (r) is based on the target density of the image. The value is determined adaptively and its range is [8, 128] pixels.

[0051] According to the pyramid hierarchy With target density ,calculate

[0052] in, It is a pyramid hierarchy. For the target density, This is the minimum value of the overlapping band. This represents the maximum value of the overlapping band.

[0053] This application's embodiments generate a block index table and a sideband mask for subsequent sparse connections and KV caching.

[0054] Some embodiments of this application divide the entire image into a series of tiles with overlapping boundaries and generate a spatial index and sideband mask for each tile, providing structured information for subsequent sparse attention and caching mechanisms.

[0055] Optionally, the tile recognition model is obtained in the following way: Collect image sample data; Based on image sample data, a neural network model is trained to obtain a patch recognition model. The backbone network of the neural network model is a Transformer network, and the neural network model includes an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the segmented image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the segmented image.

[0056] Specifically, the embodiments of this application employ hybrid attention computation: Densely packed inside the window: for each section implement;

[0057] Where: A is the attention output matrix, K is the key matrix, Q is the query matrix, d is the feature dimension, T is the transpose operation, and V is the value matrix.

[0058] Sparse window spacing: radius of structural ring and cross bandwidth ( piece, (blocks), sparse attention is computed only for adjacent blocks within the band, and fast addressing is achieved using index tables and LUTs; Sparse attention can be used to employ an approximate softmax (such as a piecewise logarithmic scale) to improve numerical stability and bandwidth utilization.

[0059] Attention calculation adopts a mixed precision calculation scheme, in which the main calculation path uses fp16 or bf16 format, the accumulation process uses fp32 format, and the sparse calculation path supports INT8 quantization.

[0060] During the model training phase, a boundary consistency loss is introduced into the prediction results of tiles in the overlapping area to constrain the consistency of prediction results of adjacent tiles.

[0061] Some embodiments of this application avoid simultaneous loading of all image features by using sliding window segmentation, sparse attention, and KV caching, significantly reducing peak memory usage and improving inference efficiency for ultra-high resolution images.

[0062] Optionally, the method further includes: Overlapping regions in the split image are cached according to key and value vectors. The key-value caching mechanism includes a sideband key-value table and a positional offset mapping table to achieve accurate alignment and fast indexing of cross-window features.

[0063] Optionally, overlapping regions in the split image are cached according to key and value vectors, including: For each overlapping region, the content of the overlapping region is cached based on the key boundary information, value boundary information, and position offset mapping information.

[0064] Specifically, the terminal device also provides a cross-block key-value cache (KV Cache) and position alignment mechanism, establishing dedicated key and value vector caches for the edge regions of processed tiles. When calculating the current tile, relevant key-value features are retrieved and reused from its adjacent and already processed tile caches based on position offset mapping relationships. This mechanism avoids redundant calculations of overlapping areas and enhances the feature coherence between tiles.

[0065] Cross-block KV caching and position alignment: Maintain KV sideband cache for each block: and its position offset mapping ; When processing blocks When, prioritize from adjacent blocks Retrieve aligned cache Reduce repeated memory accesses; The lifecycle of the KV cache follows a state machine of "write → freeze → reuse → invalidate", and is reused in conjunction with the video memory pool.

[0066] In some embodiments of this application, a dedicated key and value vector cache is established for the edge regions of processed tiles. When calculating the current tile, relevant key-value features are retrieved and reused from the cache of its adjacent and already processed tiles based on the positional offset mapping relationship. This mechanism avoids redundant calculation of overlapping areas and enhances the feature coherence between tiles.

[0067] The terminal device used in this application embodiment employs operator fusion and optimization for the Ascend processor. At the Ascend AICore level, the Tensor Byte Engine (TBE) is used to integrate the core chain of attention computation (e.g., Matrix multiplication, scaling, softmax activation, and V matrix multiplication are fused into a single high-performance operator. Within this fused operator, a tile-level double buffering mechanism is implemented to hide data prefetch latency, and a memory access mode is designed to avoid shared memory bank conflicts, maximizing hardware computational efficiency. The fused operator runs on the Ascend processor's AI Core, performing the multiplication as a single-core task. The entire computation chain from calculation to final output is optimized to avoid bank conflicts in shared memory. The fusion operator module integrates two computation paths, dense and sparse, and supports dynamic gating backoff based on online sparsity.

[0068] QK T Scale, Softmax, Dropout (optional), and multiplication by V are merged into a single TBE. Kernel: Tile size t_k, t_v are adaptively selected based on d and N to ensure L1 / UB capacity is aligned with the vector register; Double buffering: tile( During calculation, tiles are pre-fetched. ); Conflict avoidance: resolved through interline padding and interleaved memory access; Mixed precision: fp16 / bf16 backbone, fp32 for accumulation, supports INT8 sparse paths; Operator interface: supports both dense and sparse paths and direct connection to KV cache.

[0069] In a multi-processor (multi-GPU) environment, a hybrid parallelism strategy combining data parallelism and block parallelism (a form of tensor model parallelism) is employed. The Collective Communication Library (HCCL) is used to efficiently synchronize necessary sideband features across different processors. Furthermore, during model training or knowledge distillation, a boundary consistency loss function can be introduced into the prediction results of tiles in overlapping regions to encourage the model to learn to generate smooth and consistent boundary predictions.

[0070] Parallel strategy: Within a single card: multiple streams run in parallel, with computation overlapping with H2D / D2H; Multi-GPU configuration: a hybrid approach of block parallelism and data parallelism; Communication: HCCL performs AllGather / ReduceScatter on the necessary sideband features across cards, supporting 8-bit quantized transmission; Boundary Consistency: During the training or distillation phase, a consistency loss L_boundary is introduced for the prediction of adjacent blocks in the overlapping area to reduce fragmentation.

[0071] The AscendCL multi-stream parallel mechanism is used to overlap data transmission and core computation, and combined with HostPinned memory and DVPP / AIPP hardware preprocessing units to form an end-to-end pipeline.

[0072] Furthermore, the terminal device also includes system-level pipeline and memory management. Multiple execution streams are created using the AscendCL (Ascend Computing Language) interface to achieve asynchronous parallelism in data preprocessing, inter-device data transfer (H2D / D2H), and core computation, forming a highly efficient end-to-end pipeline. Simultaneously, the lifecycle of various levels of cache (such as KV cache and feature maps) is finely managed. Combined with memory pooling technology, dynamic reuse and level control of memory are achieved to ensure stable system operation. Pipeline and memory management.

[0073] Furthermore, the terminal device also includes page-locked buffers and DVPP / AIPP preprocessing pipelines; memory level conservation: lifecycle labeling and reuse are applied to features, KV caches, and temporary tile buffers; anomalies and rollback: when sparsity is below a threshold It automatically reverts to a fully dense window to ensure accuracy.

[0074] Compared with the prior art, the present invention achieves the following significant beneficial effects through the organic combination of the above technical solutions: Significantly reduced memory usage: By using sliding window partitioning, sparse attention, and KV caching, the simultaneous loading of all image features is avoided, significantly reducing peak memory usage and making inference of ultra-high resolution images possible.

[0075] Improve throughput and reduce latency: Ascend edge operator fusion and asynchronous pipeline reduce CPU / AI Core interaction and memory access round trips, improving throughput by about 20-60% and reducing end-to-end latency by about 20-40%.

[0076] Maintaining stable accuracy: Hybrid attention and boundary consistency constraints balance efficiency and context integrity, with mAP / mIoU changes of no more than 0.3 percentage points for tasks such as detection / segmentation.

[0077] Improved performance for boundaries and small targets: Cross-block KV reuse and seamless fusion alleviate boundary fragmentation, and the recall of small targets (<16 pixels) remains basically stable.

[0078] Example 1: Target Detection (DOTA / DIOR) Dataset and input: DOTA / DIOR partitioning, evaluation resolution maintains the original image. A scale pyramid {1.0×, 0.5×} is constructed; Sliding window segmentation: window w=h=1024, overlap band r=64; boundary expansion and over-boundary filling use reflection filling.

[0079] For example, the window size is: w, h ∈ [256, 2048]; the overlap band is: r ∈ [8, 128]. Sparse topology: ring radius b∈[1,4]; cross bandwidth c∈[1,3]; Mixed precision: fp16 / bf16 accumulated to fp32, optional INT8; Sparse path: communication quantization bit width ∈ {8, 12, 16}; boundary consistency loss weight λ ∈ [0.1, 1.0].

[0080] The structured sparse topology is composed of a ring topology and a cross topology, wherein the ring radius (b) ranges from [1, 4] tiles, and the cross bandwidth (c) ranges from [1, 3] tiles.

[0081] The backbone is a variant of Transformer (hierarchical pyramid and local window attention), with channel dimension d∈{96,128} and multi-head number h∈{3,4}; The head supports two detection branches: horizontal bounding box or rotated bounding box (RBox). The NMS / RBox-NMS IoU threshold is ∈ [0.4, 0.6]. Quantization (optional): INT8 is used only for QKV projection of sparse paths, using symmetric calibration; Synthesizer: Attention Main Link (QK) (Scale, Softmax, ×V) are merged into a single-core TBE, and the tile depth is adaptive according to the UB capacity; Scheduling: Two streams (H2D / preprocessing and Compute / postprocessing) run in parallel, with a double-buffered queue depth of 2–3.

[0082] Sparse topology and KV multiplexing: Sparse topology: ring radius b=2, cross bandwidth c=1; KV buffer: Maintain K_edge / V_edge for overlapping bands, and use relative coordinates + integer pixel offset for position offset; Example 2: Semantic Segmentation (LoveDA) Dataset and input: Partitioning, single-scale reasoning; window w=h=768, overlap band r=48, pyramid optional off: The main trunk is the same as above, with lightweight multiscale fusion (such as the FPN variant) added to the neck. Mixed precision: bf16 main, fp32 accumulation; Sparse topology: b=1, c=2; KV cache and weight fusion are used in the same detection task; Post-processing: Small area noise removal and boundary refinement (either morphological opening / closing or lightweight CRF can be selected).

[0083] Result range: mIoU change ≤ 0.2pp; throughput increase of approximately 1.1×; peak memory usage decrease of approximately 40% (related to the average size of the input image and the number of categories), subject to specific test results.

[0084] Ablation projects: Disable key-value caching / disable sparse paths / disable fusion operators; The boundary fusion function is replaced with linear weights, and the balance between boundary mIoU and inference latency is compared.

[0085] Example 3: Change Detection (WHU-CD) Dataset and input: WHU-CD dual-temporal remote sensing imagery, aligned and segmented according to the same index; window w=h=768, r=48.

[0086] Model and Deployment: A dual-branch shared backbone (Siamese) is used, with features extracted separately in phases A and B. Cross-scale global tokens (optional) are used to capture low-frequency consistency; the differential head employs channel-level differential and small MLP classification; The sparse topology and KV cache adopt a single-phase setting and share the overlap band alignment relationship between phases; Loss: Binary classification crossover and boundary consistency loss, with a weight λ=0.3.

[0087] Result range: The F1 score at the boundary is improved (the number of false positives / false negatives at the boundary is reduced), while the overall F1 score remains basically flat or improves slightly (≤0.3pp); the throughput increases by about 1.1; and the peak memory performance decreases by about 25–40%.

[0088] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0089] Another embodiment of this application provides a multimodal remote sensing large model inference acceleration device based on hybrid attention and collaboration with a domestic Ascend processor, used to execute the multimodal remote sensing large model inference acceleration method based on hybrid attention and collaboration with a domestic Ascend processor provided in the above embodiment.

[0090] like Figure 4 The diagram shown is a structural schematic of a multimodal remote sensing large-scale model inference acceleration device based on hybrid attention and collaboration with a domestic Ascend processor, provided in an embodiment of this application. This multimodal remote sensing large-scale model inference acceleration device includes an acquisition module 401, a splitting module 402, an identification module 403, and an execution module 404, wherein: The acquisition module 401 is used to acquire the image to be processed and the processing task. The resolution of the image to be processed is greater than a preset value. The splitting module 402 is used to split the image to be processed according to the preset window size and the overlap band size to obtain split images, and adjacent split images have an overlap area corresponding to the overlap band size; The recognition module 403 is used to extract feature information and connection information corresponding to the split image according to the pre-trained tile recognition model. The pre-trained tile recognition model includes at least an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image. The execution module 404 is used to perform processing tasks based on the overlapping region, the feature information and the connection information, and obtain the execution result corresponding to the image to be processed.

[0091] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0092] Some embodiments of this application perform window sliding splitting on high-resolution images, with overlapping areas between the split images. The split images are then input into a patch recognition model. Specifically, an intra-window dense attention network model is used to capture the local fine context information of the split images, while an inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split images. At the same time, the information of the overlapping areas is stored. In subsequent calculations, the stored information of the overlapping areas, as well as the key information and connection information identified by the patch recognition model, can be directly called to perform processing tasks, such as object detection, semantic segmentation, or change detection. This improves the inference acceleration efficiency of multimodal remote sensing large models based on hybrid attention and the collaboration of the domestic Ascend processor.

[0093] Another embodiment of this application further illustrates the multimodal remote sensing large model inference acceleration device based on hybrid attention and collaboration with the domestic Ascend processor provided in the above embodiments.

[0094] Optionally, the module can be split for: Based on the first size information, pyramid level and target density information of the image to be processed, determine the second size information of the split image and the third size information of the overlapping region; Based on the second and third size information, the image to be processed is split into split images, and a spatial index and sideband mask are generated for each split image.

[0095] Some embodiments of this application divide the entire image into a series of tiles with overlapping boundaries and generate a spatial index and sideband mask for each tile, providing structured information for subsequent sparse attention and caching mechanisms.

[0096] Optionally, the device further includes a training module, which is used for: Collect image sample data; Based on image sample data, a neural network model is trained to obtain a patch recognition model. The backbone network of the neural network model is a Transformer network, and the neural network model includes an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the segmented image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the segmented image.

[0097] Some embodiments of this application avoid simultaneous loading of all image features by using sliding window segmentation, sparse attention, and KV caching, significantly reducing peak memory usage and improving inference efficiency for ultra-high resolution images.

[0098] Optionally, the execution module is used for: The overlapping regions in the split image are cached according to the key and value vectors.

[0099] Optionally, the execution module is used for: For each overlapping region, the content of the overlapping region is cached based on the key boundary information, value boundary information, and position offset mapping information.

[0100] In some embodiments of this application, a dedicated key and value vector cache is established for the edge regions of processed tiles. When calculating the current tile, relevant key-value features are retrieved and reused from the cache of its adjacent and already processed tiles based on the positional offset mapping relationship. This mechanism avoids redundant calculation of overlapping areas and enhances the feature coherence between tiles.

[0101] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0102] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0103] This application also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it can implement the operation of any embodiment of the multimodal remote sensing large model inference acceleration method based on hybrid attention and domestic Ascend processor collaboration provided in the above embodiments.

[0104] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it can implement the operation of any embodiment of the method corresponding to the method provided in the above embodiments of the multimodal remote sensing large model inference acceleration method based on hybrid attention and domestic Ascend processor collaboration.

[0105] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 and executes the program via a bus 530, it can implement any of the methods included in the above-described method for accelerating multimodal remote sensing large model inference based on hybrid attention and collaboration with a domestic Ascend processor.

[0106] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0107] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0108] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0109] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for accelerating inference of large multimodal remote sensing models based on hybrid attention and collaboration with a domestic Ascend processor, characterized in that, The method includes: Acquire the image to be processed and the processing task, wherein the resolution of the image to be processed is greater than a preset value; The image to be processed is split according to the preset window size and the overlap band size to obtain split images, and adjacent split images have an overlap area corresponding to the overlap band size; Based on a pre-trained tile recognition model, feature information and connection information corresponding to the split image are extracted. The pre-trained tile recognition model includes at least an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image. Based on the overlapping region, the feature information, and the connection information, the processing task is executed to obtain the execution result corresponding to the image to be processed.

2. The method for accelerating inference of large multimodal remote sensing models based on hybrid attention and the collaboration of a domestic Ascend processor, as described in claim 1, is characterized in that... The step of splitting the image to be processed according to a preset window size and an overlap band size to obtain split images, wherein adjacent split images have an overlap region corresponding to the overlap band size, includes: Based on the first size information, pyramid level, and target density information of the image to be processed, determine the second size information of the split image and the third size information of the overlapping region; Based on the second size information and the third size information, the image to be processed is split to obtain the split images, and a spatial index and a sideband mask are generated for each split image.

3. The method for accelerating inference of large multimodal remote sensing models based on hybrid attention and collaboration with the domestic Ascend processor as described in claim 1, characterized in that, The tile recognition model is obtained in the following way: Collect image sample data; Based on the image sample data, a neural network model is trained to obtain the image patch recognition model. The backbone network of the neural network model is a Transformer network, and the neural network model includes an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image.

4. The method for accelerating inference of large multimodal remote sensing models based on hybrid attention and collaboration with the domestic Ascend processor as described in claim 1, characterized in that, The method further includes: The overlapping regions in the split image are cached according to the key and value vectors.

5. The method for accelerating inference of large multimodal remote sensing models based on hybrid attention and collaboration with the domestic Ascend processor as described in claim 4, characterized in that, The step of caching the overlapping regions in the split image according to the key and value vectors includes: For each overlapping region, the content of the overlapping region is cached based on key boundary information, value boundary information, and position offset mapping information.

6. A multimodal remote sensing large-scale model inference acceleration device based on hybrid attention and collaboration with a domestically produced Ascend processor, characterized in that, The device includes: The acquisition module is used to acquire the image to be processed and the processing task, wherein the resolution of the image to be processed is greater than a preset value; The splitting module is used to split the image to be processed according to a preset window size and an overlap band size to obtain split images, and adjacent split images have an overlap area corresponding to the overlap band size; The recognition module is used to extract feature information and connection information corresponding to the split image based on a pre-trained tile recognition model. The pre-trained tile recognition model includes at least an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image. The execution module is used to execute the processing task based on the overlapping region, the feature information, and the connection information, and obtain the execution result corresponding to the image to be processed.

7. The multimodal remote sensing large model inference acceleration device based on hybrid attention and the collaboration of a domestic Ascend processor as described in claim 6, characterized in that, The splitting module is used for: Based on the first size information, pyramid level, and target density information of the image to be processed, determine the second size information of the split image and the third size information of the overlapping region; Based on the second size information and the third size information, the image to be processed is split to obtain the split images, and a spatial index and a sideband mask are generated for each split image.

8. The multimodal remote sensing large model inference acceleration device based on hybrid attention and domestic Ascend processor collaboration as described in claim 6, characterized in that, The device further includes a training module, the training module being used for: Collect image sample data; Based on the image sample data, a neural network model is trained to obtain the image patch recognition model. The backbone network of the neural network model is a Transformer network, and the neural network model includes an in-window dense attention network model and an inter-window sparse attention network model. The in-window dense attention network model is used to capture the local fine context information of the split image, and the inter-window sparse attention network model is used to determine the cross-block connections and global sparse context information of the split image.

9. An electronic device, characterized in that, The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the multimodal remote sensing large model inference acceleration method based on hybrid attention and collaboration with the domestic Ascend processor as described in any one of claims 1-5.

10. The electronic device according to claim 9, characterized in that, The electronic device includes a fusion operator, which is achieved by fusion of QK... T It is obtained by fusing Scale, Softmax, Dropout, and multiplication by V.

11. The electronic device according to claim 10, characterized in that, The electronic device includes a tile-level double buffering mechanism.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, characterized in that, when the program is executed by a processor, it can implement the multimodal remote sensing large model inference acceleration method based on hybrid attention and collaboration with the domestic Ascend processor as described in any one of claims 1-5.

Images