A large model memory optimization method and device, and a storage medium

Abstract

The application discloses a large model video memory optimization method and device and a storage medium. It belongs to the technical field of image processing. The method is applied to a computing system deployed with an image segmentation model, and the computing system comprises a CPU, a GPU and a memory. The method comprises the following steps: the CPU divides an input image into a plurality of mutually non-overlapping image blocks and stores the image blocks into the memory; the GPU sequentially loads a single image block to perform encoding operation of the image segmentation model, obtains a feature map and stores the feature map into the memory; the CPU controls the GPU to interactively fuse all the feature maps, obtains a feature fusion submap and stores the feature fusion submap into the memory; the GPU sequentially loads a single feature fusion submap to perform decoding operation of the image segmentation model, obtains a decoding subimage and stores the decoding subimage into the memory; and the CPU splices all the decoding subimages after obtaining the decoding subimages to obtain a segmentation result image. Thus, the real-time occupation pressure of the GPU video memory is greatly reduced under the premise of guaranteeing the model segmentation accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a large model memory optimization method and device and a storage medium. BACKGROUND

[0002] In medical image detection services, CNN deep learning models are widely used in core scenarios such as lesion identification, tissue delineation, and disease diagnosis assistance due to their excellent feature extraction and segmentation capabilities. Hospital servers often need to run multiple such deep learning models in parallel to respond to different departments' CT, MRI, ultrasound, and other image detection requests. However, a single CNN model consumes GB-level memory when processing high-resolution medical images, while hospital server graphics cards typically have less than 24G of memory, which is limited. In a multi-model parallel scenario, if memory usage is not effectively controlled, it is easy to cause memory overflow, resulting in model errors and image detection task interruptions, which not only affects service continuity and processing efficiency, but also may delay clinical diagnosis.

[0003] For example, the patent with the publication number CN114359300A and the invention name of an image segmentation model optimization method, device, system, and storage medium, wherein the method fully utilizes the boundary point suppression method, the dynamic class weighting method, and the truncated cross-entropy loss function to calculate a balanced loss function, and uses the value of the balanced loss function to optimize the image segmentation model; the balanced loss function composed of the combination of the three algorithms solves the problem of unbalanced loss function values in the training of the industrial product surface defect image segmentation model, and ultimately significantly improves the model training stability, recognition accuracy, and further improves the model's versatility in different tasks and scenarios, providing convenience for image recognition of product surface defects.

[0004] For example, the patent with the publication number CN114627132A and the invention name of a model optimization training method and device based on instance segmentation, including a model optimization console, an instance image segmentation simulator, a model image camera, and an image model detection module. The optimization supplement module is set up to retrieve the instance segmentation image stored in the data storage through the model optimization console, detect the retrieved instance segmentation image through the image model detection module in the model optimization console, and select the instance segmentation image with appropriate pixel, color saturation, hue, and brightness as the to-be-optimized image through the image screening processing module. On the one hand, the loss calculation module calculates the optimization value required by the to-be-optimized image, and on the other hand, the optimization supplement module supplements the missing value of the to-be-optimized image. The optimized model image is displayed on the model training display through the image transmission module, achieving the optimization effect of the training model image in this way.

[0005] To reduce peak memory usage, existing technologies primarily employ optimization within the model itself, known as operator tiling. The core of this method is to divide the large feature map to be processed in a layer into multiple tiles during convolution and other operator computations, after inputting the entire image into the model. The layer's computation is then completed through multiple calculations. Existing operator tiling methods mainly fall into two strategies:

[0006] (1) Non-overlapping operator chunking: This strategy divides the feature map of the current layer into multiple non-overlapping continuous sub-blocks during model computation, and performs convolution operations on each sub-block independently. This method can minimize the instantaneous GPU memory requirements of single-layer computation. However, since the convolution kernel relies on all input values ​​within its receptive field when calculating the output pixel, pixels in the sub-block boundary region cannot access the necessary context in adjacent sub-blocks, resulting in computation results that differ from those based on the entire feature map. Figure One The results obtained from this calculation are biased. This bias propagates and accumulates layer by layer in the network, directly impairing the accuracy and boundary consistency of image segmentation.

[0007] (2) Overlapping operator blocks: To correct the above deviation, this strategy forces adjacent sub-blocks to contain an overlap region (Halo) related to the convolution kernel size when segmenting the feature map, ensuring that each sub-block computation has a complete context, thereby maintaining consistency with the whole image computation. However, this introduces new costs: First, overlapping data is repeatedly stored, weakening the actual benefits of memory optimization; second, overlapping regions are repeatedly computed in adjacent sub-blocks, generating significant redundant computational overhead and greatly increasing the processing latency of the segmentation task.

[0008] Taking a 2D convolution operator as an example, assume the input image has dimensions in FP32 format. Convolution operators With stride=1 and padding=1, the output image remains the same. If directly to If the input is used to compute the convolution, then the GPU memory usage is 552B. If an operator block method is used, the image of the input convolution operator is divided into 4 image blocks at the model level. Ignoring boundary overlap, each image block is... The single computation uses 164 bytes of video memory, which is 70% less GPU memory usage compared to inputting the entire image for convolution computation. The boundary of an image patch will have a different calculation result than the whole-image convolution due to the lack of adjacent data.

[0009] If overlapping operator tiles are used to ensure accuracy, redundant calculation and efficiency loss of memory optimization effect will be suffered. Specifically, when the input image size increases significantly due to task requirements, if the overlapping area between adjacent tiles is to be maintained, logical division of the entire image must be performed at the model level, information exchange between tiles must be performed, and finally the complete feature map must be fused. This process not only introduces a large amount of redundant calculation and prolongs the overall processing time, but also weakens the memory optimization effect, and it is difficult to ensure the consistency of the segmentation result with the direct input of the model.

[0010] In summary, the existing operator tile technology is essentially a memory optimization within the model under the premise of inputting the entire image into the model. The fundamental contradiction it faces is that if non-overlapping tiles are used to pursue low memory and high efficiency, the segmentation accuracy will be inevitably lost, and if overlapping tiles are used to ensure accuracy, redundant calculation and efficiency loss of memory optimization effect will be suffered.

[0011] In view of the technical problems in the prior art that the existing operator tile method cannot significantly reduce the memory occupation of the model while ensuring the segmentation performance of the model without losing the segmentation performance of the model when large-scale medical image segmentation is performed in a resource-limited environment, no effective solution has been proposed so far. SUMMARY

[0012] Embodiments of the present disclosure provide a large model memory optimization method and device and a storage medium to at least solve the technical problem in the prior art that the existing operator tile method cannot significantly reduce the memory occupation of the model while ensuring the segmentation performance of the model without losing the segmentation performance of the model when large-scale medical image segmentation is performed in a resource-limited environment.

[0013] According to an aspect of an embodiment of the present disclosure, there is provided a method for optimizing GPU memory of a large model, applied to a computing system in which an image segmentation model is deployed, the computing system comprising a central processor, a graphics processor and a memory, the method comprising: the central processor dividing an input image into a plurality of non-overlapping image blocks and storing the image blocks into the memory, and sending an encoding instruction to the graphics processor; the graphics processor, in response to the encoding instruction, loading a single image block from the memory into the GPU memory one by one, performing encoding operation of the image segmentation model, performing encoding processing on the image block, obtaining a corresponding feature map and storing the feature map into the memory, and sending first feedback information to the central processor; the central processor, in response to the first feedback information, controlling the graphics processor to interactively fuse all the feature maps, obtaining a feature fusion subgraph corresponding to each image block and storing the feature fusion subgraph into the memory, and sending a decoding instruction to the graphics processor; the graphics processor, in response to the decoding instruction, loading a single feature fusion subgraph from the memory into the GPU memory one by one, performing decoding operation of the image segmentation model, performing decoding processing on the feature fusion subgraph, obtaining a corresponding decoding subimage and storing the decoding subimage into the memory, and sending second feedback information to the central processor; and the central processor, in response to the second feedback information, splicing all the decoding subimages after obtaining the decoding subimages from the memory according to the division position relationship of the input image, to obtain a final segmentation result image.

[0014] According to another aspect of an embodiment of the present disclosure, there is also provided a storage medium comprising a stored program, wherein the program, when executed by a processor, performs any of the above methods.

[0015] According to another aspect of an embodiment of the present disclosure, there is also provided a device for optimizing GPU memory of a large model, comprising a central processor, a graphics processor and a memory, wherein the central processor is configured to divide an input image into a plurality of non-overlapping image blocks and store the image blocks into the memory, and send an encoding instruction to the graphics processor; the graphics processor is configured to, in response to the encoding instruction, load a single image block from the memory into the GPU memory one by one, perform encoding operation of the image segmentation model, perform encoding processing on the image block, obtain a corresponding feature map and store the feature map into the memory, and send first feedback information to the central processor; the central processor is further configured to, in response to the first feedback information, control the graphics processor to interactively fuse all the feature maps, obtain a feature fusion subgraph corresponding to each image block and store the feature fusion subgraph into the memory, and send a decoding instruction to the graphics processor; the graphics processor is further configured to, in response to the decoding instruction, load a single feature fusion subgraph from the memory into the GPU memory one by one, perform decoding operation of the image segmentation model, perform decoding processing on the feature fusion subgraph, obtain a corresponding decoding subimage and store the decoding subimage into the memory, and send second feedback information to the central processor; and the central processor is further configured to, in response to the second feedback information, splice all the decoding subimages after obtaining the decoding subimages from the memory according to the division position relationship of the input image, to obtain a final segmentation result image.

[0016] The application first divides the input image to be segmented into multiple size-regular and non-overlapping image blocks in space by a central processor, and stores them in the system memory, completing the network input level cutting, i.e. network cutting, thereby decoupling the huge task of single-time whole-image input model into multiple independent sub-tasks sequences, laying the foundation for subsequent block-by-block pipeline processing. Then, the graphics processor responds to the encoding instruction, strictly follows the execution mode of network cutting, loads only a single image block from the memory to its video memory in sequence, and independently executes the complete encoder part operation of the image segmentation model to generate the local feature map corresponding to each image block, thereby realizing the single-block data occupation to complete the original model calculation of the whole image in the encoding stage, and significantly reducing the video memory occupation in proportion to the number of blocks. Next, a core compensation mechanism is designed to solve the context fragmentation problem caused by network cutting. The central processor responds to the feedback of the completed encoding to control the graphics processor to perform global information interaction and fusion on all independently generated local feature maps. Through global convolution or attention operation at the feature level, each local feature map can be integrated into complete spatial context information, thereby obtaining a feature fusion subgraph rich in global semantics. Then, the graphics processor continues to follow the network cutting process, loads a single feature fusion subgraph into the video memory in sequence, and independently executes the complete decoder part operation of the image segmentation model to obtain the corresponding decoded sub-image, thereby maintaining extremely low single-time video memory occupation in the decoding stage. Finally, the central processor extracts all decoded sub-images from the memory according to the division position relationship of the original image and seamlessly splices them to generate the final complete and continuous segmentation result image.

[0017] The application proposes an overall architecture using network cutting, which reconstructs the data flow of model inference from the network level through non-overlapping image block division→ independent complete encoding of blocks→ global feature fusion→ independent complete decoding of blocks→ result splicing. The essential difference between the present method and the prior art which only cuts operators within the model is that the present method makes each sub-block flow independently through the main encoding-decoding path of the model, completely eliminating all redundant calculations and storage caused by overlapping areas in the prior art; at the same time, by embedding a one-time and efficient global feature fusion link, the context loss caused by non-overlapping block division is precisely compensated, ensuring the segmentation accuracy. Thus, without introducing redundant overhead, high-precision segmentation of large-scale images can be completed through single-time calculation, realizing the coordinated optimization of video memory occupation and model performance. Further, the technical problem that the existing operator cutting method cannot significantly reduce the model video memory occupation while ensuring the segmentation performance of the model in the resource-limited environment for large-scale medical image segmentation in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present disclosure and constitute a part of this application, illustrate certain illustrative embodiments of the present disclosure and together with the general description of the present disclosure given above and the detailed description of the present disclosure given below, serve to explain the present disclosure. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0020] Figure 2 is a schematic diagram of a system for optimizing the video memory of a large model according to Embodiment 1 of the present disclosure;

[0021] Figure 3 is a flowchart of a method for optimizing the video memory of a large model according to Embodiment 1 of the present disclosure;

[0022] Figure 4 is a flowchart of a method for optimizing the video memory of a large model according to Embodiment 1 of the present disclosure;

[0023] Figure 5 is a schematic diagram of image interaction fusion of a method for optimizing the video memory of a large model according to Embodiment 1 of the present disclosure;

[0024] Figure 6 is an operator cutting strategy schematic of a method for optimizing the video memory of a large model according to Embodiment 1 of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the technical personnel in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present disclosure.

[0026] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] Example 1

[0028] According to this embodiment, a method embodiment for optimizing video memory in large models is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] The method embodiments provided in this example can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Figure 1 A hardware block diagram of a computing device for implementing memory optimization for large models is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), memory for storing data, transmission devices for communication functions, and input / output interfaces. The memory, transmission devices, and input / output interfaces are connected to the central processing unit and the graphics processing unit via a bus. In addition, it may also include a display, keyboard, and cursor control device connected to the input / output interfaces. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0030] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element in a computing device. As involved in the embodiments of this disclosure, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0031] The memory can be used to store software programs of application software and modules, such as program instructions / data storage devices corresponding to the memory optimization method of the large model of the application program in the embodiments of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the memory optimization method of the large model of the application program described above. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the computing device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0032] The transmission device is used to receive or send data via a network. The specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0033] The display can be, for example, a liquid crystal display (LCD) in the form of a touch screen, which can enable a user to interact with the user interface of the computing device.

[0034] It should be noted that in some optional embodiments, the above-mentioned Figure 1 The computing device shown can include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that in some embodiments, the functions described herein can be implemented as part of an operating system or a specific application, component, program, or the like. Furthermore, the computing device can include memory or memory locations that can store program codes, data, and the like. Figure 1 is only one example of a specific implementation, and is intended to illustrate the types of components that can be present in the computing device described above.

[0035] Figure 2 is a schematic diagram of a system for memory optimization of a large model according to the present embodiment. Referring to Figure 2As shown, the system comprises a central processing unit (CPU), a graphics processing unit (GPU), a memory, and an image segmentation model. The CPU serves as a unified scheduling center, responsible for task distribution, process control, and issuing data scheduling instructions. The GPU serves as a dedicated computing unit, which, after receiving instructions, calls the image segmentation model to perform specific encoding and decoding operations, and sends completion feedback to the CPU and stores the operation results to the external memory after each computation. The external memory is responsible for temporarily storing raw data, intermediate results, and final results in the process. All data is transferred through the external memory, ensuring that only one sub-image block data currently being calculated needs to be retained in the GPU memory, minimizing the memory usage.

[0036] Under the above operating environment, according to a first aspect of the present embodiment, a GPU memory optimization method for large models is provided, which is realized by the central processing unit, the graphics processing unit, the memory, and the image segmentation model shown in Figure 2 Figure 3 The flowchart of the method is shown in Figure 3 As shown, the method comprises:

[0037] S302: The central processing unit divides the input image into multiple non-overlapping image blocks and stores them in the memory, and sends an encoding instruction to the graphics processing unit;

[0038] S304: The graphics processing unit responds to the encoding instruction, loads a single image block from the memory into the GPU memory in sequence, performs encoding operation of the image segmentation model, encodes the image block, obtains the corresponding feature map and stores it in the memory, and sends first feedback information to the central processing unit;

[0039] S306: The central processing unit responds to the first feedback information, controls the graphics processing unit to interactively fuse all feature maps, obtains the corresponding feature fusion subgraph for each image block and stores it in the memory, and sends a decoding instruction to the graphics processing unit;

[0040] S308: The graphics processing unit responds to the decoding instruction, loads a single feature fusion subgraph from the memory into the GPU memory in sequence, performs decoding operation of the image segmentation model, decodes the feature fusion subgraph, obtains the corresponding decoded sub-image and stores it in the memory, and sends second feedback information to the central processing unit; and

[0041] S310: The central processing unit responds to the second feedback information, obtains all decoded sub-images from the memory according to the division position relationship of the input image, and splices them to obtain the final segmentation result image.

[0042] Specifically, in combination with Figure 2 and Figure 4 ​As shown, as the overall processing flow center, the central processing unit (CPU) first receives the input image to be processed (such as CT, MRI and other high-resolution images in medical scenarios), and then divides the input image into multiple independent image blocks with consistent size and no overlap, and stores them in the system memory, avoiding data redundancy between adjacent image blocks, completing the network input level cutting, i.e. network cutting, thereby decoupling the large task of single image input model into multiple independent processing sub-tasks, laying the foundation for subsequent block-by-block pipeline processing. After all the image blocks are stored in the memory, the CPU sends the encoding instruction (corresponding to step S302) to the GPU, which includes the storage path of the image block, the encoding model parameters, the processing order and other main information, which is used to trigger the GPU to retrieve the image block from the memory in sequence and perform feature extraction related encoding operations.

[0043] Then, after receiving the encoding instruction sent by the CPU, the GPU retrieves only one image block from the memory at a time according to the processing order in the encoding instruction, and temporarily loads the single image block into the GPU's own video memory. Subsequently, the GPU calls the image segmentation model to perform encoding operations on the image block, processes the image block loaded into the video memory through convolution and feature extraction, and generates a feature map that can represent the key features of the image block. After the encoding processing of a single image block is completed, the GPU transmits the generated feature map to the memory for storage and releases the GPU video memory occupied by the image block to reserve resources for the loading of the next image block. When the GPU has completed the encoding of all image blocks, it sends the first feedback information (corresponding to step S304) to the CPU. The first feedback information is used to inform the CPU that all image blocks have been encoded and the subsequent process can be entered. Thus, in the encoding stage, the model calculation originally belonging to the whole image is realized with single block data, and the video memory occupation is significantly reduced in proportion to the number of blocks.

[0044] Subsequently, after receiving the first feedback information sent by the GPU (i.e., the encoding processing of all image blocks is completed, and the corresponding feature maps are stored in the memory), the CPU sends a control instruction for feature interaction and fusion to the GPU, while coordinating the memory to provide data access support for all feature maps. After responding to the control instruction, the GPU retrieves all feature maps from the memory, and through global convolution or attention operation at the feature level, each local feature map can be integrated into complete spatial context information, thereby obtaining feature fusion subgraphs rich in global semantics. After all feature fusion subgraphs are generated, the GPU transmits all feature fusion subgraphs to the memory for storage, and then sends a decoding instruction to the GPU to trigger the subsequent decoding process (corresponding to step S306). The decoding instruction includes the storage path of the feature fusion subgraph, the decoding parameter, the processing order, and other main information, which is used to trigger the GPU to sequentially retrieve the feature fusion subgraph from the memory and perform decoding operation.

[0045] Further, after receiving the decoding instruction sent by the CPU, the GPU retrieves a single feature fusion subgraph from the memory at a time according to the processing order in the decoding instruction, and temporarily loads it into the GPU's own memory. Subsequently, the GPU calls the image segmentation model to perform decoding operation, and through upsampling and feature restoration process, the abstract feature fusion subgraph is converted into a decoded subgraph image with spatial pixel information. After the decoding processing of a single feature fusion subgraph is completed, the GPU transmits the generated decoded subgraph image to the memory for storage, and synchronously releases the memory resources occupied by the feature fusion subgraph, reserving space for the loading and decoding of the next feature fusion subgraph. When all feature fusion subgraphs complete the decoding operation and store the operation results in the memory, the GPU sends second feedback information to the CPU (corresponding to step S308). The second feedback information includes the processing status and storage path of the current decoded subgraph image, and other key contents, which is used to inform the CPU that the decoding process of all feature fusion subgraphs is completed, so as to facilitate the CPU to coordinate and promote the splicing work of all decoded subgraph images. In this way, it is ensured that the single memory occupation is extremely low in the decoding stage.

[0046] Then, after receiving the second feedback information sent by the GPU, the CPU retrieves all decoded subgraph images from the memory according to the initial regular grid division position relationship of the input image (i.e., the row and column coordinate correspondence relationship when the original image is cut into image blocks, for example, the image block in the first row and the first column corresponds to the upper left corner of the original image, the image block in the mth row and the nth column corresponds to the lower right corner of the original image, etc.), and aligns and splices each decoded subgraph image pixel by pixel according to the spatial position order completely consistent with the original division, ensuring that the edges of adjacent decoded subgraph images fit together without misalignment or overlapping deviation, and finally obtaining a complete segmentation result graph consistent with the size of the original input image and corresponding to the spatial structure (corresponding to step S310).

[0047] As the content of the foregoing background art, the existing operator tiling method is mainly divided into two strategies: (1) Non-overlapping operator tiling: This strategy simply divides the feature map of the current layer into multiple non-overlapping continuous sub-blocks during model calculation, and performs convolution operation on each sub-block independently. This method can minimize the instantaneous memory requirement of single-layer calculation. However, since the convolution kernel calculates the output pixels by relying on all input values within its receptive field, the pixels at the boundary of the sub-block cannot access the necessary context in the adjacent sub-block, resulting in a deviation in the calculation result from that based on the entire feature map. This deviation propagates and accumulates layer by layer in the network, which will directly damage the accuracy and boundary consistency of image segmentation. (2) Overlapping operator tiling: To correct the above deviation, this strategy forces the adjacent sub-blocks to contain an overlapping region (Halo) related to the size of the convolution kernel when dividing the feature map, ensuring that each sub-block calculation has complete context, thus maintaining consistency with the entire map calculation. However, this introduces new costs: First, the overlapping data is repeatedly stored, which weakens the actual benefits of memory optimization; second, the overlapping region is repeatedly calculated in adjacent sub-blocks, resulting in significant redundant calculation overhead, significantly increasing the processing delay of the segmentation task. Figure One

[0048] ​In view of this, the application first divides the input image to be segmented into multiple size-regular and non-overlapping image blocks in space by the central processor, and stores them in the system memory, completing the network input level cutting, i.e. network cutting, thereby decoupling the huge task of single-time whole-image input model into multiple independent sub-tasks sequences, laying the foundation for subsequent block-by-block pipeline processing. Then, the graphics processor responds to the encoding instruction, strictly follows the execution mode of network cutting, loads only a single image block from the memory to its video memory in sequence, and independently executes the complete encoder part operation of the image segmentation model to generate a local feature map corresponding to each image block, thereby realizing the single-block data occupancy to complete the original whole-image model calculation in the encoding stage, and significantly reducing the video memory occupancy in proportion to the number of blocks. Next, a core compensation mechanism is designed to address the context fragmentation problem brought about by network cutting. The central processor responds to the feedback of the completed encoding to control the graphics processor to perform global information interaction and fusion on all independently generated local feature maps, and through global convolution or attention operation at the feature level, each local feature map can be integrated into complete spatial context information, thereby obtaining a feature fusion subgraph rich in global semantics. Then, the graphics processor continues to follow the network cutting process, loads a single feature fusion subgraph into the video memory in sequence, and independently executes the complete decoder part operation of the image segmentation model to obtain the corresponding decoded sub-image, thereby maintaining extremely low single-time video memory occupancy in the decoding stage. Finally, the central processor extracts all decoded sub-images from the memory according to the division position relationship of the original image and seamlessly splices them to generate the final complete and continuous segmentation result image.

[0049] The application proposes an overall architecture using network cutting, which reconstructs the data flow of model inference from the network level through non-overlapping image block division → block-independent complete encoding → global feature fusion → block-independent complete decoding → result splicing. The essential difference between the present method and the prior art which only cuts operators within the model is that the present method allows each sub-block to independently flow through the main encoding-decoding path of the model, completely eliminating all redundant calculations and storage caused by overlapping areas in the prior art; at the same time, by embedding a one-time and efficient global feature fusion link, the context loss caused by non-overlapping block division is precisely compensated for, ensuring segmentation accuracy. Thus, without introducing redundant overhead, high-precision segmentation of large-scale images can be completed through single-time calculation, realizing the coordinated optimization of video memory occupancy and model performance. Further, the technical problem of the prior art that the existing operator cutting method cannot significantly reduce the model video memory occupancy while ensuring the segmentation performance of the model when performing large-scale medical image segmentation in a resource-limited environment is solved.

[0050] Optionally, the central processor controls the graphic processor to interactively fuse all feature maps to obtain a feature fusion subgraph corresponding to each image block in response to the first feedback information, including: the central processor responds to the first feedback information, and according to the division position relationship, splices all feature maps obtained from the memory into a complete intermediate feature map and stores it to the memory, and sends an information interaction instruction to the graphic processor; the graphic processor responds to the information interaction instruction, loads the intermediate feature map from the memory, performs feature interaction operation of the image segmentation model, performs convolution operation on the intermediate feature map, obtains a complete feature fusion map and stores it to the memory, and sends third feedback information to the central processor; the central processor responds to the third feedback information, and according to the division position relationship, splits the feature fusion map loaded from the memory into a plurality of first feature fusion subgraphs as the feature fusion subgraph corresponding to each image block.

[0051] Specifically, referring to FIG. 1, Figure 5 As shown in FIG. 1, after receiving the first feedback information sent by the graphic processor (GPU), the central processor (CPU) first retrieves all independent feature maps from the memory according to the division position relationship of the input image (consistent with the corresponding rule of the row and column coordinates when cutting the image block). Then, all feature maps are pixel by pixel aligned and spliced according to the division position relationship, ensuring that the spatial position of each feature map corresponds to the position of the input image block one by one, and finally forming a complete intermediate feature map, and storing the intermediate feature map to the memory. Then, the CPU sends an information interaction instruction to the GPU, which includes the storage path of the intermediate feature map, the related parameters of the feature interaction operation and other key information.

[0052] After the GPU responds to the information interaction instruction, it loads the complete intermediate feature map from the memory and temporarily stores it in its own memory. Then it calls the feature interaction module in the image segmentation model to perform operation, and through convolution operation, it realizes the information interaction and fusion in the global range of the intermediate feature map, makes up for the missing adjacent area correlation information when a single feature map is independently encoded, and then generates a complete feature fusion map including global feature correlation. After the GPU stores the feature fusion map to the memory, it sends third feedback information to the CPU, informing that the feature interaction fusion process has been completed. After the CPU receives the third feedback information, it loads the complete feature fusion map from the memory again according to the division position relationship of the input image, and splits the feature fusion map into a plurality of first feature fusion subgraphs corresponding to the input image block according to the same row and column division rule as the split image block, that is, each first feature fusion subgraph corresponds to the feature fusion result of an image block, as the input data for subsequent decoding processing.

[0053] By the above mode, the global information interaction and deep fusion of the intermediate feature map are realized, the isolation of the single image block feature is broken, each feature fusion subgraph retains its own local key feature and fully integrates the global correlation information, the problem of missing information of adjacent regions caused by non-overlapping block processing is effectively made up, and the accuracy and integrity of image segmentation are ensured.

[0054] Optionally, the central processor controls the graphic processor to perform interactive fusion on all feature maps to obtain the feature fusion subgraph corresponding to each image block in response to the first feedback information, including: the central processor responds to the first feedback information, and splices all feature maps obtained from the memory into a complete intermediate feature map according to the division position relationship and stores the intermediate feature map into the memory; the central processor divides the intermediate feature map loaded from the memory into a plurality of feature subgraphs with overlapping regions by using an operator cutting block strategy and stores the feature subgraphs into the memory and sends an information interaction instruction to the graphic processor; the graphic processor responds to the information interaction instruction, loads a single feature subgraph from the memory into the video memory in sequence, performs feature interaction operation of the image segmentation model, performs convolution operation on the feature subgraph, obtains a corresponding second feature fusion subgraph and stores the second feature fusion subgraph into the memory, and sends fourth feedback information to the central processor; and the central processor responds to the fourth feedback information, removes the overlapping regions between all second feature fusion subgraphs loaded from the memory based on the spatial overlapping relationship defined by the operator cutting block strategy, and obtains a plurality of third feature fusion subgraphs as the feature fusion subgraph corresponding to each image block.

[0055] Specifically, if the output consistency is considered, the image blocks need to have boundary overlap, and the overlapping region is kernel_size / / 2, and finally each image block input is (boundary condition), the single memory occupation is 240B, which is reduced by 57% compared with the input of the whole image for calculation of convolution. It has been proved that the operator cutting block is an effective method for reducing memory, and in the actual project deployment and optimization, the operator cutting block can effectively reduce the model memory.

[0056] Therefore, the embodiment of the application combines the operator cutting block and the network cutting block (i.e., the operator cutting block strategy with boundary overlap) to reduce the memory requirement. Figure 6As shown, first, the CPU divides the input image into 4 non-overlapping image blocks (a person skilled in the art can divide the input image into a plurality of non-overlapping image blocks according to actual needs, which is not limited here, and in the embodiments of the present application, 4 image blocks are taken as an example), and calls the GPU to sequentially input all the image blocks into the image segmentation model for encoding operation to generate a corresponding number of feature maps. Subsequently, the CPU splices all the feature maps into a complete intermediate feature map according to the original input order of the image blocks, and adopts an operator cutting block strategy with boundary overlap to perform secondary splitting on the intermediate feature map to obtain 4 feature subgraphs with overlapping regions. Further, the CPU calls the GPU to respectively perform depth information interaction on all the feature subgraphs through convolution operation, and fully fuse the associated features of adjacent regions. Then, the CPU performs overlap removal processing on the fused feature subgraphs based on the spatial overlap relationship defined by the operator cutting block strategy, and further splits them into feature fusion subgraphs consistent with the size of the original image blocks. All the feature fusion subgraphs are sequentially input into the image segmentation model for decoding to generate corresponding decoded sub-images. Finally, all the decoded sub-images are spliced to obtain a complete segmentation result image.

[0057] Specifically, after receiving the first feedback information sent by the GPU, the CPU first retrieves all independent feature maps from the memory according to the division position relationship of the input image (such as the row and column coordinate correspondence rule), aligns and splices them pixel by pixel to form a complete intermediate feature map (which completely restores the feature space distribution of the input image), and stores the intermediate feature map in the memory.

[0058] Subsequently, the CPU adopts an operator cutting block strategy with boundary overlap to perform secondary division on the intermediate feature map, that is, when dividing, an overlapping region with a predetermined width (for example, an edge 5-10 pixel overlap) is reserved between adjacent feature subgraphs to ensure that each feature subgraph not only includes its own core region features, but also carries associated features of adjacent regions, avoiding edge feature breakage in subsequent convolution operation. After the division is completed, the CPU stores all the feature subgraphs in the memory and sends information interaction instructions to the GPU.

[0059] GPU responds to the information interaction instruction, each time only from the memory to call a feature subgraph, temporary loading to its own video memory, then call the feature interaction module of image segmentation model to execute convolution operation. Through the convolution operation, the local features of the feature subgraph itself and the adjacent features carried by the overlapping area will be deeply fused to generate a second feature fusion subgraph containing complete associated information. After the processing of a single feature subgraph is completed, the GPU stores the second feature fusion subgraph to the memory, releases the video memory occupied by the feature subgraph, and then processes the next feature subgraph one by one until all the feature subgraphs are fused, and sends the fourth feedback information to the CPU (informing that the fusion processing of all feature subgraphs has been completed). Thus, in the information interaction process, only the data of a single feature subgraph needs to be maintained in the video memory, rather than the entire feature map, thereby greatly reducing the video memory occupation. For example, under the example setting, the single video memory occupation is only 240B, which is reduced by about 57% compared with the direct convolution of the entire map.

[0060] After receiving all the fourth feedback information (confirming that all the second feature fusion subgraphs have been stored), the CPU retrieves all the second feature fusion subgraphs from the memory according to the spatial overlap relationship defined in the operator cutting strategy with boundary overlap (i.e. clearly defining the position and range of the overlapping area of each feature subgraph); then accurately crop the overlapping parts of adjacent subgraphs. That is, only the non-overlapping core area of each second feature fusion subgraph is retained (to avoid data redundancy), and finally four third feature fusion subgraphs are obtained, which are completely matched in size with the original image blocks and contain global associated features. Each third feature fusion subgraph corresponds to the final fusion feature of the original image block, providing complete and non-redundant input data for subsequent decoding processing.

[0061] In the information interaction stage, the present scheme uses operator-level cutting and batch processing, so that only the video memory of a single subgraph is occupied in the information interaction stage, effectively controlling the video memory occupation. At the same time, with the help of the reservation and fusion of the overlapping area, the continuity of the features at the boundary is ensured, and the consistency of the output results is improved. The fusion cutting strategy has shown good video memory optimization effect in actual project deployment.

[0062] Optionally, the central processing unit adopts an operator cutting strategy with boundary overlap to divide the intermediate feature map loaded from the memory into multiple feature subgraphs with overlapping areas, including: based on the number of rows and columns of the original regular grid used when dividing the input image and the spatial origin, dividing the complete intermediate feature map into corresponding grid cells; according to the preset overlap size parameter, expanding outward along the boundaries of the grid cells to form overlapping areas with a predetermined overlap width between adjacent grid cells; according to the area contained in the expanded grid cells, extracting corresponding image data from the intermediate feature map to generate multiple feature subgraphs containing overlapping areas.

[0063] Specifically, when the CPU divides the complete intermediate feature map into corresponding grid cells according to the operator tiling strategy with boundary overlap, the CPU first reuses the original regular grid parameters used when dividing the input image, which include the total number of rows and columns of the grid (for example, the original input image is divided into M rows and N columns of image blocks, and the intermediate feature map is also divided into M rows and N columns of grid cells), and the spatial origin of the grid (i.e., the coordinate starting point of the upper left corner of the original image), to establish a one-to-one corresponding grid framework on the complete intermediate feature map, ensuring that the spatial position of each grid cell is completely aligned with the position of the original image block, and avoiding misalignment in subsequent feature association.

[0064] Then, the CPU calls a preset overlap size parameter (which can be flexibly configured according to the image feature complexity and model convolution kernel size, for example, set the overlap width to K pixels), and performs a boundary outward expansion operation on each grid cell established in the first step. For example, for a certain grid cell, its left boundary is expanded to the left by K pixels, its right boundary is expanded to the right by K pixels, its upper boundary is expanded to the top by K pixels, and its lower boundary is expanded to the bottom by K pixels (if the grid cell is located at the edge of the intermediate feature map, it is only expanded to the adjacent direction inside the image to avoid exceeding the image range), and through this expansion method, an overlap region with a width of K pixels is formed between two adjacent grid cells, ensuring that the subsequent feature fusion can cover the associated information of the adjacent region.

[0065] Finally, each expanded grid cell corresponds to a definite rectangular region on the intermediate feature map (which contains the core feature region of the original grid cell and the expanded overlap feature region), and the CPU extracts the complete image data in each rectangular region from the intermediate feature map, and then generates multiple feature subgraphs with consistent size and a predetermined overlap width between adjacent subgraphs, ensuring that each feature subgraph not only includes its own core features, but also can associate the feature information of adjacent subgraphs through the overlap region.

[0066] Through the above method, the feature subgraph division is accurately spatially aligned with the original input image division, which not only ensures that the feature subgraph can fully cover the associated features of the adjacent region and provides data basis for deep feature interaction, but also dynamically adjusts the overlap range according to the image type and model requirements, and adapts to the feature association requirements in different scenarios.

[0067] Optionally, the central processing unit, in response to the fourth feedback information, removes the overlapping areas between all the second feature fusion subgraphs loaded from the memory based on the spatial overlap relationship defined by the operator tiling strategy, to obtain the operation of obtaining a plurality of third feature fusion subgraphs, including: in response to the fourth feedback information, determining the pixel positions belonging to the overlapping areas in each second feature fusion subgraph according to the spatial overlap relationship defined by the operator tiling strategy; and deleting the pixel parts belonging to the overlapping areas in each second feature fusion subgraph according to the pixel positions, to obtain the corresponding third feature fusion subgraph.

[0068] Specifically, when the CPU receives the fourth feedback information (all the second feature fusion subgraphs are stored in the memory), the complete spatial overlap relationship recorded in the operator tiling strategy with boundary overlap is called, including the preset overlap width (such as K pixels), the expansion direction of each feature subgraph (for example, the edge subgraph only expands in the adjacent direction inside the image, and the middle subgraph expands in four directions), and the original grid cell coordinates corresponding to each feature subgraph. Then, the CPU combines the above information to judge whether the pixels in the second feature fusion subgraph belong to the overlapping area one by one. For example, for the second feature fusion subgraph located at the top left corner, only the K pixels on the right boundary overlap with the second feature fusion subgraph on the right, and the K pixels on the lower boundary overlap with the second feature fusion subgraph below, and only the overlapping pixel positions in these two directions are marked.

[0069] Further, after the overlapping pixel positions of each second feature fusion subgraph are determined, the CPU performs a clipping deletion operation on the single second feature fusion subgraph, only retains the non-overlapping core pixel area in the second feature fusion subgraph, and deletes the marked overlapping pixel part. For example, after deleting the overlapping parts of the K pixels on the right and the K pixels on the lower side of the second feature fusion subgraph at the top left corner, the remaining area also completely matches the original image block at the top left corner. Through this directional deletion, each second feature fusion subgraph is converted into a third feature fusion subgraph without redundancy and overlap, and the spatial positions of all the third feature fusion subgraphs correspond one by one to the original image blocks, laying a non-interfering data foundation for subsequent decoding and splicing processes.

[0070] In the above manner, the directional removal of the overlapping areas in the second feature fusion subgraph is realized, the core associated features obtained after the feature interaction operation of each second feature fusion subgraph are completely retained, and the pixel redundancy between adjacent subgraphs is completely eliminated, so that the third feature fusion subgraphs finally generated are accurately aligned in size and spatial position with the original image blocks, and the interference of redundant data in subsequent decoding processing and the spatial misalignment problem in splicing are fundamentally avoided.

[0071] Optionally, the operation of the central processor dividing the input image into a plurality of non-overlapping image blocks comprises: taking a preset fixed size as the basis for division, and dividing the input image into a plurality of image blocks of uniform size and non-overlapping.

[0072] Specifically, the operation of the CPU dividing the input image into a plurality of non-overlapping image blocks according to a preset regular grid division strategy (such as the number of rows and columns determined based on the image resolution) can specifically include that: the CPU first needs to determine a preset fixed size parameter (which can be configured in advance according to factors such as GPU memory capacity, input requirements of the image segmentation model, processing efficiency, etc., for example, the size of each image block is preset to 256x256 pixels, 512x512 pixels, etc.), which is the uniform standard for all output image blocks. Subsequently, the CPU establishes a regular rectangular grid (similar to the grid distribution of graph paper) on the input image based on the fixed size - the horizontal (column direction) and vertical (row direction) division intervals of the grid are consistent with the preset fixed size, for example, when the input image size is 1024x1024 pixels and the preset fixed size is 256x256 pixels, a regular grid of 4 rows and 4 columns is established. Finally, the CPU cuts the input image along the horizontal and vertical division lines of the grid, and each image block obtained after cutting strictly follows the preset fixed size (i.e., 256x256 pixels), and the edges of adjacent image blocks are closely connected without any overlapping area, and there is no image data missing, and finally a batch of independent image blocks of uniform size, regular spatial distribution, and non-overlapping are formed.

[0073] Through the above non-overlapping division method, the storage pressure of the memory and the invalid overhead of subsequent operations are reduced, and the complete coverage of the input image is ensured without any data missing.

[0074] Optionally, the operation of the central processor, in response to the second feedback information, obtaining all decoded sub-images from the memory according to the division position relationship of the input image and splicing to obtain the final segmentation result map comprises: reading all decoded sub-images from the memory; according to the spatial position relationship defined by the regular grid relied on when dividing the input image, splicing all decoded sub-images by position to generate a complete and spatially continuous segmentation result map.

[0075] Specifically, the CPU first reads all decoded sub-images from the memory in batches, simultaneously checks the integrity of each decoded sub-image (ensures no data damage or loss), and records the identification information corresponding to each sub-image (one-to-one correspondence with the division identification of the original image block, such as "1st row 1st column block", "2nd row 1st column block", etc.), to avoid confusion of the decoded sub-images.

[0076] Then, the CPU multiplexes the rule grid parameters in the input image division, for example, divides the original input image into four image blocks according to a rule grid of 2 rows and 2 columns, and the position of each image block can be uniquely identified by "row number + column number" (for example, (1, 1) corresponds to the upper left corner and (2, 2) corresponds to the lower right corner).

[0077] Further, the CPU sorts and positions all the decoded sub-images according to the unique identification of each image block. That is, in the embodiment of the present application, the decoded sub-images in the same row can be arranged in order from left to right according to the column number, and the edges of adjacent sub-images are closely connected. After completing the horizontal splicing of one row, the splicing result of the next row is connected below the above row from top to bottom, and so on, until the decoded sub-images of all rows and columns are spliced in place. The finally generated image not only has the same size as the original input image, but also has a one-to-one correspondence with the original image in terms of spatial position, forming a complete, error-free, and spatially continuous segmentation result image, which perfectly restores the overall structure of the original image.

[0078] In this way, the edges of adjacent decoded sub-images are seamlessly connected without error deviation, and the segmentation result image after splicing is completely consistent with the original input image in terms of size and spatial structure, completely restoring the overall spatial continuity of the image.

[0079] In addition, referring to FIG. 6, according to a second aspect of the present embodiment, a storage medium is provided. The storage medium includes a stored program, wherein the program is executed by a processor when the program is run. Figure 1

[0080] According to the present embodiment, the data flow of model inference is reconstructed from the network level through non-overlapping image blocking, block-independent complete encoding, global feature fusion, block-independent complete decoding, and result splicing. The essential difference between the present method and the prior art which only cuts the operators inside the model is that the present method allows each sub-block to flow independently through the main encoding-decoding path of the model, completely eliminating all redundant calculations and storage caused by overlapping areas in the prior art. At the same time, by embedding a one-time and efficient global feature fusion link, the context loss caused by non-overlapping blocking is accurately compensated for, ensuring the segmentation accuracy. Thus, without introducing redundant overhead, high-precision segmentation of large-scale images can be completed through a single calculation, realizing the collaborative optimization of memory occupation and model performance. Further, the technical problem that the existing operator blocking method cannot significantly reduce the memory occupation of the model while ensuring the segmentation performance of the model when performing large-scale medical image segmentation in a resource-limited environment is solved.

[0081] ​It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0082] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and a necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for causing an end device (which can be a mobile phone, computer, server, or network device, etc.) to execute the method described in each embodiment of the present application.

[0083] Embodiment 2

[0084] The embodiment provides a graphics memory optimization device of a large model, which corresponds to the method according to the embodiment 1, and includes a central processing unit, a graphics processing unit and a memory, wherein the central processing unit is configured to divide an input image into a plurality of non-overlapping image blocks and store the image blocks into the memory, and send an encoding instruction to the graphics processing unit; the graphics processing unit is configured to load a single image block from the memory into the graphics memory in sequence in response to the encoding instruction, perform encoding operation of an image segmentation model, encode the image block to obtain a corresponding feature map and store the feature map into the memory, and send first feedback information to the central processing unit; the central processing unit is further configured to control the graphics processing unit to interactively fuse all the feature maps in response to the first feedback information, obtain a feature fusion subgraph corresponding to each image block and store the feature fusion subgraph into the memory, and send a decoding instruction to the graphics processing unit; the graphics processing unit is further configured to load a single feature fusion subgraph from the memory into the graphics memory in sequence in response to the decoding instruction, perform decoding operation of the image segmentation model, decode the feature fusion subgraph to obtain a corresponding decoded subimage and store the decoded subimage into the memory, and send second feedback information to the central processing unit; and the central processing unit is further configured to obtain all the decoded subimages from the memory according to the division position relationship of the input image, and splice the decoded subimages to obtain a final segmentation result image.

[0085] It should be noted that the graphics memory optimization device of the large model provided in the embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The same parts will not be described again.

[0086] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0087] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0088] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0089] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0090] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0091] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the essential part or all or part of the technical solutions that make contributions to the prior art can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The above-mentioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0092] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the protection scope of the present application.

Claims

1. A method for GPU memory optimization of a large model, characterized in that, Applied to a computing system deployed with an image segmentation model, the computing system comprising a central processor, a graphics processor and a memory, the method comprising: The central processor divides an input image into a plurality of non-overlapping image blocks and stores them to the memory, and sends an encoding instruction to the graphics processor; The graphics processor, in response to the encoding instruction, sequentially loads a single image block from the memory into the video memory, performs an encoding operation of the image segmentation model, encodes the image block, obtains a corresponding feature map and stores it to the memory, and sends first feedback information to the central processor; The central processor, in response to the first feedback information, controls the graphics processor to interactively fuse all feature maps to obtain a feature fusion subgraph corresponding to each image block and stores it to the memory, and sends a decoding instruction to the graphics processor; The graphics processor, in response to the decoding instruction, sequentially loads a single feature fusion subgraph from the memory into the video memory, performs a decoding operation of the image segmentation model, decodes the feature fusion subgraph, obtains a corresponding decoded sub-image and stores it to the memory, and sends second feedback information to the central processor; and The central processor, in response to the second feedback information, acquires all decoded sub-images from the memory according to the division position relationship of the input image, and splices them to obtain a final segmentation result image, wherein The operation of the central processor, in response to the first feedback information, to control the graphics processor to interactively fuse all feature maps to obtain a feature fusion subgraph corresponding to each image block, comprises: The central processor, in response to the first feedback information, splices all feature maps acquired from the memory into a complete intermediate feature map according to the division position relationship and stores it to the memory; The central processor uses an operator cutting strategy with boundary overlap to divide the intermediate feature map loaded from the memory into a plurality of feature subgraphs with overlapping regions and stores them to the memory, and sends an information interaction instruction to the graphics processor; The graphics processor, in response to the information interaction instruction, sequentially loads a single feature subgraph from the memory into the video memory, performs a feature interaction operation of the image segmentation model, convolves the feature subgraph, obtains a corresponding second feature fusion subgraph and stores it to the memory, and sends fourth feedback information to the central processor; The central processor, in response to the fourth feedback information, removes the overlapping regions between all second feature fusion subgraphs loaded from the memory based on the spatial overlap relationship defined by the operator cutting strategy to obtain a plurality of third feature fusion subgraphs as the feature fusion subgraphs corresponding to each image block.

2. The method of claim 1, wherein, The operation of the central processor to use an operator cutting strategy with boundary overlap to divide the intermediate feature map loaded from the memory into a plurality of feature subgraphs with overlapping regions, comprises: According to the number of rows and columns and the spatial origin of the original rule grid used when the input image is divided, the complete intermediate feature map is divided into corresponding grid cells; According to a preset overlap size parameter, the boundaries of each grid cell are expanded outward, so that an overlap region with a predetermined overlap width is formed between adjacent grid cells; According to the region contained in the expanded grid cell, corresponding image data is extracted from the intermediate feature map to generate a plurality of feature subgraphs containing the overlap region.

3. The method of claim 1, wherein, The central processor removes the overlap region between all second feature fusion subgraphs loaded from the memory to obtain a plurality of third feature fusion subgraphs according to the spatial overlap relationship defined by the operator tiling strategy in response to the fourth feedback information, including: In response to the fourth feedback information, the spatial overlap relationship defined by the operator tiling strategy is determined according to the pixel position in each second feature fusion subgraph belonging to the overlap region; According to the pixel position, the pixel part belonging to the overlap region in each second feature fusion subgraph is deleted to obtain the corresponding third feature fusion subgraph.

4. The method of claim 1, wherein, The operation of the central processor for dividing the input image into a plurality of non-overlapping image blocks includes: According to a preset fixed size as the basis for division, the input image is divided into a plurality of image blocks with uniform size and non-overlapping according to a rule grid.

5. The method of claim 4, wherein, The operation of the central processor for obtaining the final segmentation result map by splicing all decoded sub-images from the memory in response to the second feedback information according to the division position relationship of the input image includes: Reading all decoded sub-images from the memory; According to the spatial position relationship defined by the rule grid used when the input image is divided, all decoded sub-images are spliced bit by bit to generate a complete and spatially continuous segmentation result map.

6. A storage medium, characterized by The storage medium includes a stored program, wherein the program is executed by the processor when the program is run to perform the method of any one of claims 1 to 5.

7. A device for optimizing GPU memory of a large model, characterized by, The central processor, the graphics processor and the memory, wherein: The central processor is configured to divide the input image into a plurality of non-overlapping image blocks and store them in the memory, and send encoding instructions to the graphics processor; The graphics processor is configured to load a single image block from the memory into the video memory in response to the encoding instructions, perform encoding operations of the image segmentation model, encode the image block, obtain the corresponding feature map and store it in the memory, and send the first feedback information to the central processor; The central processor is further configured to control the graphics processor to interactively fuse all feature maps in response to the first feedback information to obtain a feature fusion subgraph corresponding to each image block and store it in the memory, and send decoding instructions to the graphics processor; The graphics processor is further configured to, in response to the decoding instruction, sequentially load a single feature fusion subgraph from the memory into the video memory, perform decoding operation of the image segmentation model, decode the feature fusion subgraph, obtain a corresponding decoded subgraph image, and store the decoded subgraph image into the memory, and send second feedback information to the central processor; and The central processor is further configured to, in response to the second feedback information, splice all decoded subgraph images obtained from the memory according to the division position relationship of the input image, to obtain a final segmentation result image, wherein The central processor is configured to, in response to the first feedback information, control the graphics processor to interactively fuse all feature maps to obtain a feature fusion subgraph corresponding to each image block, including: The central processor is configured to, in response to the first feedback information, splice all feature maps obtained from the memory into a complete intermediate feature map according to the division position relationship, and store the intermediate feature map into the memory; The central processor is configured to divide the intermediate feature map loaded from the memory into a plurality of feature subgraphs with overlapping regions according to an operator tiling strategy with boundary overlap, and store the feature subgraphs into the memory, and send information interaction instruction to the graphics processor; The graphics processor is configured to, in response to the information interaction instruction, sequentially load a single feature subgraph from the memory into the video memory, perform feature interaction operation of the image segmentation model, perform convolution operation on the feature subgraph, obtain a corresponding second feature fusion subgraph, and store the second feature fusion subgraph into the memory, and send fourth feedback information to the central processor; and The central processor is configured to, in response to the fourth feedback information, remove overlapping regions between all second feature fusion subgraphs loaded from the memory based on a spatial overlap relationship defined by the operator tiling strategy, to obtain a plurality of third feature fusion subgraphs as feature fusion subgraphs corresponding to each image block.

Images