Visual model reasoning method and device based on image segmentation

Abstract

The invention provides a visual model reasoning method and device based on image segmentation, and relates to the technical field of image processing, and the method comprises the steps: segmenting a to-be-processed image according to a preset size to obtain a plurality of image slices, and recording a corresponding image index into a block list, and a physical video memory for storing the image to be processed is divided into a plurality of video memory blocks for storing the image slices. And when the visual model performs reasoning, obtaining image slices from the video memory blocks according to the block list, and splicing the image slices into a to-be-reasoned image to obtain an image reasoning result. According to the invention, the to-be-processed image and the physical video memory are respectively divided, the segmented to-be-processed image is stored through the divided video memory blocks, and the segmented to-be-processed image is recorded in the block list for reasoning of the visual model, so that the problem that the utilization rate of the video memory is not high when the visual model is used for reasoning images with various dynamic sizes in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a visual model inference method and device based on image segmentation. Background Art

[0002] In the field of computer vision, deep learning models have been widely used in tasks such as image recognition, classification, and segmentation. When a visual model is trained, the input images generally have a fixed size, and the batch size and the size of intermediate calculation results (such as feature maps) are also fixed. However, during inference, the input batch size of the model is not fixed, and the size of each image may not be fixed either. Moreover, during the long-term inference of the entire model, the input size may also change dynamically. If the model can only receive inputs of a fixed size, it cannot be used, and the inference engine needs to be adjusted and optimized accordingly to adapt to the inference of dynamic shapes. When the shape of the input changes, the video memory allocation during the model calculation process is a difficult problem. If the maximum size of the image is directly preset, video memory fragmentation will occur, resulting in waste of video memory, and this problem will be more prominent in mobile devices with smaller memory capacity.

[0003] To solve the above technical problems, in the prior art, in order to support dynamic shapes, a deep learning inference engine of TensorRT is proposed. It introduces an "optimized configuration file", which allows users to specify a series of possible input tensor size ranges according to this optimized configuration file. However, TensorRT still pre-allocates video memory according to the maximum size in the optimized configuration file, resulting in low video memory utilization. If the actual input size is much smaller than the specified maximum size, most of the pre-allocated video memory will not be fully utilized, causing resource waste. Summary of the Invention

[0004] The present invention provides a visual model inference method and device based on image segmentation, which are used to solve the problem of low video memory utilization rate when the visual model in the prior art infers images of various dynamic sizes.

[0005] The present invention provides a visual model inference method based on image segmentation, which is characterized by including: Segmenting the image to be processed according to a preset size to obtain a plurality of image slices, and recording the image index of each image slice into a block list; Dividing the physical video memory storing the image to be processed into a plurality of video memory blocks, and storing the image slices through the video memory blocks; When the visual model performs inference, obtaining the image slices from the video memory blocks according to the block list, and splicing the image slices according to the image index to obtain the image to be inferred; Input the image to be inferred into the visual model for inference to obtain an image inference result.

[0006] In some embodiments, storing the image slices in the video memory block includes: Combining the image slices into image blocks according to a preset quantity; Recording the image blocks and the number of image slices included in the image blocks in a block list, and establishing a mapping relationship between the image blocks and the video memory blocks in the block list; Storing the image slices in the image blocks into the video memory blocks according to the mapping relationship.

[0007] In some embodiments, obtaining the image slices from the video memory blocks according to the block list includes: Querying the mapping relationship between the image blocks and the video memory blocks from the block list according to the image index of the image slices; Determining the target video memory block corresponding to the storage of the image slices from the physical video memory according to the mapping relationship, and obtaining the image slices corresponding to the image index according to the target video memory block.

[0008] In some embodiments, inputting the image to be inferred into the visual model for inference to obtain an image inference result includes: During the process of the visual model extracting the feature map of the image to be inferred, obtaining the feature maps extracted by each layer of the network, and slicing the feature maps to obtain a plurality of feature sub-maps; Recording the feature indexes of the feature sub-maps in the block list, and storing the feature sub-maps in the video memory blocks divided by the physical video memory; When the visual model performs inference, querying the feature sub-maps from the block list according to the feature indexes; Performing splicing processing on the feature sub-maps according to the feature indexes to obtain a feature map to be processed; Performing mapping processing on the feature map to be processed output by the last layer of the network in the visual model to obtain an image inference result.

[0009] In some embodiments, before slicing the image to be processed according to a preset size, the method further includes: Performing size filling processing on the image to be processed, and constructing a filling configuration file according to the size filling result; The filling configuration file is used to perform size cropping processing on the predicted feature map according to the size filling result before the visual model outputs the image inference result, and the predicted feature map is the feature map used for mapping processing to obtain the image inference result.

[0010] In some embodiments, after inputting the image to be inferred into the visual model for inference to obtain an image inference result, the method further includes: Removing the image index corresponding to the image slice of the image to be inferred and the feature index corresponding to the feature subgraph from the block list; Releasing the storage space of the video memory block used to store the image slice and the feature subgraph in the physical video memory.

[0011] According to an image segmentation-based visual model inference device provided by the present invention, including: A segmentation module, configured to segment an image to be processed according to a preset size to obtain a plurality of image slices, and record the image index of each image slice in a block list; A storage module, configured to divide the physical video memory storing the image to be processed into a plurality of video memory blocks, and store the image slices through the video memory blocks; An acquisition module, configured to, when the visual model performs inference, acquire the image slices from the video memory blocks according to the block list, and splice the image slices according to the image index to obtain an image to be inferred; An inference module, configured to input the image to be inferred into the visual model for inference to obtain an image inference result.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for inferring a visual model based on image segmentation as described in any one of the above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for inferring a visual model based on image segmentation as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for inferring a visual model based on image segmentation as described in any one of the above is implemented.

[0015] The method and device for inferring a visual model based on image segmentation provided by the present invention divide the image to be processed and the physical video memory respectively, and then use the divided video memory blocks to store the segmented image to be processed, which is recorded in the block list for the visual model to perform inference. In this way, even for images to be processed with different dynamic sizes, they can be segmented according to a fixed size and stored in the divided video memory blocks as needed, which can effectively utilize the video memory resources, without the need for pre-allocation in advance, and improve the video memory utilization rate. Description of the Drawings

[0016] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce one by one the attached drawings required in the description of the embodiments or the prior art. Obviously, the attached drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other attached drawings can also be obtained based on these attached drawings.

[0017] Figure 1 It is a schematic flowchart of the visual model inference method based on image segmentation provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the segmentation of the image to be processed provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the block list provided by the present invention.

[0020] Figure 4 It is a schematic diagram of the visual model inference process provided by the present invention.

[0021] Figure 5 It is a schematic structural diagram of the visual model inference device based on image segmentation provided by the present invention.

[0022] Figure 6 It is a schematic structural diagram of the electronic device provided by the present invention. Specific Embodiments

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the attached drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0024] The following describes the visual model inference method and device based on image segmentation of the present invention in conjunction with the attached drawings. Figure 1 It is a schematic flowchart of the visual model inference method based on image segmentation provided by the present invention. As Figure 1 shown, the method includes the following steps 101 to 104, which will be specifically described below.

[0025] Step 101: Segment the image to be processed according to a preset size to obtain a plurality of image slices, and record the image index of each image slice in the block list.

[0026] Here, there are multiple images to be processed that need to be inferred using a visual model, and the image sizes of each image to be processed are different. The image size generally refers to the width and height of the image. Therefore, for any image to be processed of any size, it is sliced according to a preset size to obtain multiple image slices. The preset size can be, for example, 32×32. When the image to be processed is smaller than the preset size, it can be padded with blank pixels to reach the preset size or an integer multiple of the preset size.

[0027] Next, record the image index of each image slice in a block list. This image index is set according to the order of the image slices and is used to query the corresponding image slice and can also be used to represent the position in the image. Here, in the embodiment of the present invention, a block list is constructed to record the corresponding image index for subsequent storage of the image slices.

[0028] For example Figure 2 As shown, a to-be-processed image is sliced according to a preset size, and finally 15 image slices are obtained, and image indices 1, 2, 3, …, 15 are set for each image slice.

[0029] Step 102: Divide the physical video memory storing the image to be processed into multiple video memory blocks, and store the image slices through the video memory blocks.

[0030] Before inference, the image to be processed generally needs to be stored by allocating corresponding physical video memory, such as the memory of a computer's graphics card device or a graphics processing unit. Similar to the slicing process of the image to be processed, here the physical video memory storing the image to be processed is also divided into multiple video memory blocks, denoted as block. Of course, the memory allocation size of the video memory block is also preset, and specifically, it can be reasonably preset according to the model structure of the visual model, the model input characteristics, the model calculation precision type, etc. Then, the sliced image slices are stored one by one through the divided video memory blocks.

[0031] For example Figure 3 As shown, the physical video memory storing the image to be processed is divided into 7 video memory blocks, such as video memory block 1, video memory block 2, …, video memory block 7, and the 15 image slices obtained by slicing the image to be processed are respectively stored in video memory blocks 1, 2, 4, 6, while video memory blocks 3, 5, 7 in the physical video memory where no image slices are stored are in an idle state.

[0032] Step 103: When the visual model performs inference, obtain the image slices from the video memory blocks according to the block list, and splice the image slices according to the image index to obtain the image to be inferred.

[0033] When the visual model performs inference, it needs to obtain the image to be processed from the physical video memory. Therefore, here, image slices are obtained from the video memory blocks according to the block list. The image index of the image slice can be used to query the block list, and then the video memory blocks of the physical video memory can be traversed to determine the video memory block where the image slice is stored, so as to obtain the corresponding image slice. Then, the image slices can be stitched together according to the image index to obtain the image to be inferred. The image index can be used to represent the position of the image slice in the image. Therefore, stitching according to the image index can restore a complete image as the image to be inferred.

[0034] Step 104: Input the image to be inferred into the visual model for inference to obtain the image inference result.

[0035] Finally, perform image inference. Input the image to be inferred into the visual model for inference to obtain the image inference result. The visual model can be a convolutional neural network, a Transformer model, or other models or encoders used for image processing. The inference process can be image segmentation, object detection, or image reconstruction, etc. The finally obtained image inference result can be the corresponding image classification category, image segmentation result, or the detected object in the image, etc.

[0036] In the embodiment of the present invention, by dividing the image to be processed and the physical video memory respectively, and then using the divided video memory blocks to store the sliced image to be processed, and recording it in the block list for the visual model to perform inference. In this way, even for images to be processed with different dynamic sizes, they can be sliced according to a fixed size and stored in the divided video memory blocks as needed, which can effectively utilize the video memory resources, without the need for pre-allocation in advance, and improve the video memory utilization rate.

[0037] In some embodiments, the storage process of storing image slices through video memory blocks can be implemented through the following process, which specifically includes: first, combine the image slices into image blocks according to a preset quantity, record the image blocks and the number of image slices included in the image blocks in the block list, and establish a mapping relationship between the image blocks and the video memory blocks in the block list. Finally, store the image slices in the image blocks into the video memory blocks according to the mapping relationship.

[0038] Here, since the number of image slices may be very large, it is impossible to allocate a video memory block for each image slice to store. Therefore, here, the image slices are first combined into image blocks according to a preset quantity, so as to allocate video memory blocks for storage with the image blocks as the storage unit.

[0039] Such as Figure 2As shown, the image to be processed is segmented into 15 image slices, and then combined into 4 image blocks in groups of 4. Image slices 1 to 4, 5 to 8, and 9 to 12 are respectively combined into image blocks 1, 2, and 3, while the remaining 3 image slices 13 to 15 are combined into image block 4.

[0040] Then, the image blocks and the number of image slices included in the image blocks are recorded in the block list. As Figure 3 shown, when storing the image, image blocks 1, 2, 3, and 4 are respectively recorded in the block list, and the number of image slices corresponding to each image block is also recorded in the filled column of the block list. Here, it is also necessary to establish a mapping relationship between the image block and the video memory block in the block list. Since the video memory block is allocated for storage in units of image blocks, a mapping relationship needs to be established for each image block. The video memory block serves as the actual address for storing the image slices, and the actual address of the image slices can be conveniently queried through the mapping relationship during query. For example Figure 3 shown, the mapping relationship between the image block and the video memory block is recorded in the physical block number column of the block list. The image blocks 1, 2, 3, and 4 composed of the image slices of the image to be processed respectively map to the video memory blocks 1, 4, 6, and 2 of the physical video memory.

[0041] Finally, the image slices in the image block are stored in the video memory block according to the mapping relationship. After the mapping relationship is determined, the image storage can be executed, and the image slices in the image block are stored in the corresponding video memory block. As Figure 3 shown, image blocks 1, 2, 3, and 4 are respectively stored in the video memory blocks 1, 4, 6, and 2 of the physical video memory.

[0042] In the embodiment of the present invention, the image slices are combined into image blocks according to a reasonable quantity, and the video memory blocks are allocated in units of image blocks, and a mapping relationship between the image block and the video memory block is established for image storage. In this way, there is no need to allocate video memory blocks for each image slice, which can effectively save video memory resources. And by establishing a mapping relationship between the image and the physical video memory, the image slices can be quickly queried and obtained during model inference.

[0043] In some embodiments, the process of obtaining image slices from the video memory block according to the block list can be implemented through the following process. First, according to the image index of the image slice, the mapping relationship between the image block and the video memory block is queried from the block list. Then, according to the mapping relationship, the target video memory block corresponding to the storage of the image slice is determined from the physical video memory, and the image slice corresponding to the image index is obtained according to the target video memory block.

[0044] When the visual model performs inference, it is necessary to obtain the corresponding image to be inferred from the physical video memory. This requires querying for the corresponding image slice in the physical video memory. First, query the block list according to the image index. Based on the image index, the image block corresponding to the image slice can be determined. Then, query the physical block number column of the block list to find the mapping relationship between the image block and the video memory block, so as to determine the corresponding target video memory block in the physical video memory. By traversing the target video memory block, the image slice corresponding to the image index can be queried and obtained.

[0045] For example, as Figure 3 shown, for the image slice with image index 7, first determine that the image block it is in is image block 2. Then, query the physical block number column to determine that there is a mapping relationship between image block 2 and video memory block 4 in the physical video memory. Video memory block 4 is the target video memory block. Finally, in video memory block 4, the corresponding image slice can be obtained according to the image index 7. For each image slice, a similar method can be used to obtain the corresponding image slice from the physical video memory one by one.

[0046] In the embodiment of the present invention, when obtaining the image slice from the physical video memory, by using the mapping relationship established between the preset image index and the block list, the image slice can be quickly queried and obtained for stitching into the image to be inferred for inference, which can improve the acquisition efficiency of the image.

[0047] In some embodiments, when the image to be inferred is input into the visual model for inference, some intermediate activation values during the inference process can also be stored in the divided video memory blocks. The intermediate activation values can be, for example, the feature maps of the image to be inferred, that is, the feature maps output by each layer of the network in the visual model. The following specifically introduces the specific process when the visual model performs inference on the image to be inferred.

[0048] First, during the process of the visual model extracting the feature maps of the image to be inferred, obtain the feature maps extracted by each layer of the network and divide the feature maps to obtain multiple feature sub - maps.

[0049] As Figure 4 shown, the visual model is, for example, composed of multiple convolutional layer networks connected in series. After the image to be inferred is input into the visual model, each convolutional layer network will output the corresponding feature maps, and these feature maps also need to be allocated corresponding physical video memory for storage.

[0050] Next, similar to the image to be processed, the feature map is sliced to obtain multiple feature sub - maps. Since the sizes of the feature maps obtained by each layer of the network are different, they cannot be sliced according to a unified size. Therefore, here, for the size of the feature map obtained by each layer of the network, the slicing size is reasonably preset to ensure that the number of feature sub - maps obtained by slicing each feature map is the same as the number of image slices obtained when slicing the image to be processed. For example, if the image to be processed is sliced into 9 image slices, then the feature maps obtained by each layer of the network are also sliced into 9 feature sub - maps.

[0051] Similarly, here, a corresponding feature index is also set for each sliced feature sub - map. The feature indices of the feature sub - maps are recorded in the block list, and the feature sub - maps are stored in the memory blocks divided by the physical video memory. The storage method of the feature sub - maps is similar to that of the image slices. First, the feature sub - maps are combined into feature blocks according to a preset number, then the feature blocks and the number of feature sub - maps included in the feature blocks are recorded in the block list, and a mapping relationship between the feature blocks and the memory blocks is established in the block list. Finally, according to this mapping relationship, the feature sub - maps in the feature blocks are stored in the memory blocks.

[0052] When the vision model performs inference, the feature sub - maps are queried from the block list according to the feature indices. Similarly, the query and acquisition of the feature sub - maps are similar to those of the image slices. First, according to the feature indices of the feature sub - maps, the mapping relationship between the feature blocks and the memory blocks is queried from the block list, then according to the mapping relationship, the target memory block where the feature sub - map is stored is determined from the physical video memory, and the feature sub - map corresponding to the feature index is obtained according to the target memory block.

[0053] After obtaining the feature sub - maps, the feature sub - maps are stitched according to the feature indices to obtain the feature map to be processed. The stitching process of the feature sub - maps here is similar to that of the image slices and will not be elaborated here.

[0054] Finally, the feature map to be processed output by the last layer of the network in the vision model is mapped to obtain the image inference result.

[0055] The feature maps output by each layer of the network in the vision model are stored in the physical video memory in the above - mentioned similar way. During the processing of the next layer of the network, the corresponding feature map to be processed is obtained from the physical video memory for processing, and the corresponding feature map to be processed output after processing is also input into the physical video memory. Until the last layer of the network of the vision model, the feature map to be processed output by the last layer of the network in the vision model is mapped to obtain the image inference result. The mapping process can be realized through the activation function of the fully - connected layer neural network or the multi - layer perceptron. The image inference result can be the category of the corresponding image classification, the image segmentation result, or the target detected in the image, etc.

[0056] As shown Figure 4 in the figure, nine image slices are obtained by slicing a to-be-processed image, namely image slices 1 to 9, and then they are respectively stored in corresponding memory blocks of the physical video memory. When the vision model performs inference, it queries from the physical video memory to obtain the nine image slices, and inputs the to-be-inferred image obtained by splicing into the first convolutional layer network of the vision model, that is, convolutional layer network 1. Then, the feature map output by convolutional layer network 1 is also sliced into nine feature sub-maps, which are stored in the corresponding memory blocks of the physical video memory in a similar manner to the image slices. After that, continue to query from the physical video memory in a similar manner to obtain the nine feature sub-maps, splice them to obtain the corresponding to-be-processed feature map, and input it into the second convolutional layer network of the vision model for processing, that is, convolutional layer network 2. And so on, until the to-be-processed feature map corresponding to the Nth convolutional layer network (that is, convolutional layer network N) is output. Here, the to-be-processed feature map output by the last convolutional layer network of the vision model, that is, the to-be-processed feature map output by convolutional layer network N, is subjected to mapping processing to obtain the corresponding image inference result.

[0057] In the embodiment of the present invention, in a similar manner to the image slices obtained by slicing the to-be-processed image, the feature map output by each convolutional layer network of the vision model is also sliced and stored in the divided memory blocks. In this way, during the inference process of the vision model, the utilization rate of the video memory can be effectively improved, and the inference efficiency of the vision model can be improved.

[0058] In some embodiments, considering that the to-be-processed images have various different sizes and it is difficult to preset the slicing size, therefore, before slicing the to-be-processed image according to the preset size, the present invention embodiment also uniformly performs padding processing on the to-be-processed image to ensure that it can be sliced according to the preset size.

[0059] Specifically, perform size padding processing on the to-be-processed image, and construct a padding configuration file according to the size padding result. For each to-be-processed image, size padding processing is uniformly performed. For example, the preset slicing size of the to-be-processed image is 32×32, and the size of some to-be-processed images is less than 32×32 or not an integer multiple of 32×32. In this way, the to-be-processed image cannot be reasonably sliced according to 32×32. Therefore, here through size padding processing, it is ensured that the size of each to-be-processed image is an integer multiple of 32×32. The padding processing method can be to fill with blank pixels to increase the width and height of the to-be-processed image and make the size increase to an integer multiple of 32×32.

[0060] Due to the blank pixels filled in the image to be processed, the size and the number of pixels of the image are increased. These pixels may affect the inference calculation process of the visual model, resulting in errors in the inference results. Therefore, before the inference output result, the features of these pixels need to be cropped off.

[0061] For the convenience of subsequent cropping, in the embodiment of the present invention, after performing size filling processing on the image to be processed, a filling configuration file is constructed according to the size filling result. The size filling result of this filling configuration file records the size filling situation of the image to be processed, such as how many pixels are filled in height and width, how many times the size is increased, and so on. The size filling result of this filling configuration file is used to perform size cropping processing on the predicted feature map according to the size filling result before the visual model outputs the image inference result. The predicted feature map is the feature map used for mapping processing to obtain the image inference result, that is, the feature map before mapping processing through a fully connected layer neural network or a multi-layer perceptron.

[0062] For example, the size of a certain image to be processed is 16×16. After the size filling processing on the periphery of the image, the size is doubled, that is, 32×32. Then, when the visual model performs inference and outputs the result, the periphery of the predicted feature map is subjected to size cropping processing, cropped to half of the original size, and then the final image inference result is output using the predicted feature map after the size cropping processing.

[0063] In the embodiment of the present invention, after the image to be processed is segmented, size filling processing is performed on the image to be processed, so that for images to be processed of various sizes, they can all be segmented according to a preset size to ensure the integrity of the image. And through the post-cropping processing, the influence of the filled part on the inference process is also eliminated, ensuring that the output image inference result is not affected.

[0064] In some embodiments, after the image to be inferred is input into the visual model for inference to obtain the image inference result, the image index corresponding to the image slice of the image to be inferred and the feature index corresponding to the feature sub-map are also removed from the block list, and the storage space of the video memory block used to store the image slice and the feature sub-map is released in the physical video memory.

[0065] Since there is more than one image to be processed that needs to be input into the visual model for inference, when a certain image to be processed is inferred, the image index corresponding to the image slice of the processed image to be inferred and the feature index corresponding to the feature sub-map need to be removed. For example Figure 3 in, the image index, feature index, and mapping relationship recorded in the physical block number column of the block list are all cleared, thereby updating the block list to facilitate storing the image index, feature index, and mapping relationship corresponding to the next image to be processed.

[0066] After the block list is updated, next, the storage space of the video memory blocks used to store the image slices and the feature sub - graphs is released in the physical video memory. For example Figure 4 In [example], after the inference of the image to be processed is completed, the storage spaces of the video memory blocks 1, 2, 3 for storing the image slices and the video memory blocks 4, 5, 6 for storing the feature sub - graphs in the physical video memory are all released, facilitating the processing and storage of the next image to be processed.

[0067] In this way, for each image to be processed, after the inference using the visual model is completed, the block list is updated and the storage space is released in the above - mentioned manner.

[0068] In the embodiment of the present invention, after the inference of the image to be processed is completed, the block list and the physical video memory are updated in a timely manner and the storage space is released, which can reasonably and effectively utilize the physical video memory, avoiding the over - load of the use of the physical video memory and affecting the inference efficiency of the visual model.

[0069] Next, the visual model inference device based on image segmentation provided by the present invention will be described. The visual model inference device based on image segmentation described below can be correspondingly referred to the visual model inference method based on image segmentation described above.

[0070] As Figure 5 shown, the visual model inference device based on image segmentation specifically includes: a segmentation module 501, a storage module 502, an acquisition module 503, and an inference module 504. Specifically, the segmentation module 501 is used to segment the image to be processed according to a preset size to obtain a plurality of image slices, and record the image index of each image slice in the block list; the storage module 502 is used to divide the physical video memory storing the image to be processed into a plurality of video memory blocks, and store the image slices through the video memory blocks; the acquisition module 503 is used to, when the visual model performs inference, obtain the image slices from the video memory blocks according to the block list, and splice the image slices according to the image index to obtain the image to be inferred; the inference module 504 is used to input the image to be inferred into the visual model for inference to obtain the image inference result.

[0071] It should be noted that the beneficial effects of the visual model inference device based on image segmentation here and the visual model inference method based on image segmentation above can correspond to each other. Therefore, the beneficial effects of the visual model inference device based on image segmentation will not be elaborated here.

[0072] Figure 6 Illustrates a schematic diagram of the physical structure of an electronic device, as Figure 6As shown in the figure, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communications interface 620, and the memory 630 complete communication with each other through the communication bus 640. The processor 610 may call logical instructions in the memory 630 to execute a visual model inference method based on image segmentation. The method includes: segmenting the image to be processed according to a preset size to obtain a plurality of image slices, and recording the image index of each image slice in a block list; dividing the physical video memory storing the image to be processed into a plurality of video memory blocks, and storing the image slices through the video memory blocks; when the visual model performs inference, obtaining the image slices from the video memory blocks according to the block list, and splicing the image slices according to the image index to obtain an image to be inferred; inputting the image to be inferred into the visual model for inference to obtain an image inference result.

[0073] In addition, when the logical instructions in the above-mentioned memory 630 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0074] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the visual model inference method based on image segmentation provided by the above-mentioned various methods. The method includes: segmenting the image to be processed according to a preset size to obtain a plurality of image slices, and recording the image index of each image slice into a block list; dividing the physical video memory storing the image to be processed into a plurality of video memory blocks, and storing the image slices through the video memory blocks; when the visual model performs inference, obtaining the image slices from the video memory blocks according to the block list, and splicing the image slices according to the image index to obtain an image to be inferred; inputting the image to be inferred into the visual model for inference to obtain an image inference result.

[0075] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the visual model inference method based on image segmentation provided by the above-mentioned various methods. The method includes: segmenting the image to be processed according to a preset size to obtain a plurality of image slices, and recording the image index of each image slice into a block list; dividing the physical video memory storing the image to be processed into a plurality of video memory blocks, and storing the image slices through the video memory blocks; when the visual model performs inference, obtaining the image slices from the video memory blocks according to the block list, and splicing the image slices according to the image index to obtain an image to be inferred; inputting the image to be inferred into the visual model for inference to obtain an image inference result.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0077] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual model reasoning method based on image segmentation, characterized in that: include: The image to be processed is divided into multiple image slices according to a preset size, and the image index of each image slice is recorded in a block list; Dividing a physical video memory storing the image to be processed into a plurality of video memory blocks, and storing the image slices through the video memory blocks; When the visual model performs inference, the image slices are obtained from the video memory block according to the block list, and the image slices are spliced ​​according to the image index to obtain the image to be inferred; Inputting the image to be inferred into the visual model for inference to obtain an image inference result; The step of inputting the image to be inferred into the visual model for inference to obtain an image inference result includes: In the process of extracting the feature graph of the image to be inferred by the visual model, obtaining the feature graph extracted by each layer of the network, and dividing the feature graph to obtain a plurality of feature subgraphs; Recording the feature index of the feature subgraph into the block list, and storing the feature subgraph through the video memory blocks divided by the physical video memory; When the visual model performs reasoning, the feature subgraph is obtained by querying from the block list according to the feature index; The feature subgraphs are spliced ​​according to the feature indexes to obtain a feature graph to be processed; The feature map to be processed output by the last layer of the network in the visual model is mapped to obtain the image inference result.

2. The visual model reasoning method based on image segmentation according to claim 1, characterized in that: The storing the image slices by the video memory block includes: Combining the image slices into image blocks according to a preset number; Recording the image block and the number of image slices included in the image block in a block list, and establishing a mapping relationship between the image block and the video memory block in the block list; The image slices in the image block are stored in the video memory block according to the mapping relationship.

3. The visual model reasoning method based on image segmentation according to claim 1, characterized in that: The acquiring the image slice from the video memory block according to the block list comprises: According to the image index of the image slice, query the mapping relationship between the image block and the video memory block from the block list; According to the mapping relationship, a target video memory block corresponding to the storage of the image slice is determined from the physical video memory, and the image slice corresponding to the image index is obtained according to the target video memory block.

4. The visual model reasoning method based on image segmentation according to claim 1, characterized in that: Before segmenting the image to be processed according to the preset size, the method further includes: Perform size filling processing on the image to be processed, and construct a filling profile according to the size filling result; The padding configuration file is used to perform size cropping processing on the predicted feature map according to the size padding result before the visual model outputs the image inference result. The predicted feature map is a feature map used for mapping processing to obtain the image inference result.

5. The visual model reasoning method based on image segmentation according to claim 1, characterized in that: After inputting the image to be inferred into the visual model for inference to obtain an image inference result, the method further includes: Remove the image index corresponding to the image slice of the image to be inferred and the feature index corresponding to the feature subgraph from the block list; The storage space of the video memory block used to store the image slice and the feature sub-image is released in the physical video memory.

6. A visual model reasoning device based on image segmentation, characterized in that: include: A segmentation module, used for segmenting the image to be processed according to a preset size to obtain multiple image slices, and recording the image index of each image slice into a block list; A storage module, used for dividing the physical video memory storing the image to be processed into a plurality of video memory blocks, and storing the image slices through the video memory blocks; An acquisition module, used for acquiring the image slices from the video memory block according to the block list when the visual model is inferring, and splicing the image slices according to the image index to obtain the image to be inferred; An inference module, used for inputting the image to be inferred into the visual model for inference to obtain an image inference result; The step of inputting the image to be inferred into the visual model for inference to obtain an image inference result includes: In the process of extracting the feature graph of the image to be inferred by the visual model, obtaining the feature graph extracted by each layer of the network, and dividing the feature graph to obtain a plurality of feature subgraphs; Recording the feature index of the feature subgraph into the block list, and storing the feature subgraph through the video memory blocks divided by the physical video memory; When the visual model performs reasoning, the feature subgraph is obtained by querying from the block list according to the feature index; The feature subgraphs are spliced ​​according to the feature indexes to obtain a feature graph to be processed; The feature map to be processed output by the last layer of the network in the visual model is mapped to obtain the image inference result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the visual model reasoning method based on image segmentation as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the visual model reasoning method based on image segmentation as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the visual model reasoning method based on image segmentation as described in any one of claims 1 to 5 is implemented.

Images