A model inference-based image size adjustment method and device

Abstract

This invention discloses an image size adjustment method, apparatus, and storage medium based on model inference. The method includes: generating a summary mapping table; obtaining the data size of an image or video frame; determining whether the image size or video frame size is within a specific range; if within the specific range, determining whether the image size or video frame size is greater than the maximum input size based on the memory and video memory of the execution device and the summary mapping table; if not within the specific range, preprocessing the image or video frame to scale the image size or video frame size proportionally to the specific range, performing data block processing on the compressed image or compressed video frame, performing model inference, obtaining the inference result, and outputting the stitched inference result enlarged to the original image or video frame size before compression. This method can improve super-resolution and restoration effects and has a short processing time.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image size adjustment method and apparatus based on model reasoning. Background Technology

[0002] In image processing tasks such as image classification and scene recognition, when models with high performance requirements, such as super-resolution or restoration, are deployed on computers or mobile devices, the size of the obtained image or video frames differs significantly from the inference results obtained from the original size. Furthermore, the inference speed is slow, affecting processing efficiency. In existing technologies, when performing image rotation or distortion correction on the original image, discrete mapping of the pixel positions of the original image is required. The pixel information of the original image is stored in memory, and the CPU needs to read the pixel information from memory. This method has low read / write efficiency, resulting in insufficient memory bandwidth, which in turn makes image processing time longer. Therefore, a model-based image size adjustment method is needed to shorten the processing time. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a model-based inference-based image resizing method and apparatus that can improve super-resolution and restoration effects while reducing processing time.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] The first aspect of this invention provides an image resizing method based on model inference, comprising:

[0006] Test and record the peak memory usage and peak video memory usage of the model under different input image sizes or video frame sizes, and generate summary mapping tables of image size and peak memory usage, and image size and peak video memory usage.

[0007] Obtain the memory and video memory of the execution device, receive image or video frames, and obtain the data size of the image or video frames;

[0008] Determine whether the image size or video frame size is within a specific range;

[0009] If it is within a specific range, then based on the memory and video memory of the execution device and the summary mapping table, it is determined whether the image size or video frame size is greater than the maximum input size;

[0010] If the input size is greater than the maximum input size, the image or video frame is divided into data blocks according to the device's memory and video memory, as well as the summary mapping table. The block image or video frame is then used for model inference to obtain the inference result, which is then output.

[0011] If the input size is not greater than the maximum input size, then perform model inference directly, obtain the inference result, and output the inference result.

[0012] If the image or video frame is not within a specific range, preprocessing is performed to scale the image or video frame size proportionally to the specific range to obtain a compressed image or compressed video frame.

[0013] The system performs data segmentation on compressed images or compressed video frames, performs model inference on the segmented compressed images or compressed video frames to obtain inference results, stitches the inference results together, and enlarges the stitched inference results to the size of the original image or video frame before compression for output.

[0014] Furthermore, the reasoning results are spliced ​​together, and a fixed overlapping area is retained at the splicing position between the blocks.

[0015] Furthermore, the data segmentation is performed by dividing the image or video frame into equal blocks in the width and height directions while retaining a fixed overlapping width or height.

[0016] Furthermore, the width or height of the evenly divided block is determined as follows:

[0017] Number of blocks in the width or height direction = width or height of the original image ÷ maximum input size;

[0018] The width or height of the evenly divided block = ((number of blocks in the width or height direction - 1) * width or height of the overlapping area + width or height of the original image) ÷ number of blocks in the width or height direction.

[0019] Furthermore, the maximum input size is set by looking up a mapping table to set the maximum input size of the model. When the peak memory usage and peak video memory usage are both less than the maximum size of the device's memory and video memory, the maximum size of the device's memory and video memory is set as the maximum input size of the model.

[0020] Furthermore, the method of scaling the image size or video frame size proportionally to a specific range adopts the resize method.

[0021] A second aspect of the present invention provides an image size adjustment apparatus based on model inference, comprising: a processor and a memory;

[0022] The memory stores computer-executed instructions;

[0023] The processor executes computer execution instructions stored in the memory, causing the processor to execute the image size adjustment method based on model inference.

[0024] A third aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image size adjustment method based on model inference.

[0025] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0026] This invention obtains results by inferring from a reduced image or video frame size, and then enlarges the results back to the original image or video frame size. The difference between this and the results obtained by directly inferring from the original image or video frame size is minimal. The inference speed at reduced size is much faster than direct inference, thereby improving super-resolution and restoration effects and shortening the processing time for super-resolution and restoration. This method uses an equal-division method, with only one type of block. Therefore, there is no need to infer for different sizes, and the GPU does not need extra time to switch, further shortening the image or video frame processing time. A fixed overlapping area is maintained at the stitching position between blocks to avoid block effects, ensuring that the stitched results obtained from inference can avoid differences at the stitching position. Attached Figure Description

[0027] Figure 1 This is a flowchart of an image resizing method based on model inference. Detailed Implementation

[0028] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be noted that the terms "comprising," "including," and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this invention, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. This method can be applied to computer or mobile terminal devices with high performance requirements for execution equipment, such as super-resolution or repair equipment.

[0031] like Figure 1 As shown, an image resizing method based on model inference includes:

[0032] S1. Test and record the peak memory usage and peak video memory usage of the model under different input image sizes or video frame sizes, and generate a summary mapping table of image size and peak memory usage, and image size and peak video memory usage.

[0033] S2. Obtain the memory and video memory of the execution device, receive image or video frames, and obtain the data size of the image or video frames;

[0034] S3. Determine whether the image size or video frame size is within a specific range;

[0035] S301. If it is within a specific range, then based on the memory and video memory of the execution device and the summary mapping table, determine whether the image size or video frame size is greater than the maximum input size.

[0036] S3011. If the input size is greater than the maximum input size, the image or video frame is divided into data blocks according to the device's memory and video memory, as well as the summary mapping table. The image or video frame blocks are then used for model inference to obtain the inference result, which is then output.

[0037] S3012. If the input size is not greater than the maximum input size, then directly perform model inference, obtain the inference result, and output the inference result.

[0038] S302. If the image or video frame is not within the specified range, preprocess the image or video frame to scale the image size or video frame size to the specified range to obtain a compressed image or compressed video frame.

[0039] S3021. Perform data block processing on the compressed image or compressed video frame, perform model inference on the block compressed image or compressed video frame to obtain the inference result; stitch the inference result together, and enlarge the stitched inference result to the size of the image or video frame before compression and output it.

[0040] The method of scaling the image size or video frame size proportionally to a specific range uses a resize method to obtain a compressed image or compressed video frame, and then determines whether the compressed image size or compressed video frame size scaled proportionally within the specific range is greater than the maximum input size.

[0041] The reasoning results are spliced ​​together, and a fixed overlapping area is retained at the splicing position between the blocks to avoid block effect, so that the spliced ​​results can avoid differences at the splicing position.

[0042] One embodiment involves recording the peak device memory and video memory usage of the model under different input sizes, and generating a mapping table between the model input size and memory and video memory as follows:

[0043] Prepare input data of different sizes. For example, for an image recording model, prepare image data of different resolutions; load the model and set the input size. When loading the model, set the size of the input data in the code; typically, the model will automatically adjust memory and video memory usage based on the size of the input data; perform inference on the input data and record the peak values ​​of memory and video memory usage; record memory and video memory usage during inference; use system monitoring tools or professional performance monitoring tools; repeat the test multiple times to ensure the accuracy of the results and calculate statistics such as mean and standard deviation; if it is necessary to compare the resource usage of different models or different algorithms, the same method can be used for testing and comparative analysis.

[0044] One embodiment is as follows: a specific range value is set to 4K; the memory and video memory of the execution device are obtained, and image or video frames are received to obtain the data size of the image or video frame; it is determined whether the image size or video frame size is greater than 4K; if it is not greater than 4K, based on the memory and video memory of the execution device and the summary mapping table, it is determined whether the image size or video frame size is greater than the maximum input size; if it is greater than the maximum input size, the image or video frame is processed by data block processing according to the memory and video memory of the device and the summary mapping table, model inference is performed on the block image or video frame to obtain the inference result, and the inference result is output; if it is not greater than the maximum input size, model inference is performed directly to obtain the inference result, and the inference result is output.

[0045] If the image or video frame is larger than 4K, it is preprocessed to scale the image or video frame size to within 4K to obtain a compressed image or video frame. The compressed image or video frame is then processed by data block segmentation, model inference is performed, inference results are obtained, the inference results are stitched together, and the stitched inference results are enlarged to the size of the original image or video frame before output.

[0046] Currently, it has been verified that some models with high requirements for execution device performance (super-resolution, repair) can be effectively deployed on execution devices with different performance levels after using this method. The overall effect and time consumption are better than the simple block partitioning method. After using this method to partition, the number and types of blocks are minimized. If the input size is larger than a certain size (e.g., 4K) and the original data details are relatively rich, the resize method can be used to compress it first. By keeping the number of blocks small and maintaining a specific overlapping area, the block effect in the result can be avoided as much as possible. Since there is only one type of block, the additional switching time of the GPU is avoided.

[0047] In the field of image compression, resizing refers to changing the size or resolution of an image. It is typically used to reduce the resolution of an image to decrease the size of the image file, thereby reducing the space it occupies during network transmission or storage.

[0048] In image compression, resizing typically involves two parameters: width and height. During resizing, interpolation algorithms are usually used to calculate new pixel values ​​to ensure the new image size is proportional to the original image size. Interpolation algorithms can calculate the color value of a new pixel based on the distance between consecutive pixels and their brightness values. Commonly used resizing algorithms include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Choosing a suitable resizing algorithm can balance the relationship between compressed image size and image quality.

[0049] One embodiment is: data segmentation is to divide the image or video frame into blocks in the width and height directions while retaining a fixed overlapping width or height;

[0050] The width or height of the evenly divided block is determined as follows:

[0051] Number of blocks in the width or height direction = width or height of the original image ÷ maximum input size;

[0052] The width or height of the evenly divided block = ((number of blocks in the width or height direction - 1) * width or height of the overlapping area + width or height of the original image) ÷ number of blocks in the width or height direction.

[0053] The method of calculating the width or height of equal-sized blocks is simple to implement: This algorithm only needs to divide the image into blocks of equal size, without complex calculations or transformations, making it relatively simple to implement; it has good compression effect: The method of calculating equal-sized blocks can better preserve some features and details of the image, while effectively reducing the size of the image file, thus ensuring compression effect while speeding up image transmission and storage; it can also be extended to high-dimensional data: This method can be extended to the compression of high-dimensional data, such as video compression, image compression at different resolutions, etc.; in vectorized computation: The computation of the method of calculating equal-sized blocks can be easily vectorized and parallelized, speeding up the computation.

[0054] The maximum input size is set by looking up a mapping table. When the peak memory usage and peak video memory usage are both less than the maximum size of the device's memory and video memory, the maximum size of the device's memory and video memory is set as the maximum input size of the model.

[0055] An image resizing device based on model inference includes: a processor and a memory;

[0056] The memory stores computer-executed instructions;

[0057] The processor executes computer execution instructions stored in the memory, causing the processor to execute the image size adjustment method based on model inference.

[0058] A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image size adjustment method based on model inference.

[0059] When the method of this invention is implemented on embedded terminals and mobile devices, it can activate and use the model based on different shape feature parameters, thereby solving the memory overhead problem as much as possible. It can be applied to non-end-to-end target detection networks and has strong versatility.

[0060] Unless otherwise specifically stated, terms such as processing, calculation, operation, determination, display, etc., may refer to the actions and / or processes of one or more processing or computing systems or similar devices that represent the manipulation and conversion of data representing physical (e.g., electronic) quantities within the registers or memory of the processing system into other data similarly representing physical quantities within the memory, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0061] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0062] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0063] Those skilled in the art will also understand that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments herein can be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in alternative ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0064] The steps of the methods or algorithms described in conjunction with the embodiments herein can be directly embodied in hardware, software modules executed by a processor, or a combination thereof. The software modules can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in a user terminal. Alternatively, the processor and storage medium can exist as discrete components in the user terminal.

[0065] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. This software code can be stored in memory units and executed by a processor. The memory units can be implemented within the processor or outside the processor; in the latter case, they are communicatively coupled to the processor via various means, as is well known in the art.

[0066] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

Claims

1. An image resizing method based on model inference, characterized in that, include: Test and record the peak memory usage and peak video memory usage of the model under different input image sizes or video frame sizes, and generate a summary mapping table of image size and peak memory usage, and image size and peak video memory usage; Obtain the memory and video memory of the execution device, receive image or video frames, and obtain the data size of the image or video frames; Determine whether the image size or video frame size is within a specific range; If it is within a specific range, then based on the memory and video memory of the execution device and the summary mapping table, it is determined whether the image size or video frame size is greater than the maximum input size; If the input size is greater than the maximum input size, the image or video frame is divided into data blocks according to the device's memory and video memory, as well as the summary mapping table. The block image or video frame is then used for model inference to obtain the inference result, which is then output. If the input size is not greater than the maximum input size, then perform model inference directly, obtain the inference result, and output the inference result. If the image or video frame is not within a specific range, preprocessing is performed to scale the image or video frame size proportionally to the specific range to obtain a compressed image or compressed video frame. The system performs data segmentation on compressed images or compressed video frames, performs model inference on the segmented compressed images or compressed video frames to obtain inference results, stitches the inference results together, and enlarges the stitched inference results to the size of the original image or video frame before compression for output.

2. The image size adjustment method based on model inference according to claim 1, characterized in that: The reasoning results are spliced ​​together, and a fixed overlapping area is retained at the splicing position between the blocks.

3. The image size adjustment method based on model inference according to claim 2, characterized in that: The data is divided into blocks in the width and height directions of the image or video frame, with a fixed overlap width or height.

4. The image size adjustment method based on model inference according to claim 3, characterized in that: The width or height of the evenly divided block is determined as follows: Number of blocks in the width or height direction = width or height of the original image ÷ maximum input size; The width or height of the evenly divided block = ((number of blocks in the width or height direction - 1) * width or height of the overlapping area + width or height of the original image) ÷ number of blocks in the width or height direction.

5. The image size adjustment method based on model inference according to claim 4, characterized in that: The maximum input size is set by looking up a mapping table. When the peak memory usage and peak video memory usage are both less than the maximum size of the device's memory and video memory, the maximum size of the device's memory and video memory is set as the maximum input size of the model.

6. The image size adjustment method based on model inference according to claim 5, characterized in that: The method of scaling the image size or video frame size to a specific range proportionally uses the resize method.

7. An image resizing device based on model inference, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the image resizing method based on model inference as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the image size adjustment method based on model reasoning as described in any one of claims 1 to 6.

Images