A dynamic processing method and device for image adaptive video memory and a related medium thereof

Abstract

This invention discloses a dynamic processing method, apparatus, and related medium for image adaptation to video memory. The method includes acquiring image information and video memory information of the hardware device and calculating correction coefficients; using the correction coefficients to correct the image to obtain scaling data; determining whether block segmentation is needed based on the scaling data, and processing the data to obtain output data; performing edge padding on the output data and then performing model inference to obtain model inference data; removing the edge padding from the model inference data to obtain de-edged model data; determining whether the original input image corresponding to the de-edged model data has undergone block segmentation, and processing the output image restoration data based on the determination. This invention dynamically adapts to operating devices with different computing power by segmenting and scaling the image, reducing image loss on the operating device and improving the display effect of the image on the operating device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a dynamic processing method and device for image adaptation of video memory and a related medium thereof. BACKGROUND

[0002] With the rapid development of computer technology, the image processing capability and effect of deep learning models are becoming stronger and stronger, but at the same time, the computing power requirement of the running device is also becoming higher and higher. Different resolution images have different display effects on different computing power running devices, and cannot be self-adaptively adjusted. For example, a high-resolution image displayed on a low-computing-power running device will result in image quality loss and poor display effect. Therefore, there is an urgent need for a processing method for adapting images according to the computing power of the running device. SUMMARY

[0003] Embodiments of the present application provide a dynamic processing method and device for image adaptation of video memory and a related medium thereof, aiming to solve the problem that different resolution images cannot be self-adaptively adjusted on different computing power running devices in the prior art, resulting in image quality loss and poor display effect.

[0004] In a first aspect, the embodiments of the present application provide a dynamic processing method for image adaptation of video memory, comprising:

[0005] obtaining image information of an original input image and video memory information of a hardware device, respectively, wherein the image information includes an original image resolution and a target image resolution, and the video memory information includes a video memory capacity, a target video memory occupation and a target video memory offset;

[0006] calculating a correction coefficient using the proportional relationship between the difference between the video memory capacity and the target video memory offset and the target video memory occupation, and correcting the proportional relationship between the target image resolution and the original image resolution using the correction coefficient to obtain scaling ratio data;

[0007] determining whether a block operation is needed according to the scaling ratio data, if not, scaling the original input image according to the scaling ratio data to obtain output data, and if yes, performing a block operation on the original input image, recalculating the scaling ratio data for each block, and determining again whether a block operation is needed and performing corresponding scaling to obtain output data;

[0008] performing data padding on the output data to obtain model training data;

[0009] inputting the model training data into a deep learning model for inference to obtain model inference data;

[0010] removing the data padding from the model inference data to obtain model edge-removed data;

[0011] The data output unit is configured to judge whether the original input image corresponding to the model edge-removed data has been subjected to the block operation, and if not, perform the equal-proportion scaling restoration calculation on the model edge-removed data to output image restoration data; if yes, perform the block splicing on the model edge-removed data, and then perform the equal-proportion scaling restoration calculation to output the image restoration data.

[0012] In a second aspect, an embodiment of the present application provides a dynamic processing device for image adaptation of display memory, comprising:

[0013] An information acquisition unit is configured to acquire image information of an original input image and display memory information of a hardware device respectively, wherein the image information comprises an original image resolution and a target image resolution, and the display memory information comprises a display memory capacity, a target display memory occupation and a target display memory offset;

[0014] A proportion adaptation unit is configured to calculate a correction coefficient by using a proportional relationship between a difference between the display memory capacity and the target display memory offset and the target display memory occupation, and correct a proportional relationship between the target image resolution and the original image resolution by using the correction coefficient to obtain scaling proportion data;

[0015] A block judgment unit is configured to judge whether a block operation is needed according to the scaling proportion data, and if not, scale the original input image according to the scaling proportion data to obtain output data; if yes, perform a block operation on the original input image, and re-calculate the scaling proportion data for each block, and then judge whether the block operation is needed and perform corresponding scaling again to obtain the output data;

[0016] A data padding unit is configured to perform data padding on the output data to obtain model training data;

[0017] A model training unit is configured to input the model training data into a deep learning model to perform inference and obtain model inference data;

[0018] A data edge-removing unit is configured to remove data padding from the model inference data to obtain model edge-removed data;

[0019] The data output unit is configured to judge whether the original input image corresponding to the model edge-removed data has been subjected to the block operation, and if not, perform the equal-proportion scaling restoration calculation on the model edge-removed data to output image restoration data; if yes, perform the block splicing on the model edge-removed data, and then perform the equal-proportion scaling restoration calculation to output the image restoration data.

[0020] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the dynamic processing method of image adaptive video memory of the first aspect when executing the computer program.

[0021] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the dynamic processing method of image adaptive video memory of the first aspect.

[0022] An embodiment of the present application provides a dynamic processing method of image adaptive video memory, comprising obtaining image information and video memory information of a hardware device and calculating a correction coefficient, correcting the image using the correction coefficient to obtain scaling ratio data; determining whether a block operation is needed according to the scaling ratio data, and obtaining output data after processing; performing data padding on the output data and then performing model reasoning to obtain model reasoning data; removing the data padding from the model reasoning data to obtain model edge-removed data; determining whether the original input image corresponding to the model edge-removed data has been subjected to the block operation, and outputting image restoration data according to the determination. The present application reduces image loss on a running device by dividing and scaling the image to dynamically adapt to running devices with different computing power, thereby improving the display effect of the image on the running device.

[0023] An embodiment of the present application also provides a dynamic processing device of image adaptive video memory, a computer device, and a storage medium, which also have the above beneficial effects. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A flowchart of a dynamic processing method of image adaptive video memory provided by an embodiment of the present application is shown in the figure.

[0026] Figure 2 Another flowchart of a dynamic processing method of image adaptive video memory provided by an embodiment of the present application is shown in the figure.

[0027] Figure 3 A schematic block diagram of a dynamic processing device of image adaptive video memory provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of the present application.

[0029] It should be understood that the terms "comprise" and "include" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0030] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0031] It should be further understood that the term "and / or" as used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0032] Please see the following Figure 1 , Figure 1 A flowchart of a dynamic processing method of image adaptive video memory provided by the embodiments of the present application is shown, which specifically includes steps S101-S107.

[0033] S101, image information of an original input image and video memory information of a hardware device are acquired respectively, wherein the image information includes original image resolution and target image resolution, and the video memory information includes video memory capacity, target video memory occupation and target video memory offset;

[0034] S102, a correction coefficient is calculated by using the proportional relationship between the difference between the video memory capacity and the target video memory offset and the target video memory occupation, and the proportional relationship between the target image resolution and the original image resolution is corrected by using the correction coefficient to obtain scaling ratio data;

[0035] S103, whether a block operation is needed is determined according to the scaling ratio data, if not, the original input image is scaled according to the scaling ratio data to obtain output data; if yes, the original input image is executed with a block operation, and scaling ratio data is recalculated for each block, and whether a block operation is needed and corresponding scaling is performed is determined again to obtain output data;

[0036] S104, data padding is performed on the output data to obtain model training data;

[0037] S105, the model training data is input into a deep learning model for inference to obtain model inference data;

[0038] S106, the model inference data is removed from the data padding to obtain model edge data;

[0039] S107, it is judged whether the original input image corresponding to the model edge data has been subjected to a blocking operation, if not, the model edge data is subjected to an equal proportion scaling restoration calculation, and image restoration data is output; if yes, the model edge data is subjected to a blocking splicing, and then an equal proportion scaling restoration calculation is performed to output image restoration data.

[0040] In combination Figure 2 As shown in the figure, in step S101, the original image resolution refers to the pixel resolution of the original image input into the deep learning model, which is usually represented by width x height (for example, 1920x 1080), and the resolution of the original image determines the clarity and details of the image; the target image resolution is the standard input resolution. Obtaining the original image resolution and the target image resolution helps to determine the scaling ratio of the image, so as to ensure that the image is adapted to the required resolution before being input into the model, thereby improving the performance and accuracy of the model. The video memory capacity is the total capacity of the video memory on the hardware device, which is usually represented by megabytes (MB), and the target video memory occupancy is the set allowed video memory occupancy, which represents the allocatable video memory capacity. The target video memory offset is an additional video memory capacity used to cope with fluctuations or uncertainties during calculation, which can ensure that it will not be interrupted due to insufficient video memory. Obtaining the video memory information helps to dynamically adapt the image scaling and blocking operation before model inference.

[0041] In step S102, the key step of calculating the correction coefficient is to use the proportional relationship between the difference between the video memory capacity and the target video memory offset and the target video memory occupancy. The correction coefficient reflects the relationship between the available video memory of the hardware device and the reserved space and the video memory demand during model inference, which helps to more accurately estimate the available video memory and avoid performance problems caused by insufficient video memory during inference. By using the calculated correction coefficient, the scaling relationship between the target image resolution and the original image resolution can be corrected. By multiplying this scaling ratio by the correction coefficient, the adjusted scaling data can be obtained.

[0042] Specifically, by adjusting the scaling ratio, the video memory capacity and the video memory occupation requirement can be better matched, which helps to effectively utilize the available video memory resources while maintaining reasonable image quality, thereby improving performance. Different correction coefficients for different video memory capacity intervals can be further considered. For example, for devices with smaller video memory capacity, a larger correction coefficient can be applied to more strictly limit the video memory occupation.

[0043] In an embodiment, the step S102 comprises:

[0044] The correction coefficient a is calculated according to the following formula:

[0045]

[0046] The ratio b between the target image resolution and the original image resolution is calculated according to the following formula:

[0047]

[0048] The scaling ratio data P is calculated according to the following formula:

[0049] P = a * b

[0050] Wherein, R 标 represents the target image resolution; R 原 represents the original image resolution; V 容 represents the video memory capacity; V 偏 represents the target video memory offset; V 占 represents the target video memory occupation.

[0051] In this embodiment, the correction coefficient a is used to adjust the subsequent image resolution ratio calculation. The correction coefficient a reflects the availability of the video memory on the hardware device, as well as the relationship between the target video memory occupation and the target video memory offset. By adjusting the correction coefficient a, the video memory requirement can be better matched to avoid performance problems caused by insufficient video memory. The ratio b between the target image resolution and the original image resolution is used to determine how much the original image should be scaled to adapt to the standard input resolution of the model, which helps to ensure that the input image has the correct resolution. Finally, the scaling ratio data uses the correction coefficient a and the ratio b to calculate the final scaling ratio data P.

[0052] Specifically, the scaling ratio P is determined according to the video memory condition of the hardware device and the image resolution requirement, and is used to guide the scaling operation of the image to adapt to the input requirement of the deep learning model. Different correction coefficients between different hardware device types and configurations can be further considered to better adapt to various hardware conditions. For multi-level hardware configurations, more complex correction strategies can be developed according to the availability and video memory speed of different video memory regions on the device to further improve performance. In terms of dynamic video memory management, the video memory usage can be monitored in real time, and the correction coefficient and scaling ratio can be dynamically adjusted according to the actual demand to ensure the best balance between performance and video memory resources.

[0053] In step S103, it is judged whether the block operation needs to be performed according to the scaling ratio data P. Specifically, a threshold or rule can be set. If P exceeds the threshold or rule, the block operation is not needed, and the overall scaling can be directly performed, thereby reducing unnecessary calculation and image processing and improving inference efficiency. If it is judged that the block operation needs to be performed, the original input image can be divided into multiple blocks, and each block can overlap a part to ensure the continuity of the image edge. For each block, the scaling ratio data is recalculated, which can be determined again according to the size of each block and the video memory resource. The recalculated scaling ratio data is used to judge whether each block needs further block operation; if the scaling ratio data of a block requires further block operation, the block operation can be recursively performed. For the block that does not need further block operation, the block can be scaled and then spliced into the final output data. For the block that needs further block operation, the above steps can be repeated until no block operation is needed.

[0054] In an embodiment, the step S103 comprises:

[0055] The original horizontal pixel number and the original vertical pixel number of the original image resolution are obtained, and the block horizontal pixel number and the block vertical pixel number of the preset block size are obtained;

[0056] The ratio of the original horizontal pixel number and the block horizontal pixel number is calculated and rounded up to obtain the horizontal block number, and the ratio of the original vertical pixel number and the block vertical pixel number is calculated and rounded up to obtain the vertical block number, thereby obtaining the vertical block number;

[0057] The horizontal block number and the vertical block number are multiplied to obtain the block number.

[0058] Further, a horizontal overlap length of two adjacent blocks is calculated according to a relationship of the horizontal block number, the block horizontal pixel number, and the original horizontal pixel number, and a vertical overlap length of two adjacent blocks is calculated according to a relationship of the vertical block number, the block vertical pixel number, and the original vertical pixel number.

[0059] The original input image is segmented according to the block number, the horizontal overlap length, and the vertical overlap length to obtain a corresponding number of blocks.

[0060] Further, a horizontal overlap length of two adjacent blocks is calculated according to a relationship of the horizontal block number, the block horizontal pixel number, and the original horizontal pixel number, and a vertical overlap length of two adjacent blocks is calculated according to a relationship of the vertical block number, the block vertical pixel number, and the original vertical pixel number.

[0061] L W =(N W *B W -P W ) / I W

[0062] A vertical overlap length of two adjacent blocks is calculated according to the following formula:

[0063] L H =(N H *B H -P H ) / I H

[0064] wherein L W represents the horizontal overlap length of the block; L H represents the vertical overlap length of the block; N W represents the horizontal block number; N H represents the vertical block number; B W represents the block horizontal pixel number; B H represents the block vertical pixel number; P W represents the original horizontal pixel number; P H represents the original vertical pixel number; I W represents the horizontal gap number of the block; and I H represents the vertical gap number of the block.

[0065] In this embodiment, the original image resolution includes the number of horizontal pixels and the number of vertical pixels, which are used to determine the size of the input image; the preset block size also includes the number of horizontal and vertical pixels, which are used to define how to divide the original image into small blocks; the ratio between the original number of horizontal pixels and the number of block horizontal pixels is calculated and rounded up to get the number of horizontal blocks. The ratio between the original number of vertical pixels and the number of block vertical pixels is calculated and rounded up to get the number of vertical blocks. The number of horizontal blocks and the number of vertical blocks are multiplied to get the total number of blocks, which represents how many small blocks the original image will be divided into.

[0066] Specifically, this process is used to divide a large original image into small blocks to better manage the video memory in the case of limited hardware resources. Dividing the image into small blocks can reduce the video memory requirement of each block, thus better adapting to the video memory limit. The block processing allows multiple blocks to be processed simultaneously, improving the parallelism of the calculation and thus speeding up the processing.

[0067] The horizontal overlap length refers to the number of pixels that two adjacent blocks overlap in the horizontal direction, which can be calculated according to the relationship between the number of horizontal blocks, the number of block horizontal pixels, and the number of original horizontal pixels. Generally, the horizontal overlap length can be calculated by the following formula: horizontal overlap length = (original horizontal pixel number - block horizontal pixel number * horizontal block number) / (horizontal block number - 1). The vertical overlap length refers to the number of pixels that two adjacent blocks overlap in the vertical direction, which can be calculated according to the relationship between the number of vertical blocks, the number of block vertical pixels, and the number of original vertical pixels. Generally, the vertical overlap length can be calculated by the following formula: vertical overlap length = (original vertical pixel number - block vertical pixel number * vertical block number) / (vertical block number - 1). Using the calculated horizontal overlap length and vertical overlap length, the original input image is divided into a corresponding number of blocks, each block including the number of block horizontal pixels and the number of block vertical pixels, while there is horizontal and vertical overlap between adjacent blocks. By calculating the overlap length, the continuity of the blocks is achieved, ensuring that the pixel data between adjacent blocks can overlap, thus maintaining the continuity of the image. The calculation of the overlap length is dynamically adjusted according to the number of blocks and the image size, making it adaptable to different images and hardware device conditions.

[0068] In step S104, data padding refers to the process of stitching the output data of each patch according to its position in the original image after the patching process to reconstruct the complete image. This includes filling additional pixel data at the edges of each patch to maintain the integrity of the image. Through data padding, it is ensured that the final image is continuous without gaps or missing areas, which is crucial for the quality of the training data for the model, as the model needs accurate and complete image data to learn features and patterns. The generated model training data can be used to train deep learning models, such as for super-resolution, denoising, or other computer vision tasks, so that the model can learn useful information from the output data of the patching process.

[0069] In an embodiment, the step S104 comprises:

[0070] calculating horizontal and vertical padding parameters according to the output data;

[0071] creating a padding tool based on image processing tools;

[0072] adding a corresponding number of pixel points on the left and right sides of the output data according to the horizontal padding parameters using the padding tool;

[0073] adding a corresponding number of pixel points on the top and bottom sides of the output data according to the vertical padding parameters using the padding tool;

[0074] using the output data with the added corresponding number of pixel points as the model training data.

[0075] In this embodiment, the horizontal and vertical padding parameters are calculated according to the patching of the output data and the overlap length during the patching process. The padding parameters include the number of pixels to be added on the left, right, top, and bottom sides. Based on image processing tools, a padding tool (also known as a filler) can be created to add pixel points at specific positions of the output data. This tool needs to be able to add pixels at the specified positions according to the provided padding parameters. Using the created padding tool, a corresponding number of pixel points are added on the left and right sides of the output data according to the horizontal padding parameters, ensuring that the output data has the required padding in the horizontal direction. Similarly, using the created padding tool, a corresponding number of pixel points are added on the top and bottom sides of the output data according to the vertical padding parameters, ensuring that the output data has the required padding in the vertical direction. The output data with the added corresponding number of pixel points is used as the model training data, so that the model will consider the padded image during the training process, thereby better adapting to incomplete or padded data.

[0076] Specifically, the calculation of the padding parameters and the use of the padding device can ensure that the generated training data has the required image size and continuity, so that the model can learn more information from the complete image data; it can also make the training data more suitable for specific model architectures and tasks, thereby improving the performance and generalization ability of the model; it can also adjust the padding parameters and padding strategies according to the needs of different tasks and models, for example, it can choose different padding values or padding modes to obtain the best data quality. The padding device can be customized according to different image processing tools and libraries to meet specific needs. The padding function can be implemented using popular image processing libraries such as OpenCV.

[0077] In step S105, before inference, a deep learning model suitable for the task needs to be selected first. The model can be a pre-trained model that has already been trained, or a custom model that has been fine-tuned according to specific tasks and data sets. The prepared model training data is input into the deep learning model for inference. This usually involves passing the data to the input layer of the model. The deep learning model processes the input data and performs specific tasks such as image super-resolution, image classification, object detection, etc., and then generates inference data. After the model inference is completed, the obtained inference data can include various outputs such as images after image super-resolution processing, classification labels, object detection results, etc., depending on the model and task. Deep learning models can perform various complex tasks by learning patterns and features, thereby improving automation and accuracy.

[0078] In step S106, data padding is added during model training data generation to ensure the continuity and integrity of the input data; in order to restore the model output to the original image size, it is necessary to remove these additional pixels from the model inference data. The process of removing data padding includes restoring the model inference data to the size of the original image, which can be achieved by cropping or scaling the model inference data to the size of the original image.

[0079] Specifically, removing data padding ensures that the model's output matches the size and scale of the original input image. By removing data padding, the model's output is more accurate; the model's output after removing data padding is easier to visualize and analyze. Different image processing methods and libraries can be considered for removing data padding, such as OpenCV, etc. For specific tasks, the strategy for removing data padding can be selected according to the characteristics of the model and the requirements of the output data. For example, in the super-resolution task, interpolation methods can be used to improve the quality of the output image; it can also be considered to perform other post-processing steps such as denoising, sharpening, color correction, etc. while removing data padding to further improve the quality of the output data.

[0080] In an embodiment, the step S106 comprises:

[0081] Finding the padding boundary value of the model inference data;

[0082] Using the horizontal padding parameter and the vertical padding parameter to perform difference comparison with the padding boundary value to obtain padding difference data;

[0083] According to the padding difference data, the model inference data is removed from the data padding to obtain the model edge data.

[0084] In this embodiment, first, the padding boundary value in the model inference data needs to be found, which identifies the position of the padding region in the model inference data, and can be realized by analysis or specific flag value. Using the difference comparison between the horizontal padding parameter and the vertical padding parameter and the padding boundary value, the padding difference data is obtained, which represents the distance of each pixel relative to the padding boundary. Using the padding difference data, the model inference data is removed from the data padding operation, which usually includes retaining the pixels far from the padding boundary and deleting or replacing the pixels close to the original image pixel value. Finally, the model inference data after removing the data padding is obtained.

[0085] Specifically, the calculation of the edge difference data can more accurately determine which pixels need to be retained or deleted, thereby improving the accuracy of the edge removal operation, helping to retain the detailed information in the model inference data and not losing important information in the padding region. The final model edge data is closer to the original image, so that higher quality results can be obtained in subsequent tasks.

[0086] In step S107, it is necessary to determine whether the original input image corresponding to the model edge data has been subjected to a blocking operation in the inference process, which can be determined by checking the metadata or other flags of the model inference data. If the original input image corresponding to the model edge data has not been subjected to a blocking operation (i.e., no blocking), the model edge data can be directly scaled to recover the calculation to obtain the output image recovery data. If the original input image corresponding to the model edge data has been subjected to a blocking operation, it is necessary to first splice the blocks and then perform the scaling calculation to recover the calculation, including recombining the blocking images into a complete image and then scaling it back to the size of the original input image. Through the above operation, the result obtained is the recovery data of the output image, which is consistent with the size and content of the original input image, ensuring that the output image has the same size as the original input image, so that the spatial information of the image is not lost. Moreover, the obtained output image recovery data can be more easily visualized and analyzed, and used for subsequent task display or research. The size of the output image is ensured to be consistent with the task requirements, so that it can be directly used for subsequent tasks such as image recognition, target detection, etc.

[0087] In summary, the image resolution and the level of device computing power in the present application depend on the requirements of the deep learning model. Generally, 1080P can be used as the standard value of image resolution. Greater than or equal to the standard value is high resolution, and less than the standard value is low resolution. Generally, 10 TFLOPS can be used as the standard value of computing power. Greater than the standard value is a high-power device (such as NVIDIA GeForce RTX 3060), and less than the standard value is a low-power device (such as NVIDIA GeForce GTX 780Ti).

[0088] For low-resolution input images and high-power devices, the technical selection of the present application does not perform image scaling, and the original image resolution is maintained. Therefore, no additional processing is performed on the image, and the image quality is maintained. When facing low-resolution input images and low-power devices, the technology of the present application uses an adaptive scaling ratio to better adapt to the performance of low-power devices with less quality loss, thereby improving the inference speed. For high-resolution input images and high-power devices, the technology of the present application also uses an adaptive scaling ratio to fully utilize the performance of high-power devices with less quality loss, thereby improving the inference speed. When facing high-resolution input images and low-power devices, although the adaptive scaling ratio may cause a large quality loss, through the operations of block slicing and re-adaptation, the present application successfully reduces the quality loss and better adapts to the performance of low-power devices, thereby ensuring the feasibility of inference. The technology of the present application realizes adaptive image processing between different resolution inputs and different power devices to balance the trade-off between deep learning model inference speed and image quality. By losing a small amount of image quality, more performance space is obtained, thereby improving the universality and applicability of the deep learning model and reducing the demand for computing device computing power.

[0089] In combination Figure 3 As shown in the accompanying drawings, Figure 3 A schematic block diagram of an image adaptive video memory dynamic processing device provided by an embodiment of the present application is shown. The image adaptive video memory dynamic processing device 300 includes:

[0090] An information acquisition unit 301 is configured to acquire image information of an original input image and video memory information of a hardware device, respectively. The image information includes an original image resolution and a target image resolution. The video memory information includes a video memory capacity, a target video memory occupation, and a target video memory offset.

[0091] The proportional adaptation unit 302 is configured to calculate a correction coefficient by using a proportional relationship between a difference between the video memory capacity and the target video memory offset and the target video memory occupancy, correct a proportional relationship between the target image resolution and the original image resolution by using the correction coefficient, and obtain scaling proportion data;

[0092] The block judgment unit 303 is configured to judge whether a block operation is needed according to the scaling proportion data, if not, scale the original input image according to the scaling proportion data to obtain output data, if yes, perform a block operation on the original input image, recalculate scaling proportion data for each block, and again judge whether a block operation is needed and perform corresponding scaling to obtain output data;

[0093] The data padding unit 304 is configured to perform data padding on the output data to obtain model training data.

[0094] The model training unit 305 is configured to input the model training data into a deep learning model to perform inference and obtain model inference data.

[0095] The data de-padding unit 306 is configured to remove data padding from the model inference data to obtain model de-padding data.

[0096] The data output unit 307 is configured to judge whether the original input image corresponding to the model de-padding data has been subjected to a block operation, if not, perform isometric scaling restoration calculation on the model de-padding data to output image restoration data, if yes, perform block splicing on the model de-padding data and then perform isometric scaling restoration calculation to output image restoration data.

[0097] In the embodiment, the information acquisition unit 301 acquires image information of the original input image and display memory information of the hardware device, respectively, wherein the image information includes original image resolution and target image resolution, and the display memory information includes display memory capacity, target display memory occupation and target display memory offset; the scale adaptation unit 302 calculates a correction coefficient according to the proportional relationship between the difference between the display memory capacity and the target display memory offset and the target display memory occupation, corrects the proportional relationship between the target image resolution and the original image resolution according to the correction coefficient, and obtains scaling ratio data; the block judgment unit 303 judges whether block operation is needed according to the scaling ratio data, if not, scales the original input image according to the scaling ratio data to obtain output data; if yes, performs block operation on the original input image, recalculates scaling ratio data for each block, judges again whether block operation is needed and performs corresponding scaling to obtain output data; the data padding unit 304 performs data padding on the output data to obtain model training data; the model training unit 305 inputs the model training data into a deep learning model for inference to obtain model inference data; the data de-padding unit 306 removes data padding from the model inference data to obtain model de-padding data; and the data output unit 307 judges whether the original input image corresponding to the model de-padding data has been subjected to block operation, if not, performs equal-scale scaling restoration calculation on the model de-padding data to output image restoration data; if yes, performs block splicing on the model de-padding data, and then performs equal-scale scaling restoration calculation to output image restoration data.

[0098] In an embodiment, the scale adaptation unit 302 comprises:

[0099] The correction unit is configured to calculate the correction coefficient a according to the following formula:

[0100]

[0101] The scale unit is configured to calculate the proportional relationship b between the target image resolution and the original image resolution according to the following formula:

[0102]

[0103] The scaling unit is configured to calculate the scaling ratio data P according to the following formula:

[0104] P=a*b

[0105] Wherein, R 标 represents the target image resolution; R 原 represents the original image resolution; V 容 represents the display memory capacity; V 偏 represents the target display memory offset; and V占 This indicates the target video memory usage.

[0106] In one embodiment, the block determination unit 303 includes:

[0107] The acquisition unit is used to acquire the original horizontal pixel count and the original vertical pixel count of the original image resolution, and to acquire the block horizontal pixel count and the block vertical pixel count of the preset block size.

[0108] The segmentation unit is used to calculate the ratio of the original horizontal pixel count to the segmented horizontal pixel count and round up to obtain the horizontal segment count, and to calculate the ratio of the original vertical pixel count to the segmented vertical pixel count and round up to obtain the vertical segment count, thus obtaining the vertical segment count.

[0109] The multiplication unit is used to multiply the number of horizontal blocks by the number of vertical blocks to obtain the number of blocks.

[0110] An overlap unit is used to calculate the horizontal overlap length of two adjacent blocks based on the relationship between the number of horizontal blocks, the number of horizontal pixels in the blocks, and the original number of horizontal pixels, and to calculate the vertical overlap length of two adjacent blocks based on the relationship between the number of vertical blocks, the number of vertical pixels in the blocks, and the original number of vertical pixels.

[0111] The segmentation unit is used to segment the original input image according to the number of blocks, the horizontal overlap length, and the vertical overlap length to obtain the corresponding number of blocks.

[0112] Furthermore, the overlapping unit includes:

[0113] Horizontal units are used to calculate the horizontal overlap length between two adjacent blocks using the following formula:

[0114] L W =(N W *B W -P W ) / I W

[0115] Vertical units are used to calculate the vertical overlap length between two adjacent blocks using the following formula:

[0116] L H =(N H *B H -P H ) / I H

[0117] Among them, L W L represents the horizontal overlap length of the block; H N represents the vertical overlap length of the blocks;W denotes the number of horizontal blocks; N H denotes the number of vertical blocks; B W denotes the number of horizontal pixels of the block; B H denotes the number of vertical pixels of the block; P W denotes the number of original horizontal pixels; P H denotes the number of original vertical pixels; I W denotes the number of horizontal gaps of the block; I H denotes the number of vertical gaps of the block.

[0118] In an embodiment, the data padding unit 304 comprises:

[0119] a padding unit configured to calculate horizontal padding parameters and vertical padding parameters according to the output data;

[0120] a creating unit configured to create a padding device based on an image processing tool;

[0121] a first pixel unit configured to add a corresponding number of pixels on the left and right sides of the output data using the horizontal padding parameters according to the padding device;

[0122] a second pixel unit configured to add a corresponding number of pixels on the top and bottom sides of the output data using the vertical padding parameters according to the padding device;

[0123] a training unit configured to use the output data with the added corresponding number of pixels as the model training data.

[0124] In an embodiment, the data padding unit 306 comprises:

[0125] a boundary unit configured to find padding boundary values of the model inference data;

[0126] a comparison unit configured to perform difference comparison between the horizontal padding parameters and the vertical padding parameters and the padding boundary values to obtain padding difference data;

[0127] a de-padding unit configured to remove data padding from the model inference data according to the padding difference data to obtain the model de-padding data.

[0128] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, and will not be described here.

[0129] The embodiment of the present application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program can realize the steps provided by the above embodiment when being executed. The storage medium can include a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk and various storage medium capable of storing program codes.

[0130] The embodiment of the present application further provides a computer device, which can include a memory and a processor, the memory has a computer program stored therein, and the processor can realize the steps provided by the above embodiment when calling the computer program in the memory. Of course, the computer device can further include various network interfaces, power supplies and other components.

[0131] The embodiments in the description are described in a progressive way, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed by the embodiments, the description is relatively simple because it corresponds to the method disclosed by the embodiments. The relevant parts can be referred to the method part. It should be noted that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0132] It should be further noted that, in the present specification, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

Claims

1. A dynamic processing method of image-adaptive video memory, characterized by, The method comprises the following steps: obtain image information of the original input image and memory information of the hardware device, wherein the image information comprises original image resolution and target image resolution, and the memory information comprises memory capacity, target memory occupation and target memory offset; calculate a correction coefficient based on the proportional relationship between the difference between the memory capacity and the target memory offset and the target memory occupation, correct the proportional relationship between the target image resolution and the original image resolution based on the correction coefficient, and obtain scaling ratio data; determine whether a block operation is needed according to the scaling ratio data, if not, scale the original input image according to the scaling ratio data to obtain output data, if yes, perform a block operation on the original input image, recalculate the scaling ratio data for each block, and determine again whether a block operation is needed and perform corresponding scaling to obtain output data; perform data padding on the output data to obtain model training data; input the model training data into a deep learning model for inference to obtain model inference data; remove the data padding from the model inference data to obtain model edge-removed data; determine whether the original input image corresponding to the model edge-removed data has been subjected to a block operation, if not, perform equal-proportion scaling restoration calculation on the model edge-removed data to output image restoration data, if yes, perform block splicing on the model edge-removed data, and then perform equal-proportion scaling restoration calculation to output image restoration data; the correction coefficient a is calculated according to the following formula: the proportional relationship b between the target image resolution and the original image resolution is calculated according to the following formula: the scaling ratio data P is calculated according to the following formula: P = a * b the block operation on the original input image comprises: wherein R 标 represents the target image resolution; R 原 represents the original image resolution; V 容 represents the video memory capacity; V 偏 represents the target video memory offset; V 占 represents the target video memory occupancy.

2. The dynamic processing method of image-adaptive video memory according to claim 1, wherein, obtain the original horizontal pixel number and the original vertical pixel number of the original image resolution, and obtain the block horizontal pixel number and the block vertical pixel number of the preset block size; calculate the proportion of the original horizontal pixel number and the block horizontal pixel number and take the integer part to obtain the horizontal block number, and calculate the proportion of the original vertical pixel number and the block vertical pixel number and take the integer part to obtain the vertical block number, to obtain the vertical block number; multiply the horizontal block number and the vertical block number to obtain the block number. the block operation on the original input image further comprises:

3. The dynamic processing method of image-adaptive video memory according to claim 2, wherein, calculate the horizontal overlap length of the adjacent two blocks according to the relationship among the horizontal block number, the block horizontal pixel number and the original horizontal pixel number, and calculate the vertical overlap length of the adjacent two blocks according to the relationship among the vertical block number, the block vertical pixel number and the original vertical pixel number; ​ According to the number of blocks, the horizontal overlap length, and the vertical overlap length, the original input image is segmented to obtain a corresponding number of blocks.

4. The dynamic processing method of image-adaptive video memory according to claim 3, characterized in that, The calculation of the horizontal overlap length of the two adjacent blocks according to the relationship of the number of horizontal blocks, the number of horizontal pixels of a block, and the number of horizontal pixels of the original image, and the calculation of the vertical overlap length of the two adjacent blocks according to the relationship of the number of vertical blocks, the number of vertical pixels of a block, and the number of vertical pixels of the original image, include: The horizontal overlap length of the two adjacent blocks is calculated according to the following formula: L W = (N W * B W - P W ) / I W The vertical overlap length of the two adjacent blocks is calculated according to the following formula: L H = (N H * B H - P H ) / I H wherein, L W represents the horizontal overlap length of the tile; L H represents the vertical overlap length of the tile; N W represents the number of horizontal tiles; N H represents the number of vertical tiles; B W represents the number of horizontal pixels of the tile; B H represents the number of vertical pixels of the tile; P W represents the number of original horizontal pixels; P H represents the number of original vertical pixels; I W represents the number of horizontal gaps of the tile; I H represents the number of vertical gaps of the tile.

5. The dynamic processing method of image-adaptive video memory according to claim 1, wherein, The output data is data padded to obtain model training data, including: Horizontal and vertical padding parameters are calculated according to the output data; A padding tool is created based on an image processing tool; According to the horizontal padding parameters, a corresponding number of pixel points are added to the left and right sides of the output data by using the padding tool; According to the vertical padding parameters, a corresponding number of pixel points are added to the top and bottom sides of the output data by using the padding tool; The output data with the added corresponding number of pixel points is used as the model training data.

6. The dynamic processing method of image-adaptive video memory according to claim 5, wherein, The model inference data is removed from the data padding to obtain model edge data, including: The padding boundary value of the model inference data is found; The padding difference data is obtained by difference comparison between the horizontal and vertical padding parameters and the padding boundary value; The model edge data is obtained by removing the data padding from the model inference data according to the padding difference data.

7. A dynamic processing apparatus for image-adaptive video memory, characterized by It includes: An information acquisition unit is configured to acquire image information of an original input image and memory information of a hardware device, respectively, wherein the image information includes an original image resolution and a target image resolution, and the memory information includes a memory capacity, a target memory occupation, and a target memory offset; A proportion adaptation unit is configured to calculate a correction coefficient based on a proportional relationship between a difference between the memory capacity and the target memory offset and the target memory occupation, and correct a proportional relationship between the target image resolution and the original image resolution based on the correction coefficient to obtain scaling proportion data; A block judgment unit is configured to determine whether a block operation is needed based on the scaling proportion data, and if not, scale the original input image based on the scaling proportion data to obtain output data, and if so, perform a block operation on the original input image, recalculate the scaling proportion data for each block, and determine again whether a block operation is needed and perform corresponding scaling to obtain output data; A data padding unit is configured to pad the output data to obtain model training data; A model training unit is configured to input the model training data into a deep learning model for inference to obtain model inference data; A data edge removal unit is configured to remove the data padding from the model inference data to obtain model edge data. The data output unit is configured to determine whether the original input image corresponding to the model edge-removed data has been subjected to a block operation, and if not, perform equal-scale scaling recovery calculation on the model edge-removed data to output image recovery data; if yes, perform block splicing on the model edge-removed data, and then perform equal-scale scaling recovery calculation to output image recovery data. The proportion adaptation unit is specifically configured to: The correction coefficient a is calculated according to the following formula: The proportion b between the target image resolution and the original image resolution is calculated according to the following formula: The scaling proportion data P is calculated according to the following formula: P = a * b wherein R 标 represents the target image resolution; R 原 represents the original image resolution; V 容 represents the video memory capacity; V 偏 represents the target video memory offset; V 占 represents the target video memory occupancy.

8. A computer device, comprising: The computer readable storage medium has a computer program stored thereon, and the computer program is executed by the processor to implement the dynamic processing method of the image adaptive video memory according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium has a computer program stored thereon, and the computer program is executed by the processor to implement the dynamic processing method of the image adaptive video memory according to any one of claims 1 to 6.

Images