Image processing method

Abstract

The invention discloses an image processing method, and the method comprises the steps: obtaining a target image in a video sequence; dividing the target image into at least two regions based on the feature information of the target image; performing super-division processing on the at least two areas by utilizing a super-division processing model corresponding to each area to obtain at least two areas subjected to super-division processing; and splicing the at least two areas subjected to the super-division processing to obtain a target image subjected to the super-division processing.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and relate to but are not limited to an image processing method. Background Art

[0002] The current super-resolution of games using computer vision (CV) methods has poor results, with serious degradation phenomena such as jagged edges and blur, which directly affects the user experience; the mobile terminal uses artificial intelligence (AI) methods, most of which currently use video super-resolution, and the models are relatively large, with serious response delays, making it difficult to meet the requirements of instant game interaction. How to perform super-resolution processing on images to achieve the desired effect for users has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, an embodiment of the present application provides an image processing method.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Acquire a target image in a video sequence;

[0007] Dividing the target image into at least two regions based on feature information of the target image;

[0008] Performing super-resolution processing on at least two regions using the super-resolution processing model corresponding to each region to obtain at least two regions that have completed the super-resolution processing;

[0009] At least two regions that have completed the super-resolution processing are spliced ​​to obtain a target image that has completed the super-resolution processing.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0011] A first acquisition module is used to acquire a target image in a video sequence;

[0012] A first division module, used for dividing the target image into at least two areas based on feature information of the target image;

[0013] A super-resolution processing module, used to perform super-resolution processing on at least two regions using a super-resolution processing model corresponding to each region, to obtain at least two regions that have completed the super-resolution processing;

[0014] The splicing module is used to splice at least two regions that have completed the super-resolution processing to obtain a target image that has completed the super-resolution processing.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the above method is implemented when the processor executes the program.

[0016] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for implementing the above method when executed by a processor.

[0017] In a fifth aspect, a computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps in the above method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0019] Figure 2A A schematic diagram of an implementation process of dividing a target image provided in an embodiment of the present application;

[0020] Figure 2B A schematic diagram of a super-resolution process implementation provided in an embodiment of the present application;

[0021] Figure 3A A schematic diagram of a process for implementing division and processing of sub-regions provided in an embodiment of the present application;

[0022] Figure 3B A schematic diagram of a process for implementing division and processing of sub-regions provided in an embodiment of the present application;

[0023] Figure 3C A schematic diagram of the implementation process of dividing and processing sub-regions is provided for an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a flow chart of a lightweight adaptive hybrid game super-resolution algorithm provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0026] Figure 6 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0028] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0031] The present application embodiment provides an image processing method, such as Figure 1 As shown, the method includes:

[0032] Step S110, acquiring a target image in a video sequence;

[0033] Here, the video sequence may be a video sequence to be super-resolution processed, for example, a game video sequence to be super-resolution processed. In image processing, super-resolution processing is a technology that can reconstruct a high-resolution (HR) image from a low-resolution (LR) image. By using the information and prior knowledge in the image, the lost details in the image are inferred and restored, thereby increasing the clarity and detail level of the image.

[0034] During the implementation process, a frame of target image in the video sequence to be super-resolved can be obtained.

[0035] Step S120: dividing the target image into at least two areas based on feature information of the target image;

[0036] During the implementation process, a single-layer convolution may be performed on the target image, and feature information of the target image may be determined based on the convolution data; and the target image may be divided into at least two regions based on the feature information. Here, the feature information may be a human visual sensitivity feature. The human visual sensitivity feature includes at least spectral sensitivity, brightness perception, resolution, and other features.

[0037] Step S130, performing super-resolution processing on the at least two regions using the super-resolution processing model corresponding to each region, to obtain at least two regions that have completed the super-resolution processing;

[0038] During the implementation process, the feature information of different regions can be matched with a super-resolution processing model for the region; then, the super-resolution processing model corresponding to each region can be used to perform super-resolution processing on at least two regions to obtain at least two regions that have completed super-resolution processing. For example, the CV model can be used to perform super-resolution processing on the background region, and the AI ​​model can be used to perform super-resolution processing on the key areas that are sensitive to the human eye.

[0039] Step S140: splicing the at least two regions that have completed the super-resolution processing to obtain a target image that has completed the super-resolution processing.

[0040] In the implementation process, at least two regions that have completed the super-resolution processing can be spliced ​​first, and then the boundaries of the adjacent regions that have completed the splicing can be fused to obtain a target image that has completed the super-resolution processing. Here, the image fusion process can integrate the information in two or more regions into one image to improve the quality, information volume and usability of the image. The fusion result can utilize the correlation in time and space and the complementarity of information of multiple regions, so that the fused image has a more comprehensive and clear description of the scene.

[0041] In an embodiment of the present application, a target image in a video sequence is first acquired; then, based on the feature information of the target image, the target image is divided into at least two regions; super-resolution processing is performed on at least two regions using a super-resolution processing model corresponding to each region to obtain at least two regions that have completed the super-resolution processing; finally, at least two regions that have completed the super-resolution processing are spliced ​​to obtain a target image that has completed the super-resolution processing. In this way, a target image is divided into multiple regions based on feature information, and super-resolution processing is performed according to different super-resolution processing models corresponding to each region, thereby effectively improving image processing efficiency. Based on feature information, models with different computational loads or processing effects are allocated according to different processing requirements. On the premise of obtaining an image quality that meets the super-resolution requirements, the amount of computation is reduced, and power consumption and latency are significantly reduced while image quality remains unchanged.

[0042] In some embodiments, the above step S120 "dividing the target image into at least two regions based on the feature information of the target image" is as follows: Figure 2A As shown, this can be achieved by following the steps below:

[0043] Step S210, obtaining high-frequency feature values ​​in the characteristic information;

[0044] Here, high-frequency features are reflected in places where the image signal intensity (brightness / grayscale) changes dramatically, that is, edges (contours) and details. The high-frequency features of an image can reflect the rapid changes in local areas of the image and are an important part of the image.

[0045] In some embodiments, the high-frequency features may include texture features. For example, more or less luminance texture, more or less chrominance texture, the discrete degree of luminance texture, and the discrete degree of chrominance texture are all high-frequency features.

[0046] During the implementation process, a single-layer convolution operation can be performed on the target image, and the high-frequency features can be obtained using the convolved data.

[0047] Step S220: dividing the target image into a first area and a second area based on a preset high-frequency feature threshold and a high-frequency feature value of the target image, wherein the high-frequency feature value of the first area is less than the high-frequency feature threshold, and the high-frequency feature value of the second area is greater than or equal to the high-frequency feature threshold.

[0048] Here, a preset high-frequency feature threshold TH1 may be set according to actual needs, and a region with a high-frequency feature value less than TH1 is divided into a first region, and a region with a high-frequency feature value greater than or equal to TH1 is divided into a second region.

[0049] In some embodiments, the preset high-frequency characteristic threshold value can be configured based on at least one of the following: power consumption, remaining power, and resource information of the electronic device. In this way, the preset high-frequency characteristic threshold value can achieve the effect of reducing power consumption and saving electric energy based on the above information configuration, so as to effectively ensure the battery life of the electronic device when the remaining power is low or the power consumption is high.

[0050] In the embodiment of the present application, the high-frequency feature value in the characteristic information is first obtained; then the target image is divided into a first area and a second area based on a preset high-frequency feature threshold and the high-frequency feature value of the target image. In this way, the target image can be divided into two areas corresponding to different high-frequency feature values ​​by using the high-frequency feature value.

[0051] In some embodiments, the above step S130 "using the super-resolution processing model corresponding to each region to perform super-resolution processing on the at least two regions to obtain at least two regions that have completed super-resolution processing" is as follows: Figure 2B As shown, this can be achieved by following the steps below:

[0052] Step S230, using a first image processing model to perform super-resolution processing on the first region to obtain a processed first region;

[0053] Here, the first image processing model may be a CV model, wherein the CV model includes a residual network (ResNet), a visual geometry group (Visual Geometry Group, VGG) network, an EfficientNet, a MobileNet, and other models.

[0054] During the implementation process, the CV model may be used to perform super-resolution processing on the first region to obtain the first region after the super-resolution processing.

[0055] Step S240: Use a second image processing model to perform super-resolution processing on the second region to obtain a processed second region, wherein the second image processing model is different from the first image processing model.

[0056] Here, the second image processing model may be an AI model, wherein the AI ​​model includes a Transformer model, a Deep Generative Model, a Generative Adversarial Network (GAN) model, and the like.

[0057] During the implementation process, the AI ​​model can be used to perform super-resolution processing on the second area to obtain a second area that has completed the super-resolution processing.

[0058] In some embodiments, since the high-frequency eigenvalue of the second region is greater than the high-frequency eigenvalue of the first region, a simpler super-resolution processing model with general super-resolution processing capability, such as a CV model, can be selected to perform super-resolution processing on the first region; a more complex super-resolution processing model with better super-resolution processing capability, such as an AI model, can be selected to perform super-resolution processing on the second region, and there is no limitation on the specific super-resolution processing model.

[0059] In the embodiment of the present application, the first image processing model is first used to perform super-resolution processing on the first region to obtain the processed first region; then the second image processing model is used to perform super-resolution processing on the second region to obtain the processed second region. In this way, different models can be used to process different regions in the target image.

[0060] In some embodiments, before the above step S230 of "using the first image processing model to perform super-resolution processing on the first region to obtain the processed first region", the first image processing model and the second image processing model may also be determined by the following steps:

[0061] Step S250: determine the first image processing model and the second image processing model based on the image optimization contribution parameters of each image processing model, so that the second image quality parameters of the processed second area are better than the first image quality parameters of the processed first area.

[0062] Here, the image optimization contribution parameters of the image processing model play a role in improving image quality, enhancing image features, and improving processing efficiency. In the implementation process, the contribution of the image processing model to optimizing image parameters when processing an image can be determined based on the test of the image processing model or the properties of the image processing model obtained in advance. For example, the image optimization contribution parameter can characterize the improvement rate of the image processing model on clarity, the improvement rate of the image processing model on chroma, and the improvement rate of the image processing model on saturation.

[0063] During implementation, the first image processing model and the second image processing model can be determined based on the image optimization contribution parameters of the image processing model. Since the high-frequency feature value of the first region is smaller than the high-frequency feature value of the second region, that is, the human eye is more sensitive to the second region, the image processing capability of the second image model matched to the second region is better than the image processing capability of the first image model matched to the first region. In this way, it can be achieved that the second image optimization parameters of the second region after processing are better than the first image optimization parameters of the first region after processing.

[0064] In the embodiment of the present application, the first image processing model and the second image processing model are determined based on the image optimization contribution parameters of each image processing model. In this way, the second image optimization parameters of the processed second area can be better than the first image optimization parameters of the processed first area.

[0065] In some embodiments, the present application provides a method for dividing and processing sub-regions, such as Figure 3A As shown, this can be achieved by following the steps below:

[0066] Step S310: performing convolution residual extraction on at least one target area, so as to divide the target area into at least two sub-areas based on the convolution residual extraction result;

[0067] Here, convolution residual extraction is an image processing method in deep learning. It combines the advantages of convolutional neural network (CNN) and residual network (ResNet), and solves the gradient vanishing and gradient exploding problems in deep neural network training through residual connection, thereby improving the performance of the model.

[0068] In the implementation process, a target area can be processed by using a single or multiple convolution residual extraction, and the target area can be divided into at least two sub-areas based on the convolution residual extraction result. Corresponds to The area processed by the AI ​​model is divided into at least two sub-areas.

[0069] Step S320: Match a corresponding sub-super-resolution processing model for each sub-region from the target super-resolution processing model corresponding to the target region, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models.

[0070] Here, the model type can be assigned to the first area and the second area by high-frequency feature thresholds, for example, the first area uses a CV model and the second area uses an AI model; in the device, there may be multiple CV specific models and multiple AI specific models, and here, the specific CV model is selected by optimizing the image contribution parameters, such as the residual network (ResNet), the visual geometry group (Visual Geometry Group, VGG) network, EfficientNet, MobileNet and other models. AI models include Transformer models, Deep Generative Models, Generative Adversarial Network models (Generative Adversarial Network, GAN) and other models.

[0071] In some embodiments, models with different parameters in the same type of CV model can also be selected by image optimization contribution parameters, such as networks with different weight parameters and / or different numbers of layers in a residual network (ResNet).

[0072] For example, the AI ​​method analyzes the complexity of features after shallow convolution and uses different network models for different regions, including Transformer models, Deep Generative Models, Generative Adversarial Networks (GANs), etc. In some embodiments, models with different parameters in the same type of AI model can also be selected by optimizing the contribution parameters of the image, such as networks with different weight parameters and / or different numbers of layers in the residual network Transformer model.

[0073] The more complex the feature distribution is, the larger the statistic of the feature part is, and the larger the statistic is, the more complex the corresponding model is. Here, the complex model may have more layers and more weight parameters than the simple model.

[0074] For example, the processing model a of the sub-region with a high frequency value greater than the threshold TH2 is more complex, and the amount of calculation is larger. The processing model b of the sub-region with a high frequency value less than the threshold TH2 and greater than TH3 is slightly simpler than model a, and the amount of calculation of model b is also less than that of model a, ..., model n has the smallest complexity and the smallest calculation.

[0075] In the embodiment of the present application, at least one target area is first subjected to convolution residual extraction, so as to divide the target area into at least two sub-areas based on the convolution residual extraction result; then, a corresponding sub-super-resolution processing model is matched for each sub-area from the target super-resolution processing model corresponding to the target area. In this way, further division of the target area can be achieved, and then the corresponding sub-super-resolution processing models are matched for different sub-areas, which effectively improves the image processing efficiency, avoids the need for all pixels in the target area to participate in complex network calculations, reduces the amount of calculations exponentially, and significantly reduces power consumption and delay while maintaining image quality.

[0076] In some embodiments, the above step S310 "performing convolution residual extraction on at least one target area to divide the target area into at least two sub-areas based on the convolution residual extraction result" can be implemented by the following steps:

[0077] Step 311: performing convolution residual extraction on the at least one target area to obtain a feature proportion distribution of the target area;

[0078] Here, Feature Proportion Distribution refers to the distribution ratio of each feature value or feature category in the data set.

[0079] During the implementation process, convolution residual extraction can be performed on at least one target area to obtain the feature proportion distribution of the target area. The target area can be further divided based on the feature proportion distribution and a suitable network model can be matched to the divided sub-areas.

[0080] For example, when the feature is a brightness feature, the distribution ratio of each brightness feature value in the brightness feature data set can be determined. The range of image brightness values ​​is 0 to 255, so the distribution of brightness values ​​in the image, for example, 0 to 255, can be determined. The target area is divided based on the distribution of brightness values ​​from 0 to 255, and a suitable network model is matched to the sub-areas obtained by the division.

[0081] Step 312: Divide the target area into the at least two sub-areas based on the feature proportion distribution.

[0082] During implementation, the target area can be divided based on the feature proportions. For example, pixels concentrated on a certain feature value can be divided into a sub-area, or pixels of the same feature category can be divided into a sub-area.

[0083] In some embodiments, the target area may also be an area determined to use a CV model, and a single or multiple convolution residual extraction processes may be used to process the area that has been divided into the area processed by the CV model to obtain the feature proportion distribution of the area, and then the area processed by the CV model is divided into at least two or more sub-areas based on the feature proportion distribution. CV models include residual networks (ResNet), visual geometry group (Visual Geometry Group, VGG) networks, EfficientNet, MobileNet and other models.

[0084] In the embodiment of the present application, at least one target area is firstly subjected to convolution residual extraction to obtain the feature proportion distribution of the target area; then the target area is divided into at least two or more sub-areas based on the feature proportion distribution. In this way, the target area can be divided into at least two or more sub-areas using the feature proportion distribution of the target area.

[0085] The present application embodiment provides a method for dividing and processing sub-regions, such as Figure 3B As shown, the following steps are included:

[0086] Step S330: dividing at least one target area based on the characteristic information of the target category, so as to divide the target area into at least two sub-areas;

[0087] Here, the target category can be time domain or space domain. Based on the space domain attributes, the strength, discreteness, structural integrity, and feature richness of the feature can be examined; based on the time domain attributes, the number of moving features, the intensity of the movement, the disappearance of feature points, the addition of new feature points, translational movement, curvilinear movement, etc. can be examined.

[0088] In the implementation process, the target area may be divided based on the temporal characteristics of the time domain or the spatial characteristics of the spatial domain. For example, the temporal characteristics may be obtained, and the target area may be divided into a plurality of sub-areas based on the temporal characteristics.

[0089] Step S340, matching a corresponding sub-super-resolution processing model for each sub-region based on the target category from the target super-resolution processing model corresponding to the target region, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models;

[0090] During implementation, in the target super-resolution processing model corresponding to the acquired target area, a corresponding sub-super-resolution processing model may be matched for each sub-area based on the spatial domain features or the temporal domain features. For example, in the AI ​​processing model corresponding to the target area, a corresponding AI processing sub-model may be matched for each sub-area based on the spatial domain features or the temporal domain features.

[0091] Correspondingly, step S130 "performing super-resolution processing on the at least two regions using the super-resolution processing model corresponding to each region to obtain at least two regions that have completed the super-resolution processing" can be implemented by the following steps:

[0092] Step 131, using the sub-super-resolution processing model matched with each sub-region to process each sub-region respectively, to obtain each sub-region that has been processed;

[0093] For example, each sub-region may be processed respectively using an AI processing sub-model matched to each sub-region.

[0094] Step 132: splice each sub-region that has been processed to obtain a target region that has completed super-resolution processing.

[0095] In the embodiment of the present application, at least one target region is first divided based on the characteristic information of the target category to divide the target region into at least two sub-regions; then, from the target super-resolution processing model corresponding to the target region, a corresponding sub-super-resolution processing model is matched for each sub-region based on the target category; finally, each sub-region is processed respectively using the sub-super-resolution processing model matched with each sub-region to obtain each sub-region that has been processed; each sub-region that has been processed is spliced ​​to obtain the target region that has been super-resolution processed. In this way, it is possible to divide the target region based on the characteristic information of the target category, and then process each sub-region based on the target category to obtain each sub-region that has been divided and the corresponding sub-super-resolution processing model is matched to obtain each sub-region, and each sub-region that has been processed is spliced ​​to obtain the target region that has been super-resolution processed.

[0096] The target category includes multiple categories. The embodiment of the present application provides a method for dividing and processing sub-areas, such as Figure 3C As shown, this can be achieved by following the steps below:

[0097] Step S350, continuously dividing the target area after super-resolution processing for multiple times based on other categories except the target category;

[0098] Here, the target category and other categories include at least one of the following categories: color feature, texture feature, shape feature, spatial domain feature, temporal domain feature, etc.

[0099] During the implementation process, we can first determine the target category as the spatial feature, divide the target area based on the spatial feature, and match the sub-regions obtained by the sub-super-resolution processing model to obtain the target area processed by the spatial feature; then divide the target area processed by the spatial feature based on the color feature, and match the sub-regions obtained by the sub-super-resolution processing model, and so on, and continue to divide the target area based on multiple categories.

[0100] Step S360, matching the corresponding sub-super-resolution processing model for each sub-region obtained for each division based on the other categories, processing each sub-region multiple times based on the sub-super-resolution processing model, and splicing each sub-region that has been processed to obtain a target region that has completed multiple super-resolution processes.

[0101] Here, multiple divisions can be made from the perspective of feature categories such as color features, texture features, shape features, spatial features, and time domain features. For example, sub-areas can be divided according to spatial features, and after matching model processing, they can be re-divided according to time domain features and matched model processing. In this way, the same position in the target image may be processed by multiple mixed models, and finally the results are spliced ​​and fused.

[0102] In the embodiment of the present application, the target area that has completed the super-resolution processing is first divided multiple times based on other categories other than the target category; then, a corresponding sub-super-resolution processing model is matched for each sub-area obtained by each division based on other categories. In this way, each sub-area can be processed multiple times using sub-super-resolution processing models matched with different categories, and each sub-area that has completed the processing can be spliced ​​to obtain the target area that has completed multiple super-resolution processing.

[0103] The present application embodiment provides a method for dividing and processing sub-regions, such as Figure 3B As shown, this can be achieved by following the steps below:

[0104] Step S370: dividing at least one target area based on the characteristic information of the target motion, so as to divide the target area into at least two sub-areas;

[0105] Here, the motion feature information refers to the position change of the pixel points of the two frames of images in the video. In the implementation process, the motion information can be provided by the game manufacturer or obtained based on adjacent frames in the video.

[0106] During implementation, the target area may be divided based on the acquired characteristic information of the target motion.

[0107] Step S380, matching a corresponding sub-super-resolution processing model for each sub-region based on the target motion from the target super-resolution processing model corresponding to the target region, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models.

[0108] In the implementation process, a sub-super-resolution processing model can be matched for each sub-region based on the target motion. For example, the sub-super-resolution processing model can be determined based on different speed, direction, shape and other parameters of the motion. The sub-super-resolution processing model is then used to process the corresponding sub-region.

[0109] In the embodiment of the present application, at least one target region is first divided based on the characteristic information of the target motion to divide the target region into at least two sub-regions; then, from the target super-resolution processing model corresponding to the target region, a corresponding sub-super-resolution processing model is matched for each sub-region based on the target motion. In this way, it is possible to divide the target region based on the characteristic information of the motion, and then match the corresponding sub-super-resolution processing model based on the characteristic information of the motion to obtain each sub-region.

[0110] In some embodiments, the above image processing method further includes the following steps:

[0111] Step S150, determining, based on the motion information of each of the regions, that the target region has the same data as the region at the same position in the previous frame of image of the target region;

[0112] In the implementation process, based on the position change of the pixel points of the two frames of the video, that is, the motion information, it can be determined that the target area has the same data as the same position area of ​​the previous frame of the target area, that is, the data in the target area of ​​the two frames is completely the same and is in a static state. For example, the background area that does not change between the two frames can be determined as the target area.

[0113] Step S160: multiplex the processed previous frame image in the same position area data as processed target area data.

[0114] During the implementation process, the changes in the scene content of the previous and next frames can be judged, and a combination of replacement and processing can be used. In completely static or smooth areas, that is, when the content of the previous and next frames has not changed at all, the content of the previous frame super-resolution result is directly used to fill the corresponding position of the current frame without performing super-resolution operations or directly sharing the weight of the previous frame. Here, the weight can refer to the model weight used for the super-resolution of the previous frame, that is, a series of parameters trained for super-resolution.

[0115] In the embodiment of the present application, the target area is determined to be the same as the same position area data of the previous frame image of the target area based on the motion information of each area; then the previous frame image that has been processed is reused as the target area data that has been processed. In this way, when the target area is determined to be a static area or a smooth area, the target area data that has been processed in the previous frame is reused, which can effectively improve the processing efficiency without affecting the processing effect.

[0116] Figure 4 A flowchart of a lightweight adaptive hybrid game super-resolution algorithm provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the algorithm can be implemented by the following steps:

[0117] Step 1: Use CV model and AI model to process respectively;

[0118] During the implementation process, firstly, the low-resolution image and the corresponding high-resolution image as well as the corresponding depth information and motion information can be rendered in the game. In the game, the depth information refers to the distance between the object and the camera in the image and the level of image distribution when the acquisition device acquires the image; the motion information refers to the position change of the pixel points between the previous and next two frames; the depth information in the game can be provided by the game manufacturer for rendering. The motion information is provided by the game manufacturer (which can be obtained by calculation in the video) for inter-frame processing.

[0119] Then input the game image for single-layer convolution, and use the data after convolution to analyze high-frequency information (texture features), such as the richness and discreteness of brightness and chroma (not limited to these two attributes), set the threshold TH1, and divide different areas. Among them, the two attributes are examples of high-frequency information, such as more or less brightness texture, more or less chroma texture, the discreteness of brightness texture, and the discreteness of chroma texture. Other attributes can be selected as needed, as long as they can distinguish high-frequency information (texture features), such as the structural attributes of image texture, frequency domain distribution, etc.; any method that can distinguish the distribution of texture features is included in this method.

[0120] Finally, according to the analysis data, the contribution of the image quality to the result of super-resolution processing is analyzed. For example, the visual resolution is used as the basis for judgment, and the appropriate game super-resolution method is adaptively selected. For example, if the feature is less than TH1, the CV model is used, and if the feature is greater than or equal to TH1, the AI ​​model is used. This method is not only applicable to game super-resolution, but also to video super-resolution. Here, the contribution is the degree of influence on the image quality, which can also be understood as the degree to which different processing methods can improve the image quality. For example, after different processing methods, the visual distinguishability (clarity) of the final image will also be different. Different processing methods have different clarity (brightness, chroma), and the better the image quality, the greater the contribution. Features can represent the distribution of image content; contribution can refer to the degree of distinguishability of image texture.

[0121] Step 2: AI model hybrid processing;

[0122] During the implementation process, the AI ​​method analyzes the complexity of features after shallow convolution and adopts different network models for different areas. For example, it can judge whether it is suitable to be divided into different network models according to the distribution of feature proportions (TH2, TH3, ... THn) extracted from single-layer or multi-layer convolution residuals. It can also be divided continuously according to different mode attributes of features (not limited to mode attributes) after these models, thereby forming a mixed processing of multiple models. Among them, feature proportion distribution refers to the distribution ratio of each feature value or feature category in the data set. For example, when the feature is a brightness feature, the distribution ratio of each brightness feature value in the brightness feature data set can be determined. The range of image brightness values ​​is 0 to 255, so the distribution of brightness values ​​in the image, for example, 0 to 255, can be determined, and the network model corresponding to the division is determined based on the distribution of brightness values ​​from 0 to 255.

[0123] Here, the feature is a value obtained by calculation, a statistic that measures the distribution of features in a certain area, and is used to compare with a series of thresholds; here, the complex model may have more layers and more weight parameters than the simple model. The more complex the feature distribution, the larger the statistic of the feature part. The larger the statistic. The processing model b for the sub-region with a high-frequency value less than the threshold TH2 and greater than TH3 is slightly simpler than model a, and the calculation amount of model b is also less than that of model a, ..., and the complexity of model n is the smallest and the calculation amount is also the smallest.

[0124] like Figure 4 As shown, after models a to n process the image, multiple or multiple types of AI model results are spliced ​​and fused, including weight sharing, compression and other processing.

[0125] In some embodiments, a method combining replacement and processing can be used to determine changes in scene content between previous and next frames. That is, in a completely static or smooth area, where the content of the previous and next frames has not changed at all, the previous frame super-resolution result content is directly used to fill the corresponding position of the current frame without performing super-resolution operations or directly sharing the previous frame weight.

[0126] In some embodiments, different threshold parameters can be adjusted according to the power situation of the electronic device to achieve the control of the super-resolution method and thus control power consumption. For example, TH1, TH2, TH3, ..., THn are different thresholds, and the thresholds themselves are set in advance; according to the size of the power of the electronic device, different coefficients w are taken. When the power is less than a certain value (in order to ensure the battery life), the coefficient begins to increase gradually (that is, the value of w gradually increases, and the coefficient w is used to adjust the threshold size: TH1*w, TH2*w, ..., THn*w; the larger the threshold, the smaller the area that can be divided and processed using a more complex model, which can achieve a power saving effect. For example, the larger the TH1 value, the smaller the area that can be divided and processed using the AI ​​model for the same image, and the larger the area that can be processed using the CV model, which can achieve a power saving effect.

[0127] Step 3: The sub-images are fused and output.

[0128] During the implementation process, after using the CV model and the AI ​​model to perform super-resolution processing on different areas, the obtained sub-super-resolution images can be spliced ​​and fused to output the results.

[0129] In the embodiments of the present application, combined with the visual sensitivity characteristics of the human eye, the scene content is statistically analyzed according to different attributes. The method is not limited to a certain method or a certain type of method. Any method that can complete the statistics of the scene content attributes can be implemented. Different complexity methods are adaptively adopted according to the statistical results, such as CV and AI network models. The network model also adopts different network models according to the different attributes of the calculation for super-resolution processing; the corresponding algorithm is used according to the complexity of the scene content: the feature information that the human eye is sensitive to is processed, that is, the model with corresponding calculation amount is adopted according to the complexity of the feature, the receptive field is increased in the area with complex features, and the residual layer is deepened to extract more advanced features. The receptive field is reduced while the number of residual layers is reduced in the area with simple features. Different levels or types of network models are divided according to the different feature complexities; the game information is reasonably used to accurately process the moving part, that is, network models of different complexity are selected based on the motion of the previous and next frames and the complexity of the features; finally, the results of various models are compressed, spliced ​​and fused. In this way, this method avoids the need for all pixels in the scene to participate in complex network calculations, and the actual content of most scenes that need to be processed by the AI ​​model is reduced to about 8%. According to the final effect requirements, the threshold is adjusted to control the size of the content that the AI ​​model needs to process. The amount of calculation is reduced exponentially, and the power consumption and delay are significantly reduced while the image quality remains unchanged. The image quality and calculation amount can be automatically adjusted according to different scene requirements and the current power supply situation.

[0130] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes various modules included, each module includes various sub-modules, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0131] Figure 5 A schematic diagram of the structure of the image processing device provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the device 500 includes:

[0132] A first acquisition module 510 is used to acquire a target image in a video sequence;

[0133] A first division module 520, configured to divide the target image into at least two regions based on feature information of the target image;

[0134] A super-resolution processing module 530 is used to perform super-resolution processing on the at least two regions using the super-resolution processing model corresponding to each region to obtain at least two regions that have completed the super-resolution processing;

[0135] The stitching module 540 is used to stitch the at least two regions that have completed the super-resolution processing to obtain a target image that has completed the super-resolution processing.

[0136] In some embodiments, the first division module 520 includes a first acquisition submodule and a first division submodule, wherein the first acquisition submodule is used to obtain the high-frequency feature value in the characteristic information; the first division submodule is used to divide the target image into a first area and a second area based on a preset high-frequency feature threshold and the high-frequency feature value of the target image, wherein the high-frequency feature value of the first area is less than the high-frequency feature threshold, and the high-frequency feature value of the second area is greater than or equal to the high-frequency feature threshold.

[0137] In some embodiments, the super-resolution processing module 530 includes a first super-resolution processing sub-module and a second super-resolution processing sub-module, wherein the first super-resolution processing sub-module is used to perform super-resolution processing on the first region using a first image processing model to obtain a processed first region; the second super-resolution processing sub-module is used to perform super-resolution processing on the second region using a second image processing model to obtain a processed second region, wherein the second image processing model is different from the first image processing model.

[0138] In some embodiments, the super-resolution processing module 530 also includes a determination submodule for determining the first image processing model and the second image processing model based on the image optimization contribution parameters of each image processing model, so that the second image quality parameters of the second area after processing are better than the first image quality parameters of the first area after processing.

[0139] In some embodiments, the image processing method also includes a convolution residual extraction module and a first matching module, wherein the convolution residual extraction module is used to perform convolution residual extraction on at least one target area to divide the target area into at least two sub-areas based on the convolution residual extraction result; the first matching module is used to match the corresponding sub-super-resolution processing model for each sub-area from the target super-resolution processing model corresponding to the target area, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models.

[0140] In some embodiments, the convolution residual extraction module includes a convolution residual extraction submodule and a second division submodule, wherein the convolution residual extraction submodule is used to perform convolution residual extraction on the at least one target area to obtain the feature proportion distribution of the target area; the second division submodule is used to divide the target area into the at least two or more sub-areas based on the feature proportion distribution.

[0141] In some embodiments, the image processing module also includes a second division module and a second matching module, wherein the second division module is used to divide at least one target area based on the feature information of the target category, so as to divide the target area into at least two or more sub-areas; the second matching module is used to match the corresponding sub-super-resolution processing model for each sub-area based on the target category from the target super-resolution processing model corresponding to the target area, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models; correspondingly, the super-resolution processing module 530 includes a processing sub-module and a splicing sub-module, wherein the processing sub-module is used to use the sub-super-resolution processing model matched with each sub-region to process each sub-region respectively, so as to obtain each sub-region that has been processed; the splicing sub-module is used to splice each sub-region that has been processed, so as to obtain the target area that has been super-resolution processed.

[0142] In some embodiments, the target category includes multiple categories, and the image processing module also includes a third division module and a third matching module, wherein the third division module is used to continuously divide the target area that has completed the super-resolution processing for multiple times based on other categories except the target category; the third matching module is used to match the corresponding sub-super-resolution processing model for each sub-area obtained for each division based on the other categories, so as to process each sub-area multiple times based on the sub-super-resolution processing model, and splice each sub-area that has completed the processing to obtain the target area that has completed multiple super-resolution processing.

[0143] In some embodiments, the image processing module also includes a fourth division module and a fourth matching module, wherein the fourth division module is used to divide at least one target area based on feature information of target motion so as to divide the target area into at least two or more sub-areas; the fourth matching module is used to match a corresponding sub-super-resolution processing model for each sub-area based on the target motion from the target super-resolution processing model corresponding to the target area, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models.

[0144] In some embodiments, the image processing module also includes a determination module and a multiplexing module, wherein the determination module is used to determine that the target area is the same as the same position area data of the previous frame image of the target area based on the motion information of each of the areas; and the multiplexing module is used to multiplex the previous frame image that has completed processing at the same position area data as the target area data that has completed processing.

[0145] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0146] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0147] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the image processing method provided in the above embodiment are implemented.

[0148] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and the processor 602 implements the steps of the image processing method provided in the above embodiment when executing the program.

[0149] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0150] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0151] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0152] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0154] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0156] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0157] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0158] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0159] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0160] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0161] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, the method comprising: Acquire a target image in a video sequence; Dividing the target image into at least two areas based on feature information of the target image; Performing super-resolution processing on the at least two regions using the super-resolution processing model corresponding to each region to obtain at least two regions that have completed the super-resolution processing; The at least two regions that have completed the super-resolution processing are spliced ​​to obtain a target image that has completed the super-resolution processing.

2. The method according to claim 1, wherein dividing the target image into at least two regions based on feature information of the target image comprises: Obtaining high-frequency feature values ​​in the characteristic information; The target image is divided into a first area and a second area based on a preset high-frequency feature threshold and a high-frequency feature value of the target image, wherein the high-frequency feature value of the first area is less than the high-frequency feature threshold, and the high-frequency feature value of the second area is greater than or equal to the high-frequency feature threshold.

3. The method according to claim 2, wherein the super-resolution processing model corresponding to each region is used to perform super-resolution processing on the at least two regions to obtain at least two regions that have completed super-resolution processing, comprising: Performing super-resolution processing on the first region using a first image processing model to obtain a processed first region; The second region is super-resolution processed by using a second image processing model to obtain a processed second region, wherein the second image processing model is different from the first image processing model.

4. The method of claim 3, further comprising: Based on the image optimization contribution parameters of each image processing model, the first image processing model and the second image processing model are determined so that the second image quality parameters of the processed second area are better than the first image quality parameters of the processed first area.

5. The method of claim 1, further comprising: Performing convolution residual extraction on at least one target area to divide the target area into at least two sub-areas based on the convolution residual extraction result; From the target super-resolution processing model corresponding to the target region, a corresponding sub-super-resolution processing model is matched for each sub-region, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models.

6. The method according to claim 5, wherein the step of performing convolution residual extraction on at least one target region to divide the target region into at least two sub-regions based on the convolution residual extraction result comprises: Performing convolution residual extraction on the at least one target area to obtain a feature proportion distribution of the target area; The target area is divided into the at least two sub-areas based on the feature proportion distribution.

7. The method of claim 1, further comprising: Dividing at least one target area based on feature information of the target category to divide the target area into at least two or more sub-areas; From the target super-resolution processing model corresponding to the target region, matching a corresponding sub-super-resolution processing model for each sub-region based on the target category, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models; Correspondingly, performing super-resolution processing on the at least two regions using the super-resolution processing model corresponding to each region to obtain at least two regions that have completed the super-resolution processing includes: Using the sub-super-resolution processing model matched with each sub-region, respectively processing each sub-region, to obtain each sub-region that has been processed; Each sub-region that has been processed is spliced ​​together to obtain a target region that has completed super-resolution processing.

8. The method according to claim 7, wherein the target category comprises a plurality of categories, and the method further comprises: Continuously dividing the target area that has completed the super-resolution processing multiple times based on other categories except the target category; Based on the other categories, each sub-region obtained for each division is matched with a corresponding sub-super-resolution processing model, so that each sub-region is processed multiple times based on the sub-super-resolution processing model, and each sub-region that has been processed is spliced ​​to obtain a target region that has completed multiple super-resolution processing.

9. The method of claim 1, further comprising: Dividing at least one target area based on characteristic information of target motion, so as to divide the target area into at least two or more sub-areas; From the target super-resolution processing model corresponding to the target region, a corresponding sub-super-resolution processing model is matched for each sub-region based on the target motion, wherein the target super-resolution processing model includes more than two matchable sub-super-resolution processing models.

10. The method according to any one of claims 1 to 9, further comprising: Determine, based on the motion information of each of the regions, that the target region has the same data as the region at the same position in the previous frame of image of the target region; The processed previous frame image is multiplexed in the same position area data as the processed target area data.

Images