Scene-based super-division image processing method, program product and equipment

Abstract

The invention discloses a scene-based super-resolution image processing method, a program product and equipment. The method comprises the following steps: acquiring a current scene image; determining a current super-division image processing model matched with the current scene image; and performing super-resolution processing on the current scene image based on the current super-resolution image processing model to obtain a target scene image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a scene-based super-resolution image processing method, a computer program product, and a scene-based super-resolution image processing device. Background Art

[0002] With the development of imaging technology, imaging products are becoming more and more popular. However, some images (such as infrared images) have the characteristics of low resolution, low signal-to-noise ratio, and low detail, which leads to shortcomings in user experience. Currently, there are some methods on the market to improve image quality, which combine image processing with AI intelligence to effectively improve and enhance the quality of infrared images; super-resolution processing algorithms bring about improvements in image quality. However, the above-mentioned methods of changing image quality in the existing technology use a single model for various scenarios, which cannot achieve the best performance for each scenario, resulting in a decline in user experience. Summary of the Invention

[0003] In order to solve the existing technical problems, the present invention provides a scene-based super-resolution image processing method, a computer program product and a scene-based super-resolution image processing device, which can improve the image quality in different scenes and meet the image processing requirements in different scenes.

[0004] In a first aspect, a scene-based super-resolution image processing method is provided, comprising: acquiring a current scene image; determining a current super-resolution image processing model that matches the current scene image; and performing super-resolution processing on the current scene image based on the current super-resolution image processing model to obtain a target scene image.

[0005] In a second aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the scene-based super-resolution image processing method as described in any one of the first aspects of the present application.

[0006] In a third aspect, a scene-based super-resolution image processing device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the scene-based super-resolution image processing method as described in any one of the first aspects of the present application.

[0007] In a fourth aspect, a storage medium is provided, comprising a computer program, which stores a computer program. When the computer program is executed by a processor, the processor executes the scene-based super-resolution image processing method described in any one of the first aspects of the present application.

[0008] In the fifth aspect, the present application provides a scene-based super-resolution image processing system, which includes a super-resolution image processing device and a server that executes the scene-based super-resolution image processing method described in any one of the first aspects of the present application, and the server communicates with the super-resolution image processing device, and the server is used to train an image processing model or a super-resolution image processing model.

[0009] This application obtains the current scene image and determines the current super-resolution image processing model that matches the current scene image. The current super-resolution image processing model is a super-resolution image processing model that matches the scene type of the current scene image. Therefore, using the current super-resolution image processing model to perform super-resolution processing on the current scene image is more suitable for the scene requirements of the current scene image, improves the image quality in different scenes, and can meet the image processing requirements in different scenes, meeting the diversity of users' scene requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 FIG2 is an application environment diagram of a scene-based super-resolution image processing method in one embodiment;

[0011] Figure 2 is a flow chart of a scene-based super-resolution image processing method in one embodiment;

[0012] Figure 3 is a schematic diagram of selecting scene information based on a user interface in one embodiment;

[0013] Figure 4 is a schematic diagram of an image after super-resolution image processing in a jungle scene in one embodiment;

[0014] Figure 5 is a schematic diagram of current scene images collected in a jungle scene and a bird watching scene in one embodiment;

[0015] Figure 6 1 is a schematic diagram of a process for training an image processing model in a scene-based super-resolution image processing method according to an embodiment;

[0016] Figure 7 A schematic diagram of a network structure of an image processing model in one embodiment;

[0017] Figure 8 is a flowchart of a scene-based super-resolution image processing method in another embodiment;

[0018] Figure 9 is a flowchart of a scene-based super-resolution image processing method in another embodiment;

[0019] Figure 10 is a schematic diagram of a scene-based super-resolution image processing device in one embodiment;

[0020] Figure 11 Schematic diagram of a scene-based super-resolution image processing device in one embodiment. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the scope of protection of the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] In the following description, reference is made to “some embodiments” which describe a subset of all possible embodiments, but it should be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0024] See Figure 1 , is an application environment diagram of the image processing device control method in one embodiment. The image processing device control method is applied to a scene-based super-resolution image processing device 10, hereinafter referred to as "image processing device 10", and the image processing device 10 includes an image acquisition device 12, a processor 13 and a memory 14. The image acquisition device 12 is used to acquire images of the scene where the image processing device 10 is located. The processor 13 calls the super-resolution image processing model that matches the scene type according to the scene type of the acquired scene image to perform super-resolution processing on the current scene image to obtain a high-resolution image. The image processing device 10 can also communicate with the server 20 to upload the acquired training sample data set to the server 20. The server 20 trains the image processing model or the super-resolution image processing model based on the data set, and then sends the parameter file of the trained model to the image processing device 10. The parameter file can be stored in the memory 14 before the image processing device 10 leaves the factory, or the parameter file can be requested from the server 20 during use. In other embodiments, it is understandable that the server 20 may not be included, and the trained image processing model or super-resolution image processing model may be trained on other computing devices and stored in the memory before the image processing device 10 leaves the factory.

[0025] The scene-based super-resolution image processing device 10 includes, but is not limited to, a handheld detection device, a non-handheld, autonomously movable detection device, and a non-handheld, non-autonomously movable detection device. Handheld detection devices include, but are not limited to, handheld imaging devices with infrared imaging capabilities and handheld imaging devices with visible light imaging capabilities. Non-handheld, autonomously movable detection devices include, but are not limited to, autonomously movable detection devices with infrared imaging capabilities and autonomously movable detection devices with visible light imaging capabilities. Non-handheld, non-autonomously movable detection devices include, but are not limited to, non-handheld and non-autonomously movable devices with infrared imaging capabilities and non-handheld and non-autonomously movable devices with visible light imaging capabilities. Using the handheld detection device and the non-handheld, autonomously movable detection device, a user can perform scene-based super-resolution image processing on a current scene in motion while in motion, or on a fixed scene. Using the non-handheld, non-autonomously movable detection device, a fixed scene can be subjected to scene-based super-resolution image processing. Therefore, the scene-based super-resolution image processing method provided in this application can be applied to various complex motion-changing scenes and fixed scenes.

[0026] The image acquisition device 12 may be a combination of one or more sensors. The image acquisition device 12 may be a monocular vision sensor or a multi-vision vision sensor. For example, the image acquisition device 12 may be a combination of one or more sensors selected from the group consisting of a thermal imaging sensor, a visible light image sensor, a millimeter wave sensor, a lidar sensor, an infrared thermal imaging sensor, and a depth sensor.

[0027] The processor 13 may be one or more processors. When there are multiple processors 13, the multiple processors may be integrated on a single chip or independently provided on each chip. The scene-based super-resolution image processing device 10 is a device installed on any type of mobile object, such as a vehicle, electric vehicle, hybrid electric vehicle, motorcycle, bicycle, personal mobile device, airplane, drone, ship, or robot. The scene-based super-resolution image processing device 10 may also be fixedly installed on a fixed device or at a fixed location in a fixed scene.

[0028] The scene-based super-resolution image processing device 10 may also include other sensor modules, including but not limited to environmental perception sensors and motion posture sensors. Environmental perception sensors include but are not limited to one or more combinations of the following sensors: brightness sensors, temperature sensors, haze sensors, and other environmental sensors. Motion posture sensors include but are not limited to one or more combinations of the following: inertial measurement units (IMUs), speed sensors, acceleration sensors, gyroscope sensors, geomagnetic sensors, rotation vector sensors, steering wheel angle sensors, level sensors, tilt sensors, vibration sensors, displacement sensors, and gravity sensors.

[0029] The scene-based super-resolution image processing device 10 may further include a display terminal for displaying images.

[0030] See also Figure 2 , is a flow chart of a scene-based super-resolution image processing method provided in an embodiment of the present application. The scene-based super-resolution image processing method is applied to a scene-based super-resolution image processing device, and the scene-based super-resolution image processing method includes the following steps:

[0031] S11. Obtain the current scene image.

[0032] In this embodiment, the current scene image is an image captured by the image processing device 10. The current scene image can be an infrared image, a visible light image, a fused image of an infrared image and a visible light image, etc. Due to the characteristics of infrared images, the input resolution is generally low. For example, if the size of the current scene image is 256*192 or 384*288, the image quality is poor (such as noise, etc.) and needs to be processed. The most widely used infrared detector currently has a resolution of 256*192. In addition, images of 640*512, 1280*1024, or 1920*1080 can also be input to further enhance the effect.

[0033] The application scenario of the image processing device 10 may not be fixed. When the scenario type changes, the strategy for performing super-resolution image processing will also be different.

[0034] S12. Determine a current super-resolution image processing model that matches the current scene image.

[0035] Existing technologies only use one super-resolution image processing model for infrared images of different scene types or image modes. However, observation requirements vary depending on the scene type. For example, in a jungle mode, due to the rich scene details, the super-resolution model only needs to increase the image's sharpness moderately, focusing on enhancing the size of infrared targets. However, in plain scenes, due to the low detail, the model suitable for the jungle mode may not adequately enhance detail. Therefore, a model with greater sharpness enhancement is required to allow users to observe a richer range of detailed information.

[0036] In this embodiment, when the application scenario of the image processing device 10 changes, the super-resolution image processing strategy executed will also change. Different scene types correspond to different super-resolution image processing models, and the processing parameters of the super-resolution image processing models corresponding to different scene models also vary. These processing parameters include, but are not limited to, sharpening intensity, contrast, image detail noise parameters, target image prominence, and so on. Therefore, during the training process, datasets for different scene types are collected and the super-resolution image processing models for the corresponding scene types are trained separately. Thus, each scene type corresponds to a super-resolution image processing model. The super-resolution image processing model for each scene type learns the super-resolution features of the super-resolution label image corresponding to the low-resolution image for that scene type, thereby better processing the super-resolution of the images collected for that scene type. For example, in jungle mode, where trees and tree trunks have more detail, the sharpening intensity of the super-resolution image processing model is reduced to avoid excessive eye strain and discomfort caused by oversharpening. In plain mode, where the user observes fewer scene details, the sharpening intensity of the super-resolution image processing model can be increased to observe more details. The target image prominence indicates the salience or prominence of the target area in the image relative to the background or other non-target areas. For example, in Jungle Mode, there are more distracting factors, making it difficult to see the target clearly. In Plain Mode, however, there are fewer distracting factors, making the target easier to see. Therefore, the target image in Jungle Mode is more prominent than in Plain Mode. Image detail noise parameters include, but are not limited to, image detail noise level. The image detail noise level includes multiple levels, such as low, medium, and high.

[0037] In some embodiments, the super-resolution image processing model can be a separate independent model, or can be a part of an overall model, for example, a sub-model in a subsequent image processing model.

[0038] S13. Based on the current super-resolution image processing model, perform super-resolution processing on the current scene image to obtain a target scene image.

[0039] In this embodiment, different modes correspond to different scene types. Since different scene types correspond to different super-resolution image processing models, the processing parameters of the super-resolution image processing models corresponding to different scene models are also different. The current super-resolution image processing model is a super-resolution image processing model that matches the scene type of the current scene image. Therefore, using the current super-resolution image processing model to perform super-resolution processing on the current scene image is more suitable for the scene requirements of the current scene image. For example, in jungle mode, due to the rich scene details, the super-resolution image processing model at this time makes a moderate increase in the sharpness of the image, focusing on the increase and enhancement of infrared targets; in plain scenes, due to the weaker scene details, the super-resolution image processing model that meets the jungle scene at this time does not enhance the details enough, and it is necessary to train a model with a larger sharpness enhancement so that the user can observe richer detail information.

[0040] In an optional implementation, the method further includes: displaying the target scene image. The target scene image is displayed on a display terminal in real time.

[0041] In the above embodiment, the current scene image is obtained, and the current super-resolution image processing model that matches the current scene image is determined. The current super-resolution image processing model is a super-resolution image processing model that matches the scene type of the current scene image. Therefore, super-resolution processing is performed on the current scene image using the current super-resolution image processing model, which is more suitable for the scene requirements of the current scene image, improves the image quality in different scenes, and can meet the image processing requirements in different scenes, meeting the diversity of user needs.

[0042] In some embodiments, determining a current super-resolution image processing model that matches the current scene image includes one or more of the following:

[0043] Acquiring current scene information, and determining a current super-resolution image processing model that matches the current scene information based on the current scene information;

[0044] Based on the current scene image, a current scene type corresponding to the current scene image is determined, and a current super-resolution image processing model matching the current scene type is acquired.

[0045] In this embodiment, scene information indicates selected information associated with a scene type, including but not limited to a selected scene mode, a selected scene type, and so on. Scene information can be input through a user interface, including but not limited to a voice interface and a user interface. The user can select different scene information through the user interface. For example, the image processing device 10 is an infrared birdwatching device with three scene modes: plain mode, jungle mode, and birdwatching mode. Plain mode provides balanced image brightness and detail, suitable for observing animals in outdoor plain scenes, for example. Jungle mode is primarily used for observation in dense forests, requiring a certain contrast between the target and the background, resulting in richer details. Birdwatching mode is primarily for observation when the device is facing the sky, where it is necessary to suppress the sky effect to avoid obscuring details such as birds. As can be seen from the above description, the three scene modes correspond to three different scene types, meeting three different user needs. After selecting different scene modes, different super-resolution image processing models will exhibit significant differences in image detail noise, contrast, and target image prominence. Therefore, datasets can be collected for each of the three different scene modes to train the super-resolution image processing models for each scene type, thereby using different super-resolution image processing models for processing different scene types.

[0046] In this embodiment, the user can input voice information through the voice interface, and use voice processing technology to extract current scene information from the voice information, and determine the current super-resolution image processing model that matches the current scene information based on the current scene information. The user can also input the associated information of the scene type through the user interface, such as configuring the scene information control on the user interface, selecting the desired scene type through the scene information control, obtaining the selection operation through the user interface, and obtaining the current scene information through the selection operation. For example Figure 3 and Figure 4 As shown, Figure 3 is a schematic diagram of selecting scene information based on a user interface in one embodiment. Figure 4 Schematic diagram of an image after super-resolution image processing in a jungle scene in one embodiment. Figure 3 The button indicated by ⑦ in the figure selects the desired scene mode. For example, if Jungle mode is currently selected, the corresponding super-resolution image processing model for Jungle mode is obtained for processing. The parameters of the trained super-resolution image processing model for Jungle mode are used to process the captured current scene image. This is more suitable for super-resolution image processing in Jungle mode. Therefore, when the user switches between different scene modes through the user interface, the super-resolution image processing model switches accordingly.

[0047] In some implementations, image analysis can be performed based on the current scene image, that is, image background features in the current scene image are extracted, and the current scene type is determined based on the image background features, thereby determining the current super-resolution image processing model that matches the current scene type.

[0048] In the above embodiment, the current scene information can be obtained, and the scene type required by the user can be determined through the current scene information, thereby determining the current super-resolution image processing model that matches the scene type. Therefore, the user can select the required scene type through the user interface, thereby using the current super-resolution image processing model to perform super-resolution processing on the current scene image, which is more suitable for the scene requirements of the current scene image, improves the image quality in different scenes, and can meet the image processing requirements in different scenes, meeting the diversity of user needs.

[0049] In some embodiments, determining a current super-resolution image processing model that matches the current scene image includes:

[0050] Acquire a trained image processing model, wherein the image processing model includes a scene classification model and multiple super-resolution image processing models;

[0051] Based on the current scene image, classify the current scene image using a scene classification model in the image processing model to obtain a current scene type;

[0052] From the multiple super-resolution image processing models, a super-resolution image processing model that matches the current scene type is determined as the current super-resolution image processing model.

[0053] Optional,

[0054] The classifying the current scene image based on the current scene image by using the scene classification model in the image processing model to obtain the current scene type includes:

[0055] Based on the current scene image, input data of a scene classification model is formed, and the current scene type and the corresponding classification confidence are outputted through the scene classification model.

[0056] In this embodiment, the image processing model is an end-to-end pre-trained model. The trained image processing model includes a scene classification model and multiple super-resolution image processing models. Each super-resolution image processing model in the trained image processing model corresponds to a scene type with the highest degree of correlation. That is, after obtaining input data for the image processing model based on the current scene image, the image processing model can not only determine the current scene type, but also select the super-resolution image processing model with the highest degree of correlation with the current scene type from multiple super-resolution image processing models as the current super-resolution image processing model, process the current scene image, and obtain an output super-resolution image. The scene classification model is equivalent to a scene classification network in the image processing model, and a super-resolution image processing model is equivalent to a super-resolution image processing network. During the training process, data sets under different scene types are collected, namely the subsequent sample data sets. Each sample in the sample data set includes a sample image, a sample scene type label corresponding to the sample image, and a super-resolution sample label image corresponding to the sample image. Therefore, during the training process, the sample scene type label is used as the training purpose, and the background features in the sample image can be learned, so that the image processing model after training can accurately determine the scene type; at the same time, the super-resolution sample label image is used as the training purpose, and the features of the high-resolution image under each scene type can be learned. In this way, in the image model after the training is completed, it is equivalent to a super-resolution image processing model corresponding to each scene type, so as to better handle the super-resolution processing of images collected under the respective scene types.

[0057] In the above embodiment, after acquiring the current scene image, the image processing model can automatically identify the current scene type in which the current scene image is located, without the user having to manually set the relevant information of the scene type. The image processing model can not only determine the current scene type, but also select the super-resolution image processing model with the highest degree of correlation with the current scene type from multiple super-resolution image processing models as the current super-resolution image processing model, process the current scene image, and obtain an output super-resolution image, which is more suitable for the scene requirements of the current scene image, improves the image quality in different scenes, and can meet the image processing requirements in different scenes and meet the diversity of user needs.

[0058] In some embodiments, the method further comprises:

[0059] Obtain target image processing parameters corresponding to the current scene type where the current scene image is located;

[0060] The current scene image is acquired and processed according to the target image processing parameters.

[0061] In this embodiment, different scene types correspond to different image processing parameters, wherein the image processing parameters include but are not limited to the detector's response rate, contrast adjustment parameters, noise suppression parameters, and the like. The response rate is a parameter of the detector (such as an infrared detector), which is related to the ability to distinguish details. In order to achieve better results, the image processing parameters under different scene types need to be tuned in a targeted manner, that is, the optimal image processing parameters corresponding to the current scene type under the current scene environment, such as infrared ISP parameters. For example, in rainy and foggy weather, the detector's detection capability is insufficient and heat sources are not easy to detect. At this time, it is necessary to increase the response rate and increase the detector's ability to distinguish faint details. At the same time, increasing the response rate will increase the noise, and based on this, a super-resolution model with a certain denoising capability, while increasing details and strengthening the target capability is trained.

[0062] For example, Figure 5 As shown, Figure 5 This is a schematic diagram of current scene images collected in a jungle scene and a bird watching scene in one embodiment. In the jungle scene, image processing parameters corresponding to the jungle scene are used, and in the bird watching scene, image processing parameters corresponding to the bird watching scene are used.

[0063] In the above embodiment, by adopting different image processing parameters in different scene types, the captured images and processed images are more in line with the scene type requirements, the image quality in different scenes is improved, and the image processing requirements in different scenes can be met to meet the diversity of user needs.

[0064] In some embodiments, the method further comprises:

[0065] Based on the add scene control in the user interface, an add operation is obtained, and identification information of the to-be-added scene and scene sample data of the to-be-added scene are obtained according to the add operation.

[0066] In this embodiment, users can also customize and add desired scenes through the user interface, i.e., custom scenes. After the user triggers the add scene control, the device receives the add operation and obtains identification information for the scene to be added, where the identification information indicates the identification of the scene to be added, including but not limited to the name and number of the scene to be added. Scene sample data for the custom scene can also be added through the add scene control. The user interface can also include parameter configuration controls, and the user can configure the image processing parameters of the custom scene by referring to the configuration controls. In an optional implementation, the image processing device can also communicate with a server and send scene sample data for the scene to be added to the server. The server trains a super-resolution image processing model for the scene to be added to obtain a trained super-resolution image processing model for the scene to be added, and / or adds the scene sample data for the scene to be added to the sample data set to retrain the image processing model so that the retrained image processing model can better address the image processing requirements for the scene to be added. The server sends the parameter file of the trained super-resolution image processing model for the scene to be added and / or the parameter file of the retrained image processing model to the image processing device. In an optional implementation, the training step can also be performed by the image processing device.

[0067] For example, you can set other scenes based on the usage environment, such as plateau scenes, desert scenes, and ocean scenes; based on the purpose, you can use driving scenes, outdoor scenes, temperature measurement scenes, etc.; based on the season, you can set winter scenes, summer scenes, etc. In addition, you can set other scenes based on the weather, such as rain and fog scenes, snow scenes, and heavy rain scenes. Other scenes can also be added according to user needs.

[0068] In the above embodiment, when the user interface does not have the scene type required by the user, the user can customize the scene type and obtain scene sample data of the customized scene type to obtain a super-resolution image processing model under the customized scene, or retrain the image processing model so that the model outputs a super-resolution image under the customized scene, thereby improving the image quality under the customized scene and meeting the user's requirements for diversified scene types.

[0069] In some embodiments, as Figure 6 As shown, Figure 6 FIG. 1 is a flow chart of training an image processing model in a scene-based super-resolution image processing method according to an embodiment, wherein the method further includes:

[0070] S61. Obtain a sample data set.

[0071] In this embodiment, each sample in the sample data set includes a sample image, a sample scene type label corresponding to the sample image, and a super-resolution sample label image corresponding to the sample image. When the image processing model is adopted, the data set is a sample data set. The super-resolution sample label image is obtained by processing the sample image using the super-resolution algorithm in the prior art. For the super-resolution algorithm, an existing, mature super-resolution algorithm can be used. The super-resolution method can adopt a traditional machine learning-based method, such as the iterative back projection method (IBP), "Image Super-Resolution via Iterative Back Projection". This method optimizes the reconstructed image in an iterative manner so that the result after downsampling is as close as possible to the original low-resolution image. FOCUSS (Fast Iterative Shrinkage-Thresholding Algorithm for Image Super-Resolution): The FOCUSS algorithm is an iterative soft threshold algorithm used to solve the inverse problem and restore the image by minimizing the error criterion function with regularization. Although traditional machine learning-based methods can improve the resolution of images to a certain extent, they usually require manual feature design and have limited ability to restore complex image structures and textures. With the development of deep learning technology, especially the successful application of convolutional neural networks (CNNs) in image processing, super-resolution methods based on deep learning have gradually become mainstream due to their automatic feature learning capabilities and superior performance. Deep learning-based blind super-resolution algorithms not only improve image resolution but also handle unknown degradations such as noise, blur, mosaics, and JPEG compression, thereby improving image quality. For example, ZSSR (Zero-Shot Super-Resolution) provides a super-resolution method that does not require pre-training and can be directly self-adjusted to a single image during test time. It utilizes image content to optimize a small convolutional network, which is suitable for blind super-resolution scenarios because the network can be adjusted for each specific degradation instance. Alternatively, the method proposed in the paper "Learning the Degradation Distribution for Blind Image Super-Resolution" (CVPR 2022) was adopted. This paper proposed a probabilistic degradation model (PDM) that can model multiple degradation effects and bridge the gap between training and test datasets. This method allows high-resolution images to degrade into multiple low-resolution images, providing more training samples for the super-resolution model. Since the above methods are all prior arts well known to those skilled in the art, they will not be described in detail here.

[0072] When collecting samples for various scene types, it's important to cover as many situations as possible and set evaluation criteria for different scenarios to ensure data coverage and accuracy. For example, the evaluation criterion for a jungle scene is that the jungle background accounts for more than a preset proportion of the entire image, such as 50%. The evaluation criteria for a birdwatching scene are the presence of trees and a sky background, and so on. Additionally, cross-labeling can be used to reduce data labeling bias, allowing multiple people to label the same sample. Based on the labeled data, existing methods such as GoogleNet, AlexNet, and ResNet can be used for classification. Since these are mature technologies, they will not be discussed here.

[0073] S62. Construct an initial image processing model.

[0074] In this embodiment, the initial image processing model includes a pre-trained scene classification model and various pre-trained super-resolution image processing models; or the initial image processing model includes an initial scene classification model and an initial super-resolution image processing model. The initial scene classification model and the initial super-resolution image processing model are models that have not been pre-trained.

[0075] In this embodiment, when the scene classification model and the super-resolution image processing model have not been pre-trained, the two models are jointly trained from the beginning when the image processing model is trained. Since each sample in the sample data set includes a sample image and a sample scene type label corresponding to the sample image and a super-resolution sample label image corresponding to the sample image, the sample images collected under each sample scene are input into the initial scene classification model, and the scene classification model is trained with the sample scene type label as the training target, and the super-resolution image processing model is trained with the sample scene type label and the super-resolution sample label image corresponding to the sample image as the training target. The two models are trained with the same sample image input in the training iteration, so that the scene classification model in the image processing model learns scene features and the super-resolution image processing model learns scene type features and super-resolution features. After the image processing model training is completed, the scene classification model and the super-resolution image processing model are also trained.

[0076] In this embodiment, if Figure 7 As shown, Figure 7 Figure 1 is a schematic diagram of the network structure of an image processing model in one embodiment; the image processing model includes a scene classification model and multiple super-resolution image processing models. To accelerate the convergence of the image processing model, the scene classification model and each super-resolution image processing model can be pre-trained.

[0077] In an optional implementation, the step of pre-training a scene classification model includes: obtaining a scene training dataset, wherein each sample in the scene training dataset includes a sample image and a scene type label to which the sample image belongs; constructing an initial scene classification model; and training the initial scene classification model using an iterative method based on the scene training dataset to obtain a pre-trained scene classification model. Furthermore, sample images are obtained from the scene training dataset to form input data for the scene classification model under training, the scene classification model in the current iteration outputs the scene type corresponding to the input data in the current iteration, and the scene loss between the scene type corresponding to the input data in the current iteration and the scene type label is calculated. Based on the scene loss, it is determined whether the current iteration satisfies the training termination condition corresponding to the scene classification model. If the current iteration satisfies the training termination condition corresponding to the scene classification model, the scene classification model after the iteration is stopped is used as the pre-trained scene classification model. If the current iteration does not meet the training termination condition corresponding to the scene classification model, sample images are continued to be obtained from the scene training dataset for training.

[0078] The method of pre-training the super-resolution image processing model will be introduced in detail later.

[0079] S63. Based on the sample data set, the initial image processing model is iteratively trained using an iterative method to obtain a trained image processing model.

[0080] Optionally, the iterative training of the initial image processing model using an iterative method based on the sample data set to obtain the trained image processing model includes:

[0081] Obtaining a sample image from a sample dataset to form a sample input image for the image processing model in the current iteration;

[0082] using the sample input image as input to a scene classification model in a current iteration, outputting a confidence score corresponding to each scene type through the scene classification model in the current iteration, selecting the scene type with the highest confidence score as the scene type corresponding to the sample input image, and calculating a loss value between the scene type corresponding to the sample input image and a sample scene type label corresponding to the sample input image based on a first loss function to obtain a first loss value in the current iteration;

[0083] According to the scene type corresponding to the sample input image, select a target super-resolution image processing model that matches the scene type corresponding to the sample input image from multiple super-resolution image processing models, and calculate a loss value between a super-resolution output image corresponding to the target super-resolution image processing model and a super-resolution sample label image corresponding to the sample input image based on a second loss function to obtain a second loss value in the current iteration;

[0084] Calculating a total loss value based on the first loss value and the second loss value;

[0085] Based on the total loss value, when the current iteration does not meet the iteration termination condition, continue to obtain sample images from the sample data set for training; when the current iteration meets the iteration termination condition, the image processing model after stopping the iteration is used as the trained image processing model.

[0086] In this embodiment, specifically Figure 7 As shown, the image processing model includes two parts, one is the scene classification part, and the other is the image super-resolution processing part. The scene classification model performs scene classification on the sample input image, and the fully connected layer of the model outputs the probability of each scene type, and finally selects the maximum probability as the output of the final scene type, and calculates the first loss value. After determining the scene type, select the target super-resolution image processing model corresponding to the scene type, and finally obtain the super-resolution output image corresponding to the target super-resolution image processing model, and calculate the second loss value. The first loss function is the multi-classification cross entropy loss function, and the second loss function is the distance function, that is, the loss value between the super-resolution output image corresponding to the target super-resolution image processing model and the super-resolution sample label image corresponding to the sample input image is calculated. The first loss function and the second loss function are used to update the model weight parameters during back propagation. The formula of the total loss function L is as follows:

[0087] L=Lclass+αL1

[0088] L is the total loss function, and α is used to balance the image weight and classification weight. L1 is the L1 norm distance between the super-resolution output image corresponding to the target super-resolution image processing model and the super-resolution sample label image corresponding to the sample input image. Lclass represents the multi-class cross entropy loss function.

[0089] In this embodiment, each super-resolution image processing model is a pre-trained model, so each super-resolution image processing model has corresponded to its own scene type, and then the sample data set is used to combine the scene classification model and each super-resolution image processing model as a whole, so as to jointly optimize the network parameters in the two parts.

[0090] In the above embodiment, the scene classification model and the super-resolution image processing model are taken as a whole to form an image processing model for training, which can realize joint tuning of the model. By training as a whole, the scene features and the super-resolution image processing features of each scene type can be learned at the same time, so that the trained image processing model can realize end-to-end network, which can improve efficiency. The image processing model can adaptively adapt to various scenes and directly output super-resolution results.

[0091] In some embodiments, the method further comprises:

[0092] Training each super-resolution image processing model to obtain each trained super-resolution image processing model;

[0093] The training of each super-resolution image processing model to obtain each trained super-resolution image processing model includes:

[0094] For any super-resolution image processing model, a training data set corresponding to the super-resolution image processing model is collected under the scene type corresponding to the super-resolution image processing model, wherein the training data set includes each training image collected under the scene type and a super-resolution training label image corresponding to the training image;

[0095] Build an initial super-resolution image processing model;

[0096] Based on the training data set corresponding to the super-resolution image processing model, the initial super-resolution image processing model is iteratively trained using an iterative method to obtain a pre-trained super-resolution image processing model.

[0097] In this embodiment, each super-resolution image processing model is trained separately. For example, the training data set corresponding to the jungle scene is data set A, and the training data set for the bird watching scene is data set B. Then, data set A is used for training to obtain super-resolution image processing model C, and data set B is used for training to obtain super-resolution image processing model D. The super-resolution image processing model C is used to process images in the jungle scene, and the super-resolution image processing model D is used to process images in the bird watching scene.

[0098] In this embodiment, for any scene type, a training image is obtained from the training data set corresponding to the scene type to form a training input image for the super-resolution image processing model under the scene type. The super-resolution image processing model in the current iteration outputs a super-resolution output image corresponding to the training input image, and the super-resolution loss value between the super-resolution output image corresponding to the training input image and the super-resolution training label image corresponding to the training input image is calculated. Based on the super-resolution loss value, it is determined whether the current iteration meets the training termination condition corresponding to the super-resolution image processing model under the scene type. If so, the super-resolution image processing model after stopping the iteration is used as the pre-trained super-resolution image processing model under the scene type. If not, training continues. During the training process, the super-resolution image processing model under each scene type learns the super-resolution features in the super-resolution training label image under the corresponding scene type, thereby better processing the image under the scene type and outputting a super-resolution image that conforms to the scene.

[0099] During the training process of the above-mentioned image processing model, the initial image processing model may include a pre-trained super-resolution image processing model, which can speed up the training of the image processing model.

[0100] In this embodiment, the above-mentioned data set can be a sample data set or a training data set. For the sample data set or the training data set, the steps for collecting images under each scene type are the same, and the images collected under each scene type are low-resolution images (such as training images or sample images). The super-resolution image processing method corresponding to each scene type is used to super-process the low-resolution images under each scene to obtain high-resolution images under each scene type (such as super-resolution training label images or super-resolution sample label images). Under different scene types, the processing parameters corresponding to the super-resolution image processing method will be different, so that the super-resolution features in the high-resolution image (such as sharpening intensity, contrast, image detail noise parameters, and target image prominence) are also different. Therefore, the trained image processing model or the trained super-resolution image processing model can be used to optimize the images under each scene in a targeted manner.

[0101] In the above embodiment, the super-resolution image processing model under each scene type is trained by using the training data set of each scene type, so that the super-resolution image processing model learns the super-resolution features in the super-resolution training label image under the corresponding scene type, thereby better processing the image under the scene type and outputting the super-resolution image that conforms to the scene.

[0102] In some embodiments, the present application also provides a scene-based super-resolution image processing system, the system includes a super-resolution image processing device and a server that executes any scene-based super-resolution image processing method of the present application, the server communicates with the super-resolution image processing device, and the server is used to train the image processing model or the super-resolution image processing model. That is, the image processing device can communicate with the server, and the training of the above-mentioned image processing model, scene classification model, and each super-resolution image processing model can be performed in the server, and the server sends the parameter files trained for each model to the image processing device. In some embodiments, the training of the above-mentioned image processing model, scene classification model, and each super-resolution image processing model can be performed in other computing devices or image processing devices. After the training of the image processing model and / or each super-resolution image processing model is completed, after the model is trained based on the above-mentioned algorithm, a model conversion tool, such as a chip-specific tool chain, TensorRT, TNN, NCNN and other frameworks, can be used to convert it into a model that can be deployed on the corresponding platform. For example, using TensorRT, it is converted into a model that can be deployed on an Nvidia GPU to achieve real-time super-resolution. In specific implementation, the above algorithm can be deployed in but not limited to CPU, NPU or GPU.

[0103] like Figure 8 As shown, Figure 8 4 is a flowchart of a scene-based super-resolution image processing method according to another embodiment; the flowchart includes the following steps:

[0104] S81. Acquire the current scene image.

[0105] S82: Acquire current scene information, and determine a current super-resolution image processing model that matches the current scene information based on the current scene information.

[0106] S83. Based on the current super-resolution image processing model, perform super-resolution processing on the current scene image to obtain a target scene image.

[0107] like Figure 9 As shown, Figure 9 4 is a flowchart of a scene-based super-resolution image processing method in another embodiment; the flowchart includes the following steps:

[0108] S91. Acquire the current scene image.

[0109] S92. Obtain a trained image processing model, where the image processing model includes a scene classification model and multiple super-resolution image processing models.

[0110] S93. Based on the current scene image, classify the current scene image using a scene classification model in the image processing model to obtain a current scene type.

[0111] S94. Determine, from the multiple super-resolution image processing models, a super-resolution image processing model that matches the current scene type as the current super-resolution image processing model.

[0112] It is understandable that Figure 8 and Figure 9 The embodiments provided may be executed in an image processing device, or only one of them may exist.

[0113] On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the scene-based super-resolution image processing method described in any embodiment of the present application.

[0114] Among them, in the computer program product, an optional implementation form of the program module architecture of the computer program that implements each step of the scene-based super-resolution image processing method can be a scene-based super-resolution image processing device.

[0115] See also Figure 10 An embodiment of the present application provides a scene-based super-resolution image processing device, including: an acquisition module 101, used to acquire a current scene image; a determination module 102, used to determine a current super-resolution image processing model that matches the current scene image; and a processing module 103, used to perform super-resolution processing on the current scene image based on the current super-resolution image processing model to obtain a target scene image.

[0116] Optionally, the determining module 102 is further configured to:

[0117] Acquiring current scene information, and determining a current super-resolution image processing model that matches the current scene information based on the current scene information;

[0118] Based on the current scene image, a current scene type corresponding to the current scene image is determined, and a current super-resolution image processing model matching the current scene type is acquired.

[0119] Optionally, the determining module 102 is further configured to:

[0120] Acquire a trained image processing model, wherein the image processing model includes a scene classification model and multiple super-resolution image processing models;

[0121] Based on the current scene image, classify the current scene image using a scene classification model in the image processing model to obtain a current scene type;

[0122] From the multiple super-resolution image processing models, a super-resolution image processing model that matches the current scene type is determined as the current super-resolution image processing model.

[0123] Optionally, the determining module 102 is further configured to:

[0124] Based on the current scene image, input data of a scene classification model is formed, and the current scene type and the corresponding classification confidence are outputted through the scene classification model.

[0125] Optionally, the processing module 103 is further configured to:

[0126] Obtain target image processing parameters corresponding to the current scene type where the current scene image is located;

[0127] The current scene image is acquired and processed according to the target image processing parameters.

[0128] Optionally, the target image processing parameters include at least one of the following: detector response rate, contrast adjustment parameter, and noise suppression parameter.

[0129] Optionally, the processing module 103 is further configured to:

[0130] Based on the add scene control in the user interface, an add operation is obtained, and identification information of the to-be-added scene and scene sample data of the to-be-added scene are obtained according to the add operation.

[0131] Optionally, the super-resolution image processing apparatus may further include a training module 104, configured to:

[0132] Acquire a sample data set, where each sample in the sample data set includes a sample image, a sample scene type label corresponding to the sample image, and a super-resolution sample label image corresponding to the sample image;

[0133] Constructing an initial image processing model, which includes a pre-trained scene classification model and various pre-trained super-resolution image processing models;

[0134] Based on the sample data set, the initial image processing model is iteratively trained using an iterative method to obtain a trained image processing model.

[0135] Optionally, the training module 104 is further configured to:

[0136] Obtaining a sample image from a sample dataset to form a sample input image for the image processing model in the current iteration;

[0137] using the sample input image as input to a scene classification model in a current iteration, outputting a confidence score corresponding to each scene type through the scene classification model in the current iteration, selecting the scene type with the highest confidence score as the scene type corresponding to the sample input image, and calculating a loss value between the scene type corresponding to the sample input image and a sample scene type label corresponding to the sample input image based on a first loss function to obtain a first loss value in the current iteration;

[0138] According to the scene type corresponding to the sample input image, select a target super-resolution image processing model that matches the scene type corresponding to the sample input image from multiple super-resolution image processing models, and calculate a loss value between a super-resolution output image corresponding to the target super-resolution image processing model and a super-resolution sample label image corresponding to the sample input image based on a second loss function to obtain a second loss value in the current iteration;

[0139] Calculating a total loss value based on the first loss value and the second loss value;

[0140] Based on the total loss value, when the current iteration does not meet the iteration termination condition, continue to obtain sample images from the sample data set for training; when the current iteration meets the iteration termination condition, the image processing model after stopping the iteration is used as the trained image processing model.

[0141] Optionally, the training module 104 is further configured to:

[0142] Training each super-resolution image processing model to obtain each trained super-resolution image processing model;

[0143] The training of each super-resolution image processing model to obtain each trained super-resolution image processing model includes:

[0144] For any super-resolution image processing model, a training data set corresponding to the super-resolution image processing model is collected under the scene type corresponding to the super-resolution image processing model, wherein the training data set includes each training image collected under the scene type and a super-resolution training label image corresponding to the training image;

[0145] Build an initial super-resolution image processing model;

[0146] Based on the training data set corresponding to the super-resolution image processing model, the initial super-resolution image processing model is iteratively trained using an iterative method to obtain a pre-trained super-resolution image processing model.

[0147] It will be understood by those skilled in the art that Figure 10 The structure of the scene-based super-resolution image processing device does not constitute a limitation on the scene-based super-resolution image processing device. The various modules can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the scene-based super-resolution image processing device in the form of hardware, or can be stored in the memory of the scene-based super-resolution image processing device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. In other embodiments, the scene-based super-resolution image processing device can include more or fewer modules than shown in the figure.

[0148] See also Figure 11 On the other hand, an embodiment of the present application further provides a scene-based super-resolution image processing device 10, including a processor 13 and a memory 14, wherein the memory 14 stores a computer program. When the computer program is executed by the processor, the processor 13 executes the steps of the scene-based super-resolution image processing method provided in any of the above embodiments of the present application.

[0149] The processor 13 is the control center, connecting the various components of the scene-based super-resolution image processing device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 14, and calling data stored in the memory 14, the processor 13 performs various functions of the scene-based super-resolution image processing device and processes data. Optionally, the processor 13 may include one or more processing cores; preferably, the processor 13 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interfaces, and application programs, and the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 13.

[0150] The memory 14 can be used to store software programs and modules. The processor 13 executes various functional applications and data processing by running the software programs and modules stored in the memory 14. The memory 14 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created based on the use of a scene-based super-resolution image processing device, etc. In addition, the memory 14 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 14 may also include a memory processor to provide the processor 13 with access to the memory 14.

[0151] On the other hand, an embodiment of the present application further provides a storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the scene-based super-resolution image processing method provided in any of the above embodiments of the present application.

[0152] Those skilled in the art will appreciate that all or part of the processes in the methods provided in the above embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0153] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. The scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A scene-based super-resolution image processing method, characterized in that: include: Get the current scene image; Determining a current super-resolution image processing model that matches the current scene image; Based on the current super-resolution image processing model, super-resolution processing is performed on the current scene image to obtain a target scene image.

2. The scene-based super-resolution image processing method according to claim 1, wherein: The determining of the current super-resolution image processing model that matches the current scene image includes one or more of the following: Acquiring current scene information, and determining a current super-resolution image processing model that matches the current scene information based on the current scene information; Based on the current scene image, a current scene type corresponding to the current scene image is determined, and a current super-resolution image processing model matching the current scene type is acquired.

3. The scene-based super-resolution image processing method according to claim 1, wherein: Determining a current super-resolution image processing model that matches the current scene image includes: Acquire a trained image processing model, wherein the image processing model includes a scene classification model and multiple super-resolution image processing models; Based on the current scene image, classify the current scene image using a scene classification model in the image processing model to obtain a current scene type; From the multiple super-resolution image processing models, a super-resolution image processing model that matches the current scene type is determined as the current super-resolution image processing model.

4. The scene-based super-resolution image processing method according to claim 3, wherein: The classifying the current scene image based on the current scene image using the scene classification model in the image processing model to obtain the current scene type includes: Based on the current scene image, input data of a scene classification model is formed, and the current scene type and the corresponding classification confidence are outputted through the scene classification model.

5. The scene-based super-resolution image processing method according to claim 1, wherein: The method further comprises: Obtain target image processing parameters corresponding to the current scene type where the current scene image is located; The current scene image is acquired and processed according to the target image processing parameters.

6. The scene-based super-resolution image processing method according to claim 5, wherein: The target image processing parameters include at least one of the following: a detector response rate, a contrast adjustment parameter, and a noise suppression parameter.

7. The scene-based super-resolution image processing method according to claim 1, wherein: The method further comprises: Based on the add scene control in the user interface, an add operation is obtained, and identification information of the to-be-added scene and scene sample data of the to-be-added scene are obtained according to the add operation.

8. The scene-based super-resolution image processing method according to claim 1, wherein: The method further comprises: Acquire a sample data set, where each sample in the sample data set includes a sample image, a sample scene type label corresponding to the sample image, and a super-resolution sample label image corresponding to the sample image; Constructing an initial image processing model, the initial image processing model including a pre-trained scene classification model and each pre-trained super-resolution image processing model; or the initial image processing model including an initial scene classification model and an initial super-resolution image processing model; Based on the sample data set, the initial image processing model is iteratively trained using an iterative method to obtain a trained image processing model.

9. The scene-based super-resolution image processing method according to claim 8, wherein: The iterative training of the initial image processing model based on the sample data set using an iterative method to obtain a trained image processing model includes: Obtaining a sample image from a sample dataset to form a sample input image for the image processing model in the current iteration; using the sample input image as input to a scene classification model in a current iteration, outputting a confidence score corresponding to each scene type through the scene classification model in the current iteration, selecting the scene type with the highest confidence score as the scene type corresponding to the sample input image, and calculating a loss value between the scene type corresponding to the sample input image and a sample scene type label corresponding to the sample input image based on a first loss function to obtain a first loss value in the current iteration; According to the scene type corresponding to the sample input image, select a target super-resolution image processing model that matches the scene type corresponding to the sample input image from multiple super-resolution image processing models, and calculate a loss value between a super-resolution output image corresponding to the target super-resolution image processing model and a super-resolution sample label image corresponding to the sample input image based on a second loss function to obtain a second loss value in the current iteration; Calculating a total loss value based on the first loss value and the second loss value; Based on the total loss value, when the current iteration does not meet the iteration termination condition, continue to obtain sample images from the sample data set for training; when the current iteration meets the iteration termination condition, the image processing model after stopping the iteration is used as the trained image processing model.

10. The scene-based super-resolution image processing method according to claim 1 or 8, wherein: The method further comprises: Training each super-resolution image processing model to obtain each trained super-resolution image processing model; The training of each super-resolution image processing model to obtain each trained super-resolution image processing model includes: For any super-resolution image processing model, a training data set corresponding to the super-resolution image processing model is collected under the scene type corresponding to the super-resolution image processing model, wherein the training data set includes each training image collected under the scene type and a super-resolution training label image corresponding to the training image; Build an initial super-resolution image processing model; Based on the training data set corresponding to the super-resolution image processing model, the initial super-resolution image processing model is iteratively trained using an iterative method to obtain a pre-trained super-resolution image processing model.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the scene-based super-resolution image processing method according to any one of claims 1 to 10 is implemented.

12. A scene-based super-resolution image processing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the scene-based super-resolution image processing method according to any one of claims 1 to 10.

13. A storage medium comprising a computer program, characterized in that A computer program is stored, and when the computer program is executed by a processor, the processor is caused to execute the scene-based super-resolution image processing method according to any one of claims 1 to 10.

14. A scene-based super-resolution image processing system, characterized in that: The system includes a super-resolution image processing device and a server that executes the scene-based super-resolution image processing method as described in any one of claims 1 to 7, the server communicates with the super-resolution image processing device, and the server is used to train an image processing model or a super-resolution image processing model.

Images