Image super-resolution processing method and device, and electronic device
By performing segmentation and parallel processing of the image to be processed, the segmentation mode and processing mode are determined using image complexity, the problem of long super resolution processing time is solved, and efficient image reconstruction and display is achieved.
Patent Information
- Application Number
- CN202110685262.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-21
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-06-21
AI Technical Summary
In the prior art, the super-resolution processing time is long, resulting in a large delay and the processing efficiency cannot be effectively improved.
By segmenting the image to be processed, multiple super-resolution circuit modules in the super-resolution processing chip are input to parallel processing, and the target segmentation method and matching super-resolution processing mode are determined using image complexity to realize adaptive segmentation of the image and parallel processing.
The delay in super-resolution processing is significantly reduced, processing efficiency is improved, and low-resolution images can be reconstructed into high-resolution images more quickly.
Smart Images

Figure CN113421189B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of electronic equipment, and in particular to an image super-resolution processing method and device, an electronic device, and a computer-readable storage medium. Background Art
[0002] In recent years, with the development of display technology, the hardware resolution of display screens has been continuously improved. Although the resolution of display screens has been greatly improved, many content sources still have a lot of low resolution due to the influence of network bandwidth and shooting sources. In order to display at the resolution of the corresponding screen hardware, super-resolution processing is required to reconstruct low-resolution images into high-resolution images before displaying them.
[0003] However, super-resolution processing takes a long time, and there is a large delay in outputting the output to the display screen after the super-resolution processing is completed. Summary of the invention
[0004] The embodiments of the present application provide an image super-resolution processing method, device, electronic device and computer-readable storage medium, which segments the image according to the complexity of the image to be processed, and inputs the obtained segmented sub-images into at least two super-resolution circuit modules in a super-resolution processing chip for parallel super-resolution processing, thereby reducing the delay of super-resolution processing and improving the processing efficiency of super-resolution.
[0005] In a first aspect, an embodiment of the present application provides an image super-resolution processing method. The method is applied to a super-resolution processing chip, comprising:
[0006] Get the image to be processed;
[0007] Determining a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segmenting the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images;
[0008] Inputting the segmented sub-images into at least two super-resolution circuit modules respectively, and the super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks;
[0009] Each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image;
[0010] Performing super-resolution processing on the segmented sub-images according to a super-resolution processing mode that matches the segmentation category of the segmented sub-images to obtain super-resolution segmented sub-images corresponding to the segmented sub-images;
[0011] The super-resolution segmented sub-images are synthesized to obtain a target super-resolution image corresponding to the image to be processed.
[0012] In a second aspect, an embodiment of the present application provides an image super-resolution processing device. The device is applied to a super-resolution processing chip, comprising:
[0013] An acquisition module, used for acquiring an image to be processed;
[0014] A segmentation module, used to determine a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segment the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images;
[0015] A parallel super-resolution processing module, used to input the segmented sub-images into at least two super-resolution circuit modules respectively, the super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks, each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image, and performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode that matches the segmentation category of the segmented sub-image, so as to obtain a super-resolution segmented sub-image corresponding to each segmented sub-image;
[0016] The synthesis module is used to synthesize each super-resolution segmented sub-image to obtain a target super-resolution image corresponding to the image to be processed.
[0017] In a third aspect, an embodiment of the present application provides an electronic device. The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the following steps:
[0018] Get the image to be processed;
[0019] Determining a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segmenting the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images;
[0020] Inputting the segmented sub-images into at least two super-resolution circuit modules respectively, and the super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks;
[0021] Each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image;
[0022] Performing super-resolution processing on the segmented sub-images according to a super-resolution processing mode that matches the segmentation category of the segmented sub-images to obtain super-resolution segmented sub-images corresponding to the segmented sub-images;
[0023] The super-resolution segmented sub-images are synthesized to obtain a target super-resolution image corresponding to the image to be processed.
[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the processor executes the following steps:
[0025] Get the image to be processed;
[0026] Determining a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segmenting the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images;
[0027] Inputting the segmented sub-images into at least two super-resolution circuit modules respectively, and the super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks;
[0028] Each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image;
[0029] Performing super-resolution processing on the segmented sub-images according to a super-resolution processing mode that matches the segmentation category of the segmented sub-images to obtain super-resolution segmented sub-images corresponding to the segmented sub-images;
[0030] The super-resolution segmented sub-images are synthesized to obtain a target super-resolution image corresponding to the image to be processed.
[0031] The above-mentioned image super-resolution processing method, device, electronic device and computer-readable storage medium are applied to a super-resolution processing chip, obtain an image to be processed, determine a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, segment the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images, input the segmented sub-images into at least two super-resolution circuit modules respectively, and the super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks; each super-resolution circuit module respectively obtains a super-resolution processing mode matching the segmentation category of the input segmented sub-image, performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode matching the segmentation category of the segmented sub-image, obtains a super-resolution segmented sub-image corresponding to each segmented sub-image, synthesizes each super-resolution segmented sub-image to obtain a target super-resolution image corresponding to the image to be processed, determines the target segmentation method corresponding to the image to be processed according to the complexity of the image to be processed, so that the target segmentation method can be adaptively adjusted, and the segmentation effect is better determined, and the obtained segmented sub-images are respectively input into at least two super-resolution circuit modules in the super-resolution processing chip for parallel super-resolution processing, thereby reducing the delay of super-resolution processing and improving the processing efficiency of super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 A diagram showing an application environment of an image super-resolution processing method in an embodiment;
[0034] Figure 2 A block diagram of a partial structure of a mobile phone related to an electronic device provided in an embodiment;
[0035] Figure 3 Schematic diagram of a process flow of an image super-resolution processing method in one embodiment;
[0036] Figure 4 A schematic diagram of a flow chart for determining target super-resolution processing modes corresponding to different semantic categories in one embodiment;
[0037] Figure 5 A schematic diagram of a flow chart of obtaining a super-resolution processing mode that matches a segmentation category of an input segmented sub-image in one embodiment;
[0038] Figure 6A schematic diagram of a flow chart of an image super-resolution processing method in a specific embodiment;
[0039] Figure 7 A schematic diagram of equally divided image segmentation in one embodiment;
[0040] Figure 8 A schematic diagram of an original image and a segmentation result image of scene semantic segmentation in one embodiment;
[0041] Fig. 9 is a structural block diagram of an image super-resolution processing device in one embodiment;
[0042] Fig.10 FIG. 4 is a block diagram of the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] Related technical solutions often perform overall calculations based on the entire image, which takes a long time.
[0045] Figure 1 FIG. 1 is a schematic diagram of an application environment of an image super-resolution processing method in an embodiment. Figure 1 As shown, the application environment includes a terminal 102 and a server 104. The terminal 102 and the server 104 can communicate through a network. The terminal 102 can be a personal computer, a laptop, a smart phone, a tablet computer, a portable wearable device, etc., but is not limited thereto, and the server 104 can be a single server or a server cluster. Both the terminal 102 and the server 104 can include a super-resolution processing chip, and the image super-resolution processing method can be implemented in the terminal 102 or the server 104.
[0046] In one possible implementation, Figure 2 This is a block diagram of a partial structure of a mobile phone related to the electronic device provided in the embodiment of the present application. Figure 2 The mobile phone includes: a radio frequency (RF) circuit 210, a memory 220, an input unit 230, a display unit 240, a Bluetooth module 250, an audio circuit 260, a wireless fidelity (WIFI) module 270, a processor 280, and a power supply 290. Among them, the input unit 230 at least includes a video input device, such as a camera. Those skilled in the art can understand that Figure 2The structure of the mobile phone shown does not constitute a limitation on the mobile phone, and may include more or fewer components than shown, or combine certain components, or arrange the components differently. The Bluetooth module 250 supports Bluetooth function. Bluetooth technology is a radio technology that supports short-range communication between devices, and can perform wireless information exchange between a variety of wireless terminal devices including smart phones, personal computers, laptops, tablet computers, portable wearable devices, wireless headphones, wireless speakers, etc.
[0047] Figure 3 The figure is a flowchart of an image super-resolution processing method in one embodiment. Figure 3 The image super-resolution processing method shown can be applied to the terminal 102 or the server 104 including the super-resolution processing chip, including:
[0048] Step 310: Obtain an image to be processed.
[0049] The image to be processed may be an image captured by an electronic device, an image locally stored by the electronic device, or an image downloaded from the Internet, etc. The image to be processed may be a raw image, a YUV image, an RGB (Red Green Blue) image, etc. The super-resolution processing chip is a chip for performing super-resolution processing on an input image. The hardware structure of the super-resolution processing chip can be customized. The super-resolution processing chip includes at least two super-resolution circuit modules. At least two super-resolution circuit modules can simultaneously perform super-resolution processing on the segmented sub-images obtained by segmenting the image to be processed, so that the super-resolution processing can be processed in parallel for the segmented sub-images, thereby improving the efficiency of super-resolution processing.
[0050] Specifically, the image to be processed is a low-resolution image that needs to be super-resolution processed. If the resolution of the image to be processed is lower than the resolution of the display screen, the image to be processed needs to be super-resolution processed. The image to be processed can be a single-frame image with low complexity, such as a water surface image, or a multi-subject image with high complexity, such as an image including at least two subjects such as a person, a tree, a mountain, a house, a grassland, and a sky.
[0051] Step 320 , determining a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segmenting the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images.
[0052] Among them, image complexity is used to describe the complexity of image content. Image features can be extracted, such as extracting image edges, and image complexity can be calculated through image features. If the image complexity is low, the image content is simple, such as a single content image, such as an image with only one apple. If the image complexity is high, the image content is complex, such as a multi-subject content image, such as an image including people, cars, streets, etc.
[0053] Specifically, different image complexities correspond to different target segmentation methods. The correspondence between image complexity and target segmentation method can be customized. The target segmentation method can be to divide the image into sub-images of the same size or to divide the image into sub-images of different sizes. Each sub-image can be an image of a regular shape, such as a rectangle, square, etc., or an image of an irregular shape, such as a sub-image obtained by dividing the outline of a person or a sub-image obtained by dividing the outline of a mountain.
[0054] In a possible implementation, if the image complexity is less than the first threshold, the target segmentation method corresponding to the image to be processed is equal image segmentation, and the image to be processed is divided into at least two segmented sub-images of the same size. The image size of each segmented sub-image can be customized, such as 64*64, 64*16, etc.
[0055] In one possible implementation, if the image complexity is greater than or equal to a first threshold and less than a second threshold, the foreground area and background area corresponding to the image to be processed are identified, and the image to be processed is divided into a foreground sub-image and a background sub-image, wherein the second threshold is greater than the first threshold.
[0056] In a possible implementation, if the image complexity is greater than or equal to a second threshold, each image subject in the image to be processed is identified and each image subject is divided into corresponding segmented sub-images. For example, if the image to be processed includes a person, a bicycle, a building, and the sky, the image to be processed is divided into a person sub-image, a bicycle sub-image, a building sub-image, and a sky sub-image, and the remaining area outside the subject is taken as a segmented sub-image to obtain all the segmented sub-images after the image to be processed is divided.
[0057] Step 330: input the segmented sub-images into at least two super-resolution circuit modules respectively. The super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks. Each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image, and performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode that matches the segmentation category of the segmented sub-image, so as to obtain super-resolution segmented sub-images corresponding to each segmented sub-image.
[0058] Among them, the super-resolution circuit module is a multi-level hardware super-resolution processing module, which can be divided into N super-resolution circuit modules. Each super-resolution circuit module is responsible for the resolution super-resolution operation of different areas of the picture and the data output after super-resolution, where N is greater than or equal to 2, and the size of N can be customized. The number of super-resolution circuit modules can be determined according to the scene of image processing, the resource processing capacity of the device, etc. The super-resolution circuit module is located on the super-resolution processing chip, and the number of super-resolution circuit modules is determined when the super-resolution processing chip leaves the factory.
[0059] In one possible implementation, the super-resolution circuit module is a dynamically changeable hardware module used in conjunction with a super-resolution processing chip, and can be embedded in the super-resolution processing chip by insertion. Each super-resolution circuit module is an independent and movable hardware module, so that the number of super-resolution circuit modules can be increased or decreased as needed after the super-resolution processing chip leaves the factory.
[0060] Specifically, the segmented sub-images are input to N super-resolution circuit modules for super-resolution calculations. The N super-resolution circuit modules perform algorithm calculations in parallel. The more N the number, the faster the calculation speed and the higher the super-resolution processing efficiency. Each super-resolution circuit module can use the same or different super-resolution algorithms when performing super-resolution calculations on the input segmented sub-images. Different super-resolution algorithms are different super-resolution processing modes. The specific target super-resolution algorithm can be determined according to the segmentation category of the input segmented sub-image. The correspondence between the segmentation category of the segmented sub-image and the super-resolution processing mode can be customized. For example, if the segmentation category of the segmented sub-image is a person, it corresponds to the first target super-resolution processing mode. If the segmentation category of the segmented sub-image is a river, it corresponds to the second target super-resolution processing mode. The correspondence between the segmentation category of the segmented sub-image and the super-resolution processing mode can also be trained by model training to obtain the optimal super-resolution processing mode corresponding to each segmentation category.
[0061] The super-resolution circuit module can receive image processing instructions sent by other image processing modules, and perform super-resolution processing on the input segmented sub-images according to the received image processing instructions. Different segmented sub-images can correspond to different super-resolution modes, so the image processing instructions received by different super-resolution circuit modules may match the corresponding super-resolution modes, which may also be different. The segmented sub-images are super-resolution processed according to the super-resolution processing mode that matches the segmentation category of the segmented sub-images to obtain the super-resolution segmented sub-images corresponding to each segmented sub-image. For example, if the segmentation category of the first segmented sub-image matches the first super-resolution processing mode, the first segmented sub-image is super-resolution processed according to the first super-resolution processing mode to obtain the first super-resolution segmented sub-image. If the segmentation category of the second segmented sub-image matches the second super-resolution processing mode, the second segmented sub-image is super-resolution processed according to the second super-resolution processing mode to obtain the second super-resolution segmented sub-image.
[0062] Step 340 , synthesizing the super-resolution segmented sub-images to obtain a target super-resolution image corresponding to the image to be processed.
[0063] Specifically, each super-resolution segmented sub-image obtained by the previous segmentation and super-resolution processing can be synthesized by an image synthesizer to form a complete target super-resolution image, and the complete target super-resolution image can be output to a display screen for display. During the synthesis, each super-resolution segmented sub-image can be spliced according to the position of each super-resolution segmented sub-image in the original image to form a complete image. The spliced edges can be smoothed to make the quality of the target super-resolution image higher.
[0064] The image super-resolution processing method in this embodiment is applied to a super-resolution processing chip, obtains an image to be processed, determines a target segmentation mode corresponding to the image to be processed based on the image complexity of the image to be processed, segments the image to be processed according to the target segmentation mode to obtain at least two corresponding segmented sub-images, inputs the segmented sub-images into at least two super-resolution circuit modules respectively, and the super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks; each super-resolution circuit module respectively obtains a super-resolution processing mode matching the segmentation category of the input segmented sub-image, performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode matching the segmentation category of the segmented sub-image, obtains a super-resolution segmented sub-image corresponding to each segmented sub-image, synthesizes each super-resolution segmented sub-image to obtain a target super-resolution image corresponding to the image to be processed, determines the target segmentation mode corresponding to the image to be processed according to the complexity of the image to be processed, so that the target segmentation mode can be adaptively adjusted, and the segmentation effect is better determined, and the obtained segmented sub-images are respectively input into at least two super-resolution circuit modules in the super-resolution processing chip for parallel super-resolution processing, thereby reducing the delay of super-resolution processing and improving the processing efficiency of super-resolution.
[0065] The above-mentioned embodiments mainly describe how the super-resolution processing chip segments the input image to be processed according to the image complexity, and performs parallel super-resolution processing on the segmented sub-image blocks through at least two super-resolution circuit modules to improve the technical solution of super-resolution processing efficiency. In addition, other embodiments of the present application also introduce some feasible solutions on how to determine the target segmentation method corresponding to the image to be processed according to the image complexity, so as to optimize the segmentation of the image to be processed with different image contents. Step 320 includes: if it is determined that the image to be processed belongs to a single-frame image according to the image complexity of the image to be processed, the image to be processed is equally segmented to obtain at least two segmented sub-images of the same area, and the image complexity corresponding to the single-frame image is less than a first preset threshold.
[0066] Among them, a single-frame image refers to an image with a single frame, such as the sky. Equal image segmentation refers to dividing an image into sub-images of the same size, which can be regular or irregular in shape. The first preset threshold can be customized as needed. The size of the first preset threshold can be customized according to different scenes, and the way of calculating image complexity is different, and the corresponding first preset threshold is also different, corresponding to the way of calculating image complexity.
[0067] Specifically, for a single-frame image with low complexity, semantic recognition is not required. Instead, the image needs to be segmented into equal parts into sub-images of different sizes, where the equal areas can be defined by the user and each sub-image can be a rectangle or other shape.
[0068] In this embodiment, by comparing the image complexity with the first preset threshold, a single-frame image is automatically identified without semantic recognition, and the image can be equally divided and segmented to obtain at least two segmented sub-images of the same area. The image segmentation is completed quickly, and a matching equally divided image segmentation method is automatically determined for the single-frame image, which is efficient and intelligent.
[0069] The above embodiments mainly illustrate that the super-resolution processing chip can automatically identify a single-frame image, perform equal image segmentation, and quickly complete image segmentation. Other embodiments of the present application also introduce how to identify complex multi-scene images, perform AI (Artificial Intelligence) image semantic segmentation processing on non-single-frame images, and obtain segmented sub-images corresponding to different semantic categories. On the basis of the above embodiments, step 320 includes: if it is determined that the image to be processed belongs to a non-single-frame image based on the image complexity of the image to be processed, scene semantic segmentation is performed on the image to be processed to obtain at least two segmented sub-images corresponding to different semantic categories, wherein the image complexity corresponding to the single-frame image is less than the image complexity corresponding to the non-single-frame image.
[0070] Among them, scene semantic segmentation is based on the semantics of the image. Semantics refers to the content of the image and the understanding of the meaning of the image. For example, the semantics of an image is three people riding three bicycles. Segmentation means segmenting different objects in the image from the perspective of pixels and annotating each pixel in the original image. For example, different colors are used to annotate. In the image, pink represents people and green represents bicycles. Non-single-image images refer to images with complex images and multiple scenes or subjects, such as people, trees, mountains, etc.
[0071] Specifically, for non-single-frame images with higher complexity than single-frame images, scene semantic segmentation is performed on the image to be processed, and at least two segmented sub-images are formed by areas corresponding to different semantic categories. The segmented sub-images generated by the semantic categories can accurately distinguish the content of the image, which facilitates subsequent super-resolution processing for different image content areas.
[0072] In this embodiment, non-single-frame images are automatically identified through image complexity, scene semantic segmentation is performed, and accurate segmentation is performed based on the content semantics of the image to improve the accuracy of image segmentation. A scene semantic segmentation method that matches the non-single-frame image is determined to be accurate and intelligent.
[0073] The above embodiments mainly illustrate that the super-resolution processing chip can automatically identify non-single-frame images, perform scene semantic segmentation, and improve the accuracy of image segmentation. In addition, other embodiments of the present application also introduce how to determine the target super-resolution processing mode corresponding to images of different semantic categories. Figure 4 As shown, before step 310, the following steps are also included:
[0074] Step 410, obtaining a low-resolution first picture set and a corresponding high-resolution second picture set through a cloud database, wherein a first picture in the first picture set has a corresponding second picture with the same content in the second picture set, and the first picture in the first picture set has first pictures of different semantic categories.
[0075] Specifically, the super-resolution processing chip performs a super-resolution algorithm learning through a cloud database before leaving the factory to obtain a low-resolution first picture set and a corresponding high-resolution second picture set. The first picture in the first picture set has a corresponding second picture with the same content in the second picture set. For example, if the first picture set includes a low-resolution picture A, the second picture set includes a high-resolution picture B with the same content as picture A. Each low-resolution picture in the first picture set can be obtained by downsampling each high-resolution picture in the second picture set. In addition, in order to train and obtain the target super-resolution processing mode corresponding to images of different semantic categories, the semantic categories of the first pictures in the first picture set used for training also include multiple different ones. For example, the first picture set includes low-resolution pictures of different semantic categories such as flowers, trees, and people, and the second picture set includes high-resolution pictures of different semantic categories corresponding to flowers, trees, and people.
[0076] In step 420, the cloud database performs algorithm fitting on the difference between the first image and the second image of the same content, simulates the target super-resolution processing mode of each first image from low-resolution super-resolution to high-resolution second image, and obtains the target super-resolution processing mode corresponding to images of different semantic categories.
[0077] Specifically, the first image is processed by different super-resolution processing modes to obtain a plurality of corresponding super-resolution fitting images, and the difference between each super-resolution fitting image and the second image is calculated, and the super-resolution processing mode with the smallest difference is used as the target super-resolution processing mode of the semantic category corresponding to the first image. For example, for a first image with a semantic category of person, if the super-resolution fitting image obtained by processing with the first super-resolution processing mode is closest to the high-resolution second image matching the first image, the first super-resolution processing mode is used as the target super-resolution processing mode with a semantic category of person.
[0078] In a possible implementation, the first image of the same semantic category in the low-resolution first image set is input into a super-resolution processing network, and after being processed in sequence by each layer included in the super-resolution processing network, a corresponding super-resolution fitting image is output; the network parameters of the super-resolution processing network are adjusted by back-propagation according to the difference between the super-resolution fitting image and the corresponding first image, so as to obtain a trained super-resolution processing network that matches the current semantic category. First images of different semantic categories are respectively trained as training data to obtain super-resolution processing networks that match different semantic categories, and each different super-resolution processing network is used to process input images of different semantic categories in a corresponding target super-resolution processing mode. The super-resolution processing network can more accurately determine the corresponding target super-resolution processing mode for images of different semantic categories, improve the matching degree between the target super-resolution processing mode and the semantic category, and improve the processing quality of super-resolution processing for different images.
[0079] Step 430 , programming different semantic categories and corresponding target super-resolution processing modes into the intelligent image processing module of the super-resolution processing chip.
[0080] Among them, the intelligent image processing module is used to perform pre-processing on the input image to be processed before super-resolution, including determining the target super-resolution processing mode corresponding to images of different semantic categories, wherein the association between the semantic category and the corresponding target super-resolution processing mode is stored. The intelligent image processing module can also be used to determine the target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segment the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images. The intelligent image processing module can also include other image processing capabilities, which are not limited.
[0081] Specifically, different semantic categories and corresponding target super-resolution processing modes are burned into the intelligent image processing module of the super-resolution processing chip, so that the intelligent image processing module can determine the target super-resolution processing mode corresponding to each segmented sub-image according to the semantic category of the input segmented sub-image of each super-resolution circuit module, and instruct the super-resolution circuit module to perform super-resolution processing on the segmented sub-image through the target super-resolution processing mode corresponding to the segmented sub-image. The intelligent image processing module cooperates with each super-resolution circuit module to send image processing instructions matching the target super-resolution processing mode to each super-resolution circuit module. Different super-resolution circuit modules have different semantic categories of input segmented sub-images, and the corresponding target super-resolution processing modes are different, and the received image processing instructions are also different, so as to control each super-resolution circuit module to perform corresponding processing, and realize parallel processing of segmented sub-images of different semantic categories using different super-resolution processing modes. The intelligent image processing module will provide different algorithms according to the segmented sub-images of different semantic categories after segmentation, so that the corresponding super-resolution circuit module can use them when processing the corresponding segmented sub-image, so as to achieve high efficiency and high quality of super-resolution effect.
[0082] In this embodiment, a target super-resolution processing mode corresponding to each image of a different semantic category is determined by algorithm fitting through a low-resolution first image set and a corresponding high-resolution second image set, and the association between the semantic category and the corresponding target super-resolution processing mode is burned into the intelligent image processing module of the super-resolution processing chip, so that the intelligent image processing module can determine the corresponding target super-resolution processing mode according to the semantic category, thereby controlling the super-resolution circuit module to adopt the target super-resolution processing mode that matches the semantic category of the currently input segmented sub-image when performing super-resolution processing on the image, so as to achieve the best super-resolution processing effect.
[0083] The above embodiment mainly describes that the super-resolution processing chip can burn the association relationship between the semantic category and the corresponding target super-resolution processing mode into the intelligent image processing module in advance, so as to facilitate the intelligent image processing module to determine the corresponding target super-resolution processing mode according to the semantic category, and realize the parallel processing of different super-resolution processing modes for segmented sub-images of different semantic categories. In addition, other embodiments of the present application also introduce how each super-resolution circuit module obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image, and performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode that matches the segmentation category of the segmented sub-image. On the basis of the above embodiment, the segmentation category is one of the semantic categories, and step 330 includes each super-resolution circuit module receiving a super-resolution image processing instruction that matches the target super-resolution processing mode transmitted by the intelligent image processing module, and performing super-resolution processing on the segmented sub-image according to the super-resolution image processing instruction, to obtain a super-resolution segmented sub-image corresponding to each segmented sub-image; wherein the target super-resolution processing mode is determined by the intelligent image processing module according to the segmentation category of the segmented sub-image input by the super-resolution circuit module.
[0084] The resolution image processing instruction is matched with the target super-resolution processing mode, and is used to instruct the receiving device to execute the algorithm steps corresponding to the target super-resolution processing mode according to the instruction, thereby realizing super-resolution processing of the segmented sub-image.
[0085] Specifically, after the intelligent image processing module determines the target super-resolution processing mode according to the segmentation category of the segmented sub-image input by the super-resolution circuit module, it sends a super-resolution image processing instruction matching the target super-resolution processing mode to the corresponding super-resolution circuit module. For example, if the segmentation category of the segmented sub-image input by the first super-resolution circuit module is person, the target super-resolution processing mode is the first mode, and the intelligent image processing module sends the super-resolution image processing instruction corresponding to the first mode to the first super-resolution circuit module. If the segmentation category of the segmented sub-image input by the second super-resolution circuit module is mountain, the target super-resolution processing mode is the second mode, and the intelligent image processing module sends the super-resolution image processing instruction corresponding to the second mode to the second super-resolution circuit module.
[0086] In this embodiment, the super-resolution circuit module completes super-resolution processing of the input segmented sub-images in parallel by receiving super-resolution image processing instructions that match the target super-resolution processing mode. Different segmentation categories correspond to different super-resolution image processing instructions. The super-resolution circuit module only needs to receive the instructions, and the intelligent image processing module uniformly sends the corresponding super-resolution image processing instructions to each super-resolution circuit module to work together to further improve the efficiency of super-resolution image processing.
[0087] The above embodiment mainly illustrates that the super-resolution processing chip can send corresponding super-resolution image processing instructions to each super-resolution circuit module through the intelligent image processing module, and instruct the corresponding super-resolution circuit module to complete the super-resolution image processing matching the target super-resolution processing mode according to the super-resolution image processing instructions, which is efficient and intelligent. In addition, other embodiments of the present application also introduce the specific relationship between the super-resolution processing mode and the segmentation category of the segmented sub-image. On the basis of the above embodiment, Figure 5 As shown, each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image, including:
[0088] Step 510: If the segmented sub-image is of the foreground subject category, the complexity of the super-resolution processing mode matched with the foreground subject category acquired by the super-resolution circuit module is proportional to the importance of the foreground subject corresponding to the segmented sub-image in the image to be processed.
[0089] The foreground subject category indicates that the segmented sub-image is in the foreground part of the image to be processed, such as the segmented sub-images corresponding to people, trees, mountains, houses, and grass all belong to the foreground subject category.
[0090] Specifically, the importance of the foreground subject in the image to be processed can be set manually, and different subject categories correspond to different importances, such as people have the highest importance. The importance of each foreground subject in the image to be processed can also be automatically identified according to the area size or custom rules. For example, the larger the area corresponding to the foreground subject is defined, the higher the importance. The complexity of the super-resolution processing mode is proportional to the importance of the foreground subject corresponding to the segmented sub-image in the image to be processed. The higher the importance of the foreground subject, the higher the complexity of the corresponding super-resolution processing mode, so that the image content with high importance adopts a high-complexity super-resolution processing mode to ensure processing accuracy, and further improve the super-resolution processing effect of each foreground subject.
[0091] Step 520: If the segmented sub-image is of the background category, the complexity of the super-resolution processing mode matched with the background category acquired by the super-resolution circuit module is smaller than the complexity of the super-resolution processing mode corresponding to the foreground subject category.
[0092] The background category indicates that the segmented sub-image is in the background part of the image to be processed. The background part of the image to be processed may be composed of the area excluding the foreground main part, or may be identified by the background recognition module to obtain the background part of the image to be processed.
[0093] Specifically, the complexity of the super-resolution processing mode for background category matching is lower than that of the super-resolution processing mode corresponding to the foreground object category. Therefore, a super-resolution processing mode with low complexity is assigned to the background part of the image. Since the attention to the background part is generally lower than that to the foreground object part, assigning a super-resolution processing with lower complexity facilitates improving the super-resolution processing efficiency. It can be understood that when there are multiple foreground object categories, the complexity of the super-resolution processing mode for background category matching is lower than the minimum complexity of the super-resolution processing modes corresponding to the foreground object categories. For example, the complexities of the super-resolution processing modes corresponding to the foreground object categories are A1, A2, and A3 respectively, where A1 < A2 < A3, and the complexity of the super-resolution processing mode for background category matching is lower than A1.
[0094] In this embodiment, the complexities of the super-resolution processing modes corresponding to images of different semantic categories are different. Moreover, the higher the importance of the foreground object in the image to be processed, the higher the complexity of the corresponding super-resolution processing mode. The complexity of the super-resolution processing mode for the background category is lower than that of the super-resolution processing mode corresponding to the foreground object category, enabling the image content with high importance to adopt a super-resolution processing mode with high complexity to ensure the processing accuracy, further improving the super-resolution processing effect of each foreground object, and enabling the image content with low importance to adopt a super-resolution processing mode with low complexity to ensure the super-resolution processing efficiency.
[0095] The above embodiment mainly elaborates that the super-resolution processing chip processes images with different image contents according to their importance using super-resolution processing modes with different complexities, ensuring the super-resolution processing effect of each important image content while also ensuring the super-resolution processing efficiency. In addition, other embodiments of this application also introduce how to determine the complexity of the super-resolution processing mode corresponding to the segmented sub-images obtained by equally dividing an image. Based on the above embodiment, each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image, including: if the segmented sub-image is obtained by equally dividing the image to be processed, the complexities of the super-resolution processing modes that each super-resolution circuit module respectively obtains and matches the segmentation category of the input segmented sub-image are the same.
[0096] Specifically, an image of a single frame is generally equally divided. Since the content of each segmented sub-image obtained by equally dividing the image of a single frame is also simple, a super-resolution processing mode with the same complexity can be adopted for the segmented sub-images obtained by equally dividing the image. The specific complexity value can be customized as needed and defined according to the image features of the image to be processed. In one embodiment, the higher the overall image complexity of the image to be processed, the higher the complexity of the super-resolution processing mode.
[0097] In this embodiment, a super-resolution processing mode of the same complexity is used for the segmented sub-images obtained by equally segmenting the image. Since a single-frame image is generally segmented into equally divided images, the use of a super-resolution processing mode of the same complexity can ensure the integrity of the entire image processing.
[0098] The above embodiment describes that the super-resolution processing chip uses the same complexity super-resolution processing mode for the segmented sub-images obtained by equally segmenting the image to ensure the integrity of the entire image processing. In addition, other embodiments of the present application also introduce how to display the target super-resolution image. Based on the above embodiment, after step 340, it also includes: displaying the target super-resolution image through a display screen.
[0099] Specifically, if the device where the super-resolution processing chip is located has a display screen, the target super-resolution image is directly displayed through the display screen. If the device where the super-resolution processing chip is located is a terminal, the target super-resolution image is displayed through the terminal screen. If the device where the super-resolution processing chip is located does not have a display screen, the target super-resolution image can be sent to a device with a display screen for display. If the device where the super-resolution processing chip is located is a server, the target super-resolution image can be sent from the server to the terminal for display.
[0100] In this embodiment, the target super-resolution image obtained by processing is displayed on a display screen, so that the image after super-resolution processing is easy to watch.
[0101] In a specific embodiment, the super-resolution processing chip is located in a terminal, and an image super-resolution processing method is provided.
[0102] like Figure 6 The flowchart is shown, which specifically includes the following steps:
[0103] 1. The application data in the terminal is input into the encoding and decoding module. After the data is encoded and decoded, it is input into the GPU for drawing processing. The low-resolution image drawn is used as the image to be processed.
[0104] 2. After the image to be processed is obtained, the image complexity of the image to be processed can be calculated by the intelligent image processing module. If the image complexity is less than the first threshold, the image is equally segmented to obtain at least two segmented sub-images of the same area, such as Figure 7 As shown in FIG. 1 , a schematic diagram of equally divided image segmentation is shown. If the image complexity is greater than or equal to the first threshold, scene semantic segmentation is performed to obtain at least two segmented sub-images of different semantic categories. In one embodiment, after the segmentation is completed, Figure 8As shown, a sky sub-image, a green plant sub-image and a background sub-image are obtained, and then these three sub-images are respectively input into three super-resolution circuit modules for super-resolution processing.
[0105] The super-resolution processing chip is trained on the super-resolution algorithm through the cloud database before leaving the factory. The learning method is to simulate the best algorithm from low-resolution super-resolution to high-resolution by fitting the algorithm for the difference between low-resolution and high-resolution images of the same content. Different semantic categories, such as people, trees, mountains, houses, grass, sky, etc., correspond to different super-resolution algorithms, that is, to different best super-resolution processing modes. After learning the AI algorithm of the cloud library, the semantic category and the corresponding target super-resolution processing mode are burned into the intelligent image processing module to obtain the intelligent image processing super-resolution algorithm library.
[0106] 3. Input the segmented sub-images into three super-resolution circuit modules respectively, each super-resolution circuit module performs parallel super-resolution processing on the segmented sub-image blocks, each super-resolution circuit module receives the super-resolution image processing instruction sent by the intelligent image processing module that matches the target super-resolution processing mode, and performs super-resolution processing on the segmented sub-images according to the super-resolution image processing instruction, so as to obtain the super-resolution segmented sub-images corresponding to each segmented sub-image. The target super-resolution processing mode is determined by the intelligent image processing module according to the segmentation category of the segmented sub-images input by the super-resolution circuit module. For example, the sky sub-image, the green plant sub-image and the background sub-image are respectively input into the first super-resolution circuit module, the second super-resolution circuit module and the third super-resolution circuit module, and the target super-resolution processing modes are the first mode, the second mode and the third mode respectively. Then, the first super-resolution circuit module receives the super-resolution image processing instruction corresponding to the first mode, the second super-resolution circuit module receives the super-resolution image processing instruction corresponding to the second mode, and the third super-resolution circuit module receives the super-resolution image processing instruction corresponding to the third mode, and processes the sky super-resolution sub-image, the green plant super-resolution sub-image and the background super-resolution sub-image respectively.
[0107] 4. Input each super-resolution segmentation sub-image into an image synthesizer for synthesis to obtain a target super-resolution image corresponding to the image to be processed, and output it to a display screen for display.
[0108] In this embodiment, the super-resolution processing module is partitioned and can be divided into at least two super-resolution circuit modules. Each super-resolution circuit module is responsible for super-resolution processing of different areas of the picture or different semantic categories. A single picture can be processed in a single super-resolution processing mode by an equal image segmentation method. Complex image pictures can be segmented by scene semantics through an intelligent image processing module, and then the segmented sub-images of different semantic categories are given to different super-resolution circuit modules for super-resolution processing. Different super-resolution processing modes after AI learning are used according to scenes of different semantic categories to achieve the best super-resolution effect and the lowest super-resolution delay.
[0109] It should be understood that although Figure 3-5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 3-5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0110] Fig. 9 FIG. 9 is a structural block diagram of an image super-resolution processing device 900 according to an embodiment. Fig. 9 As shown, an image super-resolution processing device 900 is applied to a super-resolution processing chip, including: an acquisition module 902, a segmentation module 904, a parallel super-resolution processing module 906, and a synthesis module 908. Among them:
[0111] The acquisition module 902 is used to acquire the image to be processed.
[0112] The segmentation module 904 is used to determine a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segment the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images.
[0113] The parallel super-resolution processing module 906 is used to input the segmented sub-images into at least two super-resolution circuit modules respectively. The super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks. Each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image, and performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode that matches the segmentation category of the segmented sub-image, so as to obtain super-resolution segmented sub-images corresponding to each segmented sub-image.
[0114] The synthesis module 908 is used to synthesize the super-resolution segmented sub-images to obtain a target super-resolution image corresponding to the image to be processed.
[0115] The image super-resolution processing device 900 in this embodiment is applied to a super-resolution processing chip, obtains an image to be processed, determines a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, segments the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images, inputs the segmented sub-images into at least two super-resolution circuit modules respectively, and the super-resolution circuit modules perform parallel super-resolution processing on the segmented sub-image blocks; each super-resolution circuit module respectively obtains a super-resolution processing mode matching the segmentation category of the input segmented sub-image, performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode matching the segmentation category of the segmented sub-image, obtains a super-resolution segmented sub-image corresponding to each segmented sub-image, synthesizes each super-resolution segmented sub-image to obtain a target super-resolution image corresponding to the image to be processed, determines the target segmentation method corresponding to the image to be processed according to the complexity of the image to be processed, so that the target segmentation method can be adaptively adjusted to better determine the segmentation effect, and inputs the obtained segmented sub-images into at least two super-resolution circuit modules in the super-resolution processing chip respectively, performs parallel super-resolution processing, reduces the delay of super-resolution processing, and improves the processing efficiency of super-resolution.
[0116] In one embodiment, the segmentation module 904 is also used to divide the image to be processed into equal parts to obtain at least two segmented sub-images of the same area if it is determined that the image to be processed belongs to a single-frame image based on the image complexity of the image to be processed, and the image complexity corresponding to the single-frame image is less than a first preset threshold.
[0117] The image super-resolution processing device 900 in this embodiment automatically identifies a single-frame image by comparing the image complexity with a first preset threshold value, and can perform equal image segmentation without semantic recognition to obtain at least two segmented sub-images of the same area, quickly complete image segmentation, and automatically determine an equal image segmentation method that matches the single-frame image, which is efficient and intelligent.
[0118] In one embodiment, the segmentation module 904 is also used to perform scene semantic segmentation on the image to be processed if it is determined that the image to be processed belongs to a non-single-frame image based on the image complexity of the image to be processed, so as to obtain at least two segmented sub-images corresponding to different semantic categories, wherein the image complexity corresponding to the single-frame image is less than the image complexity corresponding to the non-single-frame image.
[0119] The image super-resolution processing device 900 in this embodiment automatically identifies non-single-frame images through image complexity, performs scene semantic segmentation, performs precise segmentation based on the content semantics of the image, improves the accuracy of image segmentation, and determines a scene semantic segmentation method that matches the non-single-frame image, which is accurate and intelligent.
[0120] In one embodiment, the apparatus further comprises:
[0121] The pre-processing module 910 is used to obtain a low-resolution first picture set and a corresponding high-resolution second picture set through a cloud database, wherein a first picture in the first picture set has a corresponding second picture with the same content in the second picture set, and the first picture set has first pictures of different semantic categories; the cloud database simulates the target super-resolution processing mode of each first picture from low-resolution super-resolution to high-resolution second picture by performing algorithm fitting on the difference between the first picture and the second picture with the same content, and obtains the target super-resolution processing mode corresponding to pictures of different semantic categories; and burns different semantic categories and corresponding target super-resolution processing modes into the intelligent image processing module of the super-resolution processing chip.
[0122] In this embodiment, a target super-resolution processing mode corresponding to each image of a different semantic category is determined by algorithm fitting through a low-resolution first image set and a corresponding high-resolution second image set, and the association between the semantic category and the corresponding target super-resolution processing mode is burned into the intelligent image processing module of the super-resolution processing chip, so that the intelligent image processing module can determine the corresponding target super-resolution processing mode according to the semantic category, thereby controlling the super-resolution circuit module to adopt the target super-resolution processing mode that matches the semantic category of the currently input segmented sub-image when performing super-resolution processing on the image, so as to achieve the best super-resolution processing effect.
[0123] In one embodiment, the segmentation category is one of the semantic categories, and the parallel super-resolution processing module 906 is also used for each super-resolution circuit module to receive a super-resolution image processing instruction that matches the target super-resolution processing mode and is transmitted by the intelligent image processing module, and to perform super-resolution processing on the segmented sub-image according to the super-resolution image processing instruction to obtain a super-resolution segmented sub-image corresponding to each segmented sub-image; wherein the target super-resolution processing mode is determined by the intelligent image processing module according to the segmentation category of the segmented sub-image input by the super-resolution circuit module.
[0124] In this embodiment, the super-resolution circuit module completes super-resolution processing of the input segmented sub-images in parallel by receiving super-resolution image processing instructions that match the target super-resolution processing mode. Different segmentation categories correspond to different super-resolution image processing instructions. The super-resolution circuit module only needs to receive the instructions, and the intelligent image processing module uniformly sends the corresponding super-resolution image processing instructions to each super-resolution circuit module to work together to further improve the efficiency of super-resolution image processing.
[0125] In one embodiment, the parallel super-resolution processing module 906 is also used to, if the segmented sub-image is a foreground subject category, then the complexity of the super-resolution processing mode matched with the foreground subject category obtained by the super-resolution circuit module is proportional to the importance of the foreground subject corresponding to the segmented sub-image in the image to be processed; if the segmented sub-image is a background category, then the complexity of the super-resolution processing mode matched with the background category obtained by the super-resolution circuit module is less than the complexity of the super-resolution processing mode corresponding to the foreground subject category.
[0126] In this embodiment, the complexity of the super-resolution processing mode corresponding to images of different semantic categories is different, and the higher the importance of the foreground subject in the image to be processed, the higher the complexity of the corresponding super-resolution processing mode. The complexity of the super-resolution processing mode of the background category is less than the complexity of the super-resolution processing mode corresponding to the foreground subject category, so that image content with high importance adopts a super-resolution processing mode with high complexity to ensure processing accuracy, further improving the super-resolution processing effect of each foreground subject, and image content with low importance adopts a super-resolution processing mode with low complexity to ensure super-resolution processing efficiency.
[0127] In one embodiment, the parallel super-resolution processing module 906 is also used to ensure that if the segmented sub-image is obtained by dividing the image to be processed into equal parts, the complexity of the super-resolution processing mode matched with the segmentation category of the input segmented sub-image obtained by each super-resolution circuit module is the same.
[0128] In this embodiment, a super-resolution processing mode of the same complexity is used for the segmented sub-images obtained by equally segmenting the image. Since a single-frame image is generally segmented into equally divided images, the use of a super-resolution processing mode of the same complexity can ensure the integrity of the entire image processing.
[0129] In one embodiment, the apparatus further comprises:
[0130] The display module 912 is used to display the target super-resolution image on a display screen.
[0131] In this embodiment, the target super-resolution image obtained by processing is displayed on a display screen, so that the image after super-resolution processing is easy to watch.
[0132] For the specific definition of the image super-resolution processing device, please refer to the definition of the image super-resolution processing method above, which will not be repeated here. Each module in the above-mentioned image super-resolution processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0133] Fig.10 FIG. 1 is a schematic diagram of the internal structure of an electronic device in one embodiment. Fig.10 As shown, the electronic device includes a processor, a memory and a camera module connected via a system bus. Among them, the processor is used to provide computing and control capabilities to support the operation of the entire electronic device, and the processor may include a super-resolution processing chip. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement the image super-resolution processing method provided in each of the above embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium. The camera module is used to provide a video acquisition function. The electronic device can be a mobile phone, a TV, a wearable device, a tablet computer, a laptop computer, etc.
[0134] The implementation of each module in the image super-resolution processing device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, the steps of the method described in the embodiment of the present application are implemented.
[0135] The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the image super-resolution processing method.
[0136] A computer program product comprising instructions, when running on a computer, enables the computer to perform an image super-resolution processing method.
[0137] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0138] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An image super-resolution processing method, applied to a super-resolution processing chip, It is characterized in that include: Get the image to be processed; Determining a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segmenting the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images; the image complexity is used to describe the complexity of the image content; Inputting the segmented sub-images into at least two super-resolution circuit modules respectively, and the super-resolution circuit modules perform parallel super-resolution processing on the at least two segmented sub-images; Each super-resolution circuit module respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image; Performing super-resolution processing on the segmented sub-images according to a super-resolution processing mode that matches the segmentation category of the segmented sub-images to obtain super-resolution segmented sub-images corresponding to the segmented sub-images; The super-resolution segmented sub-images are synthesized to obtain a target super-resolution image corresponding to the image to be processed.
2. The method according to claim 1, It is characterized in that Determining a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segmenting the image to be processed according to the target segmentation method to obtain corresponding at least two segmented sub-images includes: If the image to be processed is determined to be a single-frame image based on its image complexity, the image to be processed is divided into equal parts to obtain at least two divided sub-images of the same area, and the image complexity corresponding to the single-frame image is less than a first preset threshold.
3. The method according to claim 2, It is characterized in that Determining a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segmenting the image to be processed according to the target segmentation method to obtain corresponding at least two segmented sub-images includes: If the image to be processed is determined to be a non-single-frame image based on its image complexity, scene semantic segmentation is performed on the image to be processed to obtain at least two segmented sub-images corresponding to different semantic categories, wherein the image complexity corresponding to the single-frame image is less than the image complexity corresponding to the non-single-frame image.
4. The method according to claim 1, It is characterized in that Before obtaining the image to be processed, the method includes: Acquire a low-resolution first picture set and a corresponding high-resolution second picture set through a cloud database, wherein a first picture in the first picture set has a corresponding second picture with the same content in the second picture set, and the first picture set has first pictures of different semantic categories; The cloud database simulates the target super-resolution processing mode of each first image from low-resolution super-resolution to high-resolution second image by performing algorithm fitting on the difference between the first image and the second image with the same content, and obtains the target super-resolution processing mode corresponding to images of different semantic categories; Different semantic categories and corresponding target super-resolution processing modes are burned into the intelligent image processing module of the super-resolution processing chip.
5. The method according to claim 4, It is characterized in that The segmentation category is one of the semantic categories, and each of the super-resolution circuit modules respectively obtains a super-resolution processing mode that matches the segmentation category of the input segmented sub-image, and performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode that matches the segmentation category of the segmented sub-image, and obtains a super-resolution segmented sub-image corresponding to each segmented sub-image, including: Each super-resolution circuit module receives a super-resolution image processing instruction matching a target super-resolution processing mode transmitted by the intelligent image processing module, and performs super-resolution processing on the segmented sub-images according to the super-resolution image processing instruction to obtain a super-resolution segmented sub-image corresponding to each segmented sub-image; The target super-resolution processing mode is determined by the intelligent image processing module according to the segmentation category of the segmented sub-image input by the super-resolution circuit module.
6. The method according to claim 1, It is characterized in that The super-resolution circuit modules respectively obtain super-resolution processing modes that match the segmentation categories of the input segmented sub-images, including: If the segmented sub-image is of the foreground subject category, the complexity of the super-resolution processing mode matched with the foreground subject category acquired by the super-resolution circuit module is proportional to the importance of the foreground subject corresponding to the segmented sub-image in the image to be processed; If the segmented sub-image is of the background category, the complexity of the super-resolution processing mode matched with the background category acquired by the super-resolution circuit module is less than the complexity of the super-resolution processing mode corresponding to the foreground subject category.
7. The method according to claim 2, It is characterized in that The super-resolution circuit modules respectively obtain super-resolution processing modes that match the segmentation categories of the input segmented sub-images, including: If the segmented sub-image is obtained by dividing the image to be processed into equal parts, the complexity of the super-resolution processing modes matched with the segmentation category of the input segmented sub-image obtained by each super-resolution circuit module is the same.
8. The method according to claim 1, It is characterized in that After synthesizing the super-resolution segmented sub-images to obtain the target super-resolution image corresponding to the image to be processed, the method further includes: The target super-resolution image is displayed on a display screen.
9. An image super-resolution processing device, applied to a super-resolution processing chip, It is characterized in that include: An acquisition module, used for acquiring an image to be processed; a segmentation module, used to determine a target segmentation method corresponding to the image to be processed based on the image complexity of the image to be processed, and segment the image to be processed according to the target segmentation method to obtain at least two corresponding segmented sub-images; the image complexity is used to describe the complexity of the image content; A parallel super-resolution processing module, used for inputting the segmented sub-images into at least two super-resolution circuit modules respectively, the super-resolution circuit modules perform parallel super-resolution processing on the at least two segmented sub-images, each super-resolution circuit module respectively obtains a super-resolution processing mode matching the segmentation category of the input segmented sub-image, and performs super-resolution processing on the segmented sub-image according to the super-resolution processing mode matching the segmentation category of the segmented sub-image, so as to obtain a super-resolution segmented sub-image corresponding to each segmented sub-image; The synthesis module is used to synthesize each super-resolution segmented sub-image to obtain a target super-resolution image corresponding to the image to be processed.
10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.
11. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
12. A computer program product comprising instructions, It is characterized in that When the computer program product is run on a computer, the computer is caused to execute the steps of the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Super-resolution method and device, terminal equipment and storage medium
CN112862681A
