Super-division model training method and image scaling method

Abstract

The embodiment of the invention provides a super-resolution model training method and an image scaling method, and the method comprises the steps: obtaining a sample image pair which comprises different-resolution images of a same object; performing feature extraction on the second resolution image to obtain a second resolution image feature; determining an image size after the second resolution image is amplified according to the amplification coefficient to obtain a first image size; obtaining position vector information based on the first image size and the amplification coefficient; performing feature extraction on the position vector information to obtain position features; generating a third resolution image according to the second resolution image feature and the position feature; and training a super-resolution model based on the difference between the third resolution image and the first resolution image. Therefore, when the super-division model receives the low-resolution image, the high-resolution image corresponding to the low-resolution image can be output, and the definition of the amplified low-resolution image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a super-resolution model training method and an image scaling method. Background Art

[0002] When previewing an image, if the image is too small, you often need to zoom in on the image to see it more clearly, but after zooming in, the details on the image will become blurred. In addition, when you zoom out, the details of the image will also become blurred. Summary of the invention

[0003] The purpose of the embodiment of the present application is to provide a super-resolution model training method and an image scaling method to solve the problem that the details are not clear after the image is scaled. The specific technical solution is as follows:

[0004] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a super-resolution model training method, the method comprising:

[0005] Acquire a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are images of the same object with different resolutions, and the first resolution is greater than the second resolution;

[0006] Performing feature extraction on the second resolution image to obtain second resolution image features;

[0007] Determine the image size after the second resolution image is enlarged according to the magnification factor to obtain the first image size;

[0008] Obtaining position vector information based on the first image size and the magnification factor, the position vector information being used to describe a mapping relationship between the same point in the object on the magnified second resolution image and on the second resolution image;

[0009] Performing feature extraction on the position vector information to obtain position features;

[0010] generating a third resolution image according to the second resolution image feature and the position feature;

[0011] A super-resolution model is trained based on the difference between the third resolution image and the first resolution image.

[0012] The super-resolution model training method provided in the embodiment of the present application obtains a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are different resolution images of the same object, and the first resolution is greater than the second resolution; performs feature extraction on the second resolution image to obtain second resolution image features; determines the image size of the second resolution image after enlargement according to the magnification factor to obtain the first image size; obtains position vector information based on the first image size and the magnification factor, wherein the position vector information is used to describe the mapping relationship between the same point in the object on the enlarged second resolution image and on the second resolution image; performs feature extraction on the position vector information to obtain position features; generates a third resolution image according to the second resolution image features and the position features; and trains the super-resolution model based on the difference between the third resolution image and the first resolution image. This enables the super-resolution model to output its corresponding high-resolution image when receiving a low-resolution image, thereby improving the clarity of the low-resolution image after enlargement.

[0013] In a possible implementation, acquiring the sample image pair includes:

[0014] Acquire a first resolution image;

[0015] Downsampling the first resolution image to obtain a downsampled first resolution image;

[0016] Noise is added to the downsampled first-resolution image to obtain a second-resolution image.

[0017] In an embodiment of the present application, a second resolution image is obtained by downsampling the first resolution image and adding noise, thereby obtaining a large number of image pairs of different resolutions of the same object, enriching the data set for model training.

[0018] In a possible implementation, the method further includes:

[0019] Select a magnification factor in the preset magnification factor range;

[0020] The step of determining the size of the image after the second resolution image is enlarged according to the enlargement factor to obtain the size of the first image includes:

[0021] The image size after the second resolution image is enlarged is determined according to the currently selected magnification factor to obtain the first image size.

[0022] In an embodiment of the present application, a magnification factor interval is preset, a magnification factor is obtained from the preset magnification factor interval each time, and the image size after the second resolution image is enlarged is determined according to the obtained magnification factor. A super-resolution model is trained based on multiple groups of different magnification factors, so that the trained super-resolution model can achieve image magnification of any magnification factor.

[0023] In a possible implementation manner, the position vector information is in, Indicates the position information of the point after the image is enlarged; Indicates the pixel position information corresponding to the point after the image is enlarged on the second resolution image; Represents the positional relationship between the pixel position information corresponding to the point after image enlargement on the second resolution image and the nearby points of the pixel position information on the second resolution image; H represents the height of the position vector information, W represents the width of the position vector information, and r represents the magnification factor.

[0024] In a possible implementation manner, generating a third resolution image according to the second resolution image feature and the position feature includes:

[0025] Determining position feature information of a preset pixel point from the position features;

[0026] A third resolution image is generated according to the second resolution image features and the position feature information of the preset pixel points.

[0027] In a possible implementation manner, generating a third resolution image according to the second resolution image features and the position feature information of the preset pixel points includes:

[0028] Determining image feature information corresponding to the preset pixel point in the second resolution image feature;

[0029] Extracting a local image feature with a size of k*k centered on the image feature information from the second resolution image feature;

[0030] According to the size of the local image feature, the position feature information of the preset pixel point is processed into feature information consistent with the size of the local image feature;

[0031] The local image features and the processed position feature information of the preset pixel points are fused to generate a third resolution image.

[0032] In a possible implementation, the training of the super-resolution model based on the difference between the third resolution image and the first resolution image includes:

[0033] Determine a second pixel point corresponding to a first pixel point on the third resolution image at the same position on the first resolution image, and calculate a difference in pixel values ​​between the first pixel point and the second pixel point;

[0034] Determine a difference between the third resolution image and the first resolution image based on a difference between pixel values ​​of the first pixel point and the second pixel point;

[0035] Adjusting parameters of the super-resolution model based on a difference between the third resolution image and the first resolution image;

[0036] Return to the execution step: select an amplification factor in the preset amplification factor range until the first preset end condition is met;

[0037] Other sample image pairs are selected to continue training the super-resolution model until the second preset end condition is met, thereby obtaining a trained super-resolution model.

[0038] In the embodiment of the present application, for the same group of sample image pairs, multiple groups of magnification coefficients are selected to train the super-resolution model, so that when the super-resolution model provided by the present application is used to perform image magnification operations, any scale of magnification can be achieved. In addition, traditional super-resolution models are all based on fixed scaling ratios to achieve fixed scaling. If you want to implement super-resolution algorithms with different scaling factors, you need to deploy multiple different super-resolution models, and select the model according to the needs when calling. The disadvantages of this are that there are many deployed models, resulting in a large memory footprint; second, in the process of transitioning from 1x to 2x or from 2x to 3x, due to the switching of the model, the scaling process will be discontinuous, resulting in a fault phenomenon; the super-resolution model provided in the embodiment of the present application can support continuous scaling of images at any scale, and the details of the enlarged image will not be damaged. Since the model used for the entire scaling is the same, there will be no fault phenomenon caused by model switching.

[0039] In a second aspect, an embodiment of the present application further provides an image scaling method, the image scaling method comprising:

[0040] Get the image to be processed;

[0041] Extracting features of the image to be processed to obtain features of the image to be processed;

[0042] Determine the image size of the image to be processed after it is enlarged according to the magnification factor to obtain a first image size;

[0043] Obtaining position vector information based on the first image size and the magnification factor, the position vector information being used to describe a mapping relationship between the same point in the object on the magnified image to be processed and on the image to be processed;

[0044] Performing feature extraction on the position vector information to obtain position features;

[0045] A processed image is generated according to the features of the image to be processed and the position features.

[0046] In a possible implementation manner, the position vector information is in, Indicates the position information of the point after the image is enlarged; The pixel position information corresponding to the point after the image is enlarged on the image to be processed; Represents the positional relationship between the pixel position information corresponding to the point after image enlargement on the image to be processed and the nearby points of the pixel position information on the image to be processed; H represents the height of the position vector information, W represents the width of the position vector information, and r represents the magnification factor.

[0047] In a possible implementation, generating a processed image according to the to-be-processed image feature and the position feature includes:

[0048] Determining position feature information of a preset pixel point from the position features;

[0049] A processed image is generated according to the features of the image to be processed and the position feature information of the preset pixel points.

[0050] In a possible implementation, generating a processed image according to the features of the image to be processed and the position feature information of the preset pixel points includes:

[0051] Determine the image feature information corresponding to the preset pixel point in the image feature to be processed;

[0052] Extracting a local image feature with a size of k*k and centered on the image feature information from the image feature to be processed;

[0053] According to the size of the local image feature, the position feature information of the preset pixel point is processed into feature information consistent with the size of the local image feature;

[0054] The local image features and the processed position feature information of the preset pixel points are processed and fused to generate a processed image.

[0055] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0056] one or more processors and memory;

[0057] The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute any super-resolution model training method described in the first aspect above.

[0058] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, comprising a computer program, which, when executed on an electronic device, enables the electronic device to execute any of the super-resolution model training methods described in the first aspect.

[0059] In the fifth aspect, an embodiment of the present application further provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device inputs data into the chip system, and executes the image scaling method described in any one of the second aspects above to process the data and output the processing results.

[0060] The super-resolution model training method provided in the embodiment of the present application obtains a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are different resolution images of the same object, and the first resolution is greater than the second resolution; performs feature extraction on the second resolution image to obtain second resolution image features; determines the image size of the second resolution image after enlargement according to the magnification factor to obtain the first image size; obtains position vector information based on the first image size and the magnification factor, wherein the position vector information is used to describe the mapping relationship between the same point in the object on the enlarged second resolution image and on the second resolution image; performs feature extraction on the position vector information to obtain position features; generates a third resolution image according to the second resolution image features and the position features; and trains the super-resolution model based on the difference between the third resolution image and the first resolution image. This enables the super-resolution model to output its corresponding high-resolution image when receiving a low-resolution image, thereby improving the clarity of the low-resolution image after enlargement. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0063] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0064] Figure 3a This is a schematic diagram of the result of magnifying an image 6 times using the existing technology;

[0065] Figure 3b A schematic diagram of the result of enlarging an image 6 times using the image scaling method provided in an embodiment of the present application;

[0066] Figure 4 A schematic diagram of a process flow of a super-resolution model training method provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of a structure of position vector information provided in an embodiment of the present application;

[0068] Figure 6 Another schematic diagram of a process flow of a super-resolution model training method provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of a process for generating an SR image provided in an embodiment of the present application;

[0070] Figure 8 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0072] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first instruction and the second instruction are to distinguish different user instructions, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0073] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0074] The following describes some concepts involved in the image scaling method provided in the embodiments of the present application.

[0075] Super Resolution (SR): Super resolution is a low-level image processing task used to map a low-resolution image to a high-resolution image in order to enhance image details.

[0076] When previewing an image, if the image is too small, in order to see it more clearly, it is often necessary to enlarge the image to view it, but after the image is enlarged, the details on the image will become blurred. In addition, when the image is reduced, the image will also have a problem of blurred details. In order to solve the problem of unclear details after image scaling, the image scaling method provided in the embodiment of the present application is introduced below.

[0077] The embodiment of the present application provides an image scaling method, which can be applied to electronic devices. The electronic device is an electronic device with a picture preview function, such as a smart phone, a tablet computer, a smart watch, etc. The scheme provided by the embodiment of the present application includes: obtaining an image to be processed; performing feature extraction on the image to be processed to obtain features of the image to be processed; determining the image size after the image to be processed is enlarged according to the magnification factor to obtain a first image size; obtaining position vector information based on the first image size and the magnification factor, the position vector information is used to describe the mapping relationship between the same point in the object on the enlarged image to be processed and on the image to be processed; performing feature extraction on the position vector information to obtain position features; generating a processed image according to the features of the image to be processed and the position features. Through the above scheme, when scaling an image, the clarity of the image after scaling can be improved.

[0078] The structure of the electronic device to which the above-mentioned image scaling method is applied is introduced below.

[0079] For example, Figure 1 The structure diagram of the electronic device 100 is shown. The electronic device 100 may include a processor 110, a display screen 120, a camera 130, an internal memory 140, a SIM (Subscriber Identification Module) card interface 150, a USB (Universal Serial Bus) interface 160, a charging management module 170, a battery management module 171, a battery 172 with a cell and a battery protection device, a sensor module 180, a mobile communication module 190, a wireless communication module 200, an antenna 1 and an antenna 2, etc. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, etc.

[0080] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 110 may include one or more processing units, for example: the processor 110 may include a CPU (Central Processing Unit), an AP (Application Processor), a modem processor, a GPU (graphics processing unit), an ISP (Image Signal Processor), a controller, a video codec, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-network Processing Unit), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be provided in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 may be a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100 in processing data or executing instructions.

[0082] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an I2C (Inter-Integrated Circuit) interface, an I2S (Inter-Integrated Circuit Sound) interface, a PCM (Pulse Code Modulation) interface, a UART (Universal Asynchronous Receiver / Transmitter) interface, a MIPI interface (Mobile Industry Processor Interface), a GPIO (General-Purpose Input / Output) interface, a SIM card interface and / or a USB interface, etc. Among them, the USB interface 160 is an interface that complies with the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 160 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device. The USB interface 160 may also be used to connect headphones to play audio through the headphones.

[0083] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is for illustrative purposes only and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0084] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 190, the wireless communication module 200, the modem processor, the baseband processor, and the like.

[0085] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0086] The electronic device 100 implements the display function through a GPU, a display screen 120, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0087] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel can be LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), active matrix organic light emitting diode or AMOLED (Active-Matrix Organic Light-Emitting Diode), FLED (Flex Light-Emitting Diode), MiniLed, MicroLed, Micro-oLed, QLED (Quantum Dot Light Emitting Diodes), etc. In some embodiments, the electronic device 100 may include one or more display screens 120.

[0088] In some embodiments of the present application, when the display panel is made of materials such as OLED, AMOLED, FLED, etc., the above Figure 1 The display screen 120 in the embodiment can be bent. Here, the display screen 120 can be bent means that the display screen can be bent to any angle at any position and can be kept at the angle, for example, the display screen 120 can be folded in half from the middle to the left or right. It can also be folded in half from the middle to the top or bottom.

[0089] The display screen 120 of the electronic device 100 can be a flexible screen. At present, flexible screens have attracted much attention for their unique characteristics and huge potential. Compared with traditional screens, flexible screens have the characteristics of strong flexibility and bendability, which can provide users with a new interaction method based on bendable characteristics, and can meet users' more needs for electronic devices. For electronic devices equipped with a foldable display, the foldable display on the electronic device can be switched between a small screen in a folded form and a large screen in an unfolded form at any time. Therefore, users use the split-screen function on electronic devices equipped with a foldable display more and more frequently.

[0090] The electronic device 100 can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 120 and the application processor, etc., wherein the camera 130 includes a front camera and a rear camera.

[0091] The ISP is used to process the data fed back by the camera 130. For example, when shooting, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene and other parameters. In some embodiments, the ISP can be set in the camera 130.

[0092] The camera 130 is used to take photos or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a CCD (Charge Coupled Cevice) or a CMOS (Complementary Metal-Oxide-Semiconductor) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB (Red Green Blue), YUV (a color encoding method) or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0093] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0094] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as MPEG (Moving Picture Experts Group) 1, MPEG2, MPEG3, and MPEG4.

[0095] NPU is a neural network computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0096] The internal memory 140 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the above instructions stored in the internal memory 140, so that the electronic device 100 performs the image scaling method provided in some embodiments of the present application, as well as various applications and data processing. The internal memory 140 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 140 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, general flash memory, etc. In some embodiments, the processor 110 can execute the image scaling method provided in the embodiment of the present application, as well as other applications and data processing by running the instructions stored in the internal memory 140 and / or the instructions stored in the memory provided in the processor 110.

[0097] The internal memory 140 can be used to store the relevant programs of the image scaling method provided in the embodiment of the present application, and the processor 110 can be used to call the relevant programs of the image scaling method stored in the internal memory 140 when displaying information, and execute the image scaling method of the embodiment of the present application.

[0098] The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, and the like.

[0099] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 120. There are many types of pressure sensors 180A, for example, they can be resistive pressure sensors, inductive pressure sensors or capacitive pressure sensors. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When the force acts on the pressure sensor 180A, the capacitance between the electrodes changes, and the electronic device 100 determines the intensity of the pressure according to the change in capacitance. When the touch operation acts on the display screen 120, the electronic device 100 detects the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on a short message application icon, an instruction to view the short message is executed; when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0100] The fingerprint sensor 180B is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0101] The touch sensor 180C is also called a touch control device. The touch sensor 180C can be arranged on the display screen 120, and the touch sensor 180C and the display screen 120 form a touch screen, which is also called a touch control screen. The touch sensor 180C is used to detect a touch operation acting on or near it. The touch sensor 180C can pass the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation can be provided through the display screen 120. In other embodiments, the touch sensor 180C can also be arranged on the surface of the electronic device 100, and is arranged at a different position from the display screen 120.

[0102] The ambient light sensor 180D is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 120 according to the perceived brightness of the ambient light. The ambient light sensor 180D can also be used to automatically adjust the white balance when shooting. The ambient light sensor 180D can also transmit the environmental information of the device to the GPU.

[0103] The ambient light sensor 180D is also used to obtain the brightness, light ratio, color temperature, etc. of the environment in which the camera 130 captures images.

[0104] Figure 2A software structure block diagram of an electronic device to which the present application is applicable. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The layered architecture divides the software system of the electronic device into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into five layers, namely, an application layer, an application framework layer, a system operation layer, a HAL (Hardware Abstract Layer) and a kernel layer.

[0105] The application layer may include a camera, a gallery, and third-party applications with camera functions, etc. Exemplarily, the application layer may also include applications such as calls, calendars, maps, navigation, music, videos, and short messages.

[0106] The application framework layer provides application programming interface and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0107] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, a super-resolution model, and the like.

[0108] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0109] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0110] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0111] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0112] The notification manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, notification information is used to inform the completion of downloads, message reminders, etc. Notification information can also appear in the system top status bar in the form of icons or scroll bar text, such as notifications of applications running in the background, or in the form of dialog windows on the screen. Notification information can be, for example, a text message in the status bar, a reminder sound, vibration of electronic equipment, flashing indicator lights, etc.

[0113] The camera service is used to call the camera (including the front camera and / or the rear camera) in response to the application's request. Specifically in the embodiment of the present application, the camera service can call the camera to collect images in response to the photo command to obtain the corresponding image frame. For example, it is detected that the user clicks the camera's shooting control, determines that the photo command is received, and calls the camera to collect images.

[0114] For the captured images, they can be saved in the electronic device, and the user can view them by opening the gallery application. When the user zooms in and out on the photos, the super-resolution model of the application framework layer will perform specific processing on the photos to achieve the zooming effect of the photos, specifically, including: extracting features from the photos to obtain photo features; determining the image size after the photo is enlarged according to the magnification factor to obtain a first image size; obtaining position vector information based on the first image size and the magnification factor, and the position vector information is used to describe the mapping relationship between the same points in the object on the enlarged photo and on the photo; extracting features from the position vector information to obtain position features; obtaining the zoomed photo according to the photo features and the position features.

[0115] The system operation layer may include multiple functional modules, such as a surface manager, a media library, a 3D (Three Dimensional) graphics processing library (such as OpenGL ES), a 2D (Two Dimensional) graphics engine, and the like.

[0116] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0117] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264 (a new generation of digital video compression format), MP3 (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3), AAC (Advanced Audio Coding, advanced audio coding), AMR (Adaptive Multi-Rate, adaptive multi-rate), JPG (Joint Photographic Experts Group, an image file format), PNG (Portable Network Graphics, portable network graphics), etc.

[0118] The 3D graphics processing library is used to implement three-dimensional graphics drawing, image rendering, compositing, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0119] HAL is an interface layer between the operating system kernel and the hardware circuit. HAL includes but is not limited to: an image processing module. Among them, the image processing module is used to process the image stream. In the embodiment of the present application, after the camera is started, the image processing module can receive the image stream from the camera and can process the image stream, such as noise reduction processing, brightness processing, etc.

[0120] Exemplarily, when the camera application is running, the user-triggered photo command can be sent to the camera service. On the one hand, the camera service can send the received photo command to the image processing module, so that the image processing module can call the camera driver in the kernel layer according to the received photo command, and the camera driver drives the camera and other hardware devices to respond to the photo command to collect image frames. For example, the camera can pass each frame of the collected image to the image processing module at a certain frame rate, and the image processing module will process it and send it to the display screen for display. On the other hand, after receiving the above-mentioned photo command, the camera service can determine the shooting strategy at this time according to the received photo command, and the shooting strategy sets the specific image processing tasks that need to be performed on the collected image data. For example, in the preview mode, the photo command can set the image processing task 1 in the shooting strategy to realize the face detection function. For another example, if the user turns on the beauty function in the preview mode, the camera service can also set the image processing task 2 in the shooting strategy to realize the beauty function. Then, the camera service can send the determined shooting strategy to the image processing module. When the image processing module receives each frame of image data collected by the camera, it can perform corresponding image processing tasks on the above image data according to the shooting strategy issued by the camera service to obtain each frame of the shooting picture after image processing.

[0121] Subsequently, the image processing module may report each frame of the captured image after image processing to the camera application through the camera service, and the camera application may display each frame of the captured image in the preview area of ​​the display screen, or the camera application may save each frame of the captured image in the electronic device in the form of a photo or video.

[0122] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, image sensor driver, audio driver, etc. The hardware includes at least sensors, display screens, cameras, ISP, etc.

[0123] Understandably, Figure 2 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0124] It is understandable that in order to implement the image scaling method provided in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0125] The following describes an application scenario of the image zooming method provided by the embodiment of the present application. In this scenario, the electronic device is a mobile phone as an example for explanation. When a user opens the gallery application in the mobile phone to view pictures, he can select a picture for zooming. If the zooming scene of the current gallery is set to a maximum zooming of 2 times, when trying to set the maximum zooming of 6 times, the details on the picture will become blurred, such as Figure 3a However, by using the image scaling method of the present application, after the image is enlarged by 6 times, its display effect can be clearly and losslessly displayed, and its true contour information can be accurately restored without blurring of details. Figure 3b .

[0126] The super-resolution model training method provided in the embodiment of the present application is described in detail below through specific examples.

[0127] See also Figure 4 , is a super-resolution model training method provided in an embodiment of the present application, the method comprising:

[0128] S401, obtaining a sample image pair, where the sample image pair includes a first resolution image and a second resolution image.

[0129] The first resolution image and the second resolution image are different resolution images of the same object, and the first resolution is greater than the second resolution. The first resolution image can be a high resolution image (High Resolution, HR), the resolution of the high resolution image is such as 1920*1080, 2560*1440, etc., and the second resolution image can be a low resolution image (Low Resolution, LR), the low resolution is such as 320*240.

[0130] In real life, there are relatively few paired HR images and LR images. Therefore, a large number of image pairs can be created by the following method. The specific process includes:

[0131] Step 1, obtaining a first resolution image;

[0132] Step 2: down-sampling the first resolution image to obtain a down-sampled first resolution image;

[0133] Step three: adding noise to the downsampled first-resolution image to obtain a second-resolution image.

[0134] Downsampling is a signal processing technique that involves converting the original signal into a new signal sequence through a scaling factor, thereby reducing the sampling rate or resolution of the signal. Therefore, the second-resolution image represents a low-resolution image that is scaled by the specified scaling factor to the first-resolution image.

[0135] In an embodiment of the present application, a second resolution image is obtained by downsampling the first resolution image and adding noise, thereby obtaining a large number of image pairs of different resolutions of the same object, enriching the data set for model training.

[0136] S402: Perform feature extraction on the second resolution image to obtain second resolution image features.

[0137] The second resolution image can be feature extracted through Convolutional Neural Networks (CNN). CNN is a feed-forward neural network with deep structure and convolution calculation, and is one of the representative algorithms of deep learning. Common CNN networks include LeNet-5, VGGNet, GoogleNet, ResNet, DenseNet, MobileNet, etc. The main structure of CNN includes: Convolutional Layer, whose main function is to extract features; Max Pooling Layer, whose main function is downsampling without damaging the recognition results; Fully Connected Layer, whose main function is classification.

[0138] The second resolution image is subjected to feature extraction through CNN to obtain second resolution image features.

[0139] S403: Determine the image size of the second-resolution image after it is enlarged according to the magnification coefficient to obtain a first image size.

[0140] The magnification factor may be selected from a preset magnification factor range, for example, 2 times, 0.5 times, etc.

[0141] When the image size of the second resolution image is 40*40*3 and the magnification factor is 2.5 times, the size of the first image is 100*100*3.

[0142] S404: Obtain position vector information based on the first image size and the magnification factor.

[0143] The position vector information is used to describe the mapping relationship between the same point in the object on the enlarged second resolution image and on the second resolution image. The width and height of the position vector information are the width and height of the enlarged image. The position vector information is a vector related to the position and the magnification factor, including the position information of the point after the image is enlarged and the pixel position information corresponding to the point after the image is enlarged on the LR image.

[0144] For an example, see Figure 5 , taking the setting of 6 vector feature numbers as an example, the position vector information can be expressed as Among them, (i,j) represents any point, Indicates the position information of the point after the image is enlarged, H represents the height of the position vector information, and W represents the width of the position vector information; represents the pixel position information corresponding to the point after the image is enlarged on the second resolution image, and r represents the enlargement coefficient; Indicates the positional relationship between the pixel position information corresponding to the point after the image is enlarged on the second resolution image and the nearby points of the pixel position information on the second resolution image. is the floor symbol, It represents the positional relationship between the pixel and the upper left pixel. The size of the position vector information is (outH, outW, N), where outH and outW represent the height and width of the position vector information. That is, the H, W, and N mentioned above are the feature numbers set according to the actual situation. Here, N is 6, which means that the dimension of the position vector information is 6 vector feature numbers. Of course, N can also take other values ​​according to the actual situation.

[0145] S405: extract features from the position vector information to obtain position features.

[0146] The position vector information can be feature extracted through the MLP (Multi-Layer Perception) network, which is the simplest form of an artificial neural network. It is a three-layer network structure, including an input layer, a hidden layer, and an output layer. MLP can contain multiple hidden layers, but the most basic architecture contains only one hidden layer. This network handles complex problems by sharing weights and learning a single feature map.

[0147] S406: Generate a third resolution image according to the second resolution image feature and the position feature.

[0148] Since no pixel exists in isolation but has a certain relationship with several pixels around it, a k*k kernel is taken at the center of the second resolution image feature, and the feature is fused with the convolution layer of the position feature to calculate the fused pixel value, which is the pixel value after the resolution is increased. The pixel values ​​after the resolution is increased together constitute the third resolution image.

[0149] S407: Training a super-resolution model based on the difference between the third resolution image and the first resolution image.

[0150] In an embodiment of the present application, the first resolution image (i.e., HR image) is taken as a reference, and the difference between the third resolution image obtained by calculation and prediction and the first resolution image is adjusted to train the super-resolution model so that the output result of the super-resolution model can be infinitely close to the first resolution image.

[0151] The super-resolution model training method provided in the embodiment of the present application obtains a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are different resolution images of the same object, and the first resolution is greater than the second resolution; performs feature extraction on the second resolution image to obtain second resolution image features; determines the image size of the second resolution image after enlargement according to the magnification factor to obtain the first image size; obtains position vector information based on the first image size and the magnification factor, wherein the position vector information is used to describe the mapping relationship between the same point in the object on the enlarged second resolution image and on the second resolution image; performs feature extraction on the position vector information to obtain position features; generates a third resolution image according to the second resolution image features and the position features; and trains the super-resolution model based on the difference between the third resolution image and the first resolution image. This enables the super-resolution model to output its corresponding high-resolution image when receiving a low-resolution image, thereby improving the clarity of the low-resolution image after enlargement.

[0152] In one example, the method further comprises:

[0153] Select a magnification factor in the preset magnification factor range;

[0154] The above step S403, determining the image size of the second resolution image after being enlarged according to the magnification factor to obtain the first image size, includes:

[0155] The image size after the second resolution image is enlarged is determined according to the currently selected magnification factor to obtain the first image size.

[0156] The preset magnification factor interval may be set according to the hardware conditions of the electronic device itself. For example, the preset magnification factor interval is [1,6]. Selecting a magnification factor in the preset magnification factor interval may be to use 1 as the initial magnification factor and to obtain the magnification factor re-determined each time with a step length of 0.2. It is understandable that the magnification factors obtained in this case are 1, 1.2, 1.4…6.

[0157] In an embodiment of the present application, a magnification factor interval is preset, a magnification factor is obtained from the preset magnification factor interval each time, and the image size after the second resolution image is enlarged is determined according to the obtained magnification factor. A super-resolution model is trained based on multiple groups of different magnification factors, so that the trained super-resolution model can achieve image magnification of any magnification factor.

[0158] In one example, the step S406, generating a third resolution image according to the second resolution image feature and the position feature, includes:

[0159] Determining position feature information of a preset pixel point from the position features;

[0160] A third resolution image is generated according to the second resolution image features and the position feature information of the preset pixel points.

[0161] Since the corresponding pixel features are fused when generating the third resolution image based on the second resolution image features and the position features, it is necessary to first determine the position feature information of the preset pixel points from the position features, and generate the third resolution image based on the second resolution image features and the position feature information of the preset pixel points.

[0162] In one example, the step of generating a third resolution image according to the second resolution image features and the position feature information of the preset pixel points includes:

[0163] Step 1: determining image feature information corresponding to the preset pixel point in the second resolution image feature;

[0164] Assuming that the preset pixel point is (i, j), the corresponding pixel point in the second resolution image feature is (i', j'). Determining the image feature information corresponding to the preset pixel point in the second resolution image feature is to determine the image feature information of the pixel point (i', j').

[0165] Step 2: extracting a local image feature with a size of k*k and centered on the image feature information from the second resolution image feature;

[0166] Considering that a single center point will lead to information loss, the local image features of the center point k*k are taken here.

[0167] Step three, according to the size of the local image feature, processing the position feature information of the preset pixel point into feature information consistent with the size of the local image feature;

[0168] In order to achieve the fusion of image features and position features, it is necessary to process the position feature information of the preset pixel points into feature information with the same size as the local image features.

[0169] Step 4: Fusion the local image features with the processed position feature information of the preset pixel points to generate a third resolution image.

[0170] In one example, the step S407, training a super-resolution model based on the difference between the third resolution image and the first resolution image, includes:

[0171] Determine a second pixel point corresponding to a first pixel point on the third resolution image at the same position on the first resolution image, and calculate a difference in pixel values ​​between the first pixel point and the second pixel point;

[0172] Determine a difference between the third resolution image and the first resolution image based on a difference between pixel values ​​of the first pixel point and the second pixel point;

[0173] Adjusting parameters of the super-resolution model based on a difference between the third resolution image and the first resolution image;

[0174] That is, the difference between the third resolution image and the first resolution image can be determined based on the difference in pixel values ​​between the pixels. Exemplarily, based on each pixel pair (i.e., the aforementioned first pixel and the corresponding second pixel), the difference in pixel values ​​between the first pixel and the second pixel is obtained, all the obtained differences are squared and summed, and then divided by the number of pixel pairs, and the obtained value is used as the difference. Based on the difference, the parameters of the super-resolution model are adjusted to find the parameters of the super-resolution model that minimizes the difference.

[0175] After adjusting the parameters of the super-resolution model once based on the current magnification factor, returning to the execution step: selecting a magnification factor in the preset magnification factor interval until the first preset end condition is met;

[0176] When the preset magnification factor interval is [1,6], a magnification factor selected in the preset magnification factor interval can be 1 times as the initial magnification factor, and the magnification factor re-determined each time can be obtained with a step size of 0.2 times. It can be understood that the magnification factors obtained in this case are 1, 1.2, 1.4...6. The first preset end condition is when the magnification factor takes 6 times the upper limit of the preset magnification factor interval.

[0177] Based on the current sample image pair, a magnification factor is selected from a preset magnification factor range until a first preset end condition is satisfied, and then other sample image pairs are selected to continue training the super-resolution model until a second preset end condition is satisfied, thereby obtaining a trained super-resolution model. Exemplarily, the second preset end condition may be that the number of training times of the super-resolution model reaches a threshold, or that the difference between the third resolution image predicted by the super-resolution model and the first resolution image is less than a threshold.

[0178] In the embodiment of the present application, for the same group of sample image pairs, multiple groups of magnification coefficients are selected to train the super-resolution model, so that when the super-resolution model provided by the present application is used to perform image magnification operations, any scale of magnification can be achieved. In addition, traditional super-resolution models are all based on fixed scaling ratios to achieve fixed scaling. If you want to implement super-resolution algorithms with different scaling factors, you need to deploy multiple different super-resolution models, and select the model according to the needs when calling. The disadvantages of this are that there are many deployed models, resulting in a large memory footprint; second, in the process of transitioning from 1x to 2x or from 2x to 3x, due to the switching of the model, the scaling process will be discontinuous, resulting in a fault phenomenon; the super-resolution model provided in the embodiment of the present application can support continuous scaling of images at any scale, and the details of the enlarged image will not be damaged. Since the model used for the entire scaling is the same, there will be no fault phenomenon caused by model switching.

[0179] See also Figure 6 , which is a flow chart of the super-resolution model training method provided in an embodiment of the present application. The network structure of the entire model includes three parts: input source, feature extraction, and super-resolution generation. The input source includes the position vector information of the low-resolution image LR to be super-resolution and the high-resolution image SR to be generated. The position vector information is obtained by amplifying the LR image according to the magnification factor r. The size of the LR image is (inH, inW, inC), inH, inW represent the height and width of the LR image, inC represents the number of channels of the LR image, for RGB images, the number of channels is 3, and for grayscale images, the number of channels is 1. The size of the position vector information is (outH, outW, N), outH, outW represent the height and width of the position vector information, N is the number of features set according to actual conditions, and here N is 6 for illustration, that is, the dimension of the position vector information is 6 vector features. Then, the deep features of the LR image are extracted through the CNN network to obtain the feature layer F LR , whose size is (inH, inW, inC), and the position feature layer representing the SR position vector is obtained through the MLP network (including two fully connected layers and activation functions) Its size is (outH, outW, inC*outC*k*k), where inC is the same as outC. This is the content of the feature extraction network. Finally, the super-resolution generation part is included in the feature layer F LR Take a k*k kernel at the center position of the position feature layer The convolutional layer is used to perform feature fusion and calculate the pixel value after super-resolution. The size of the final high-resolution image SR is (outH, outW, outC).

[0180] See also Figure 7, the specific implementation process of generating the high-resolution image SR is further explained.

[0181] First, from the position feature layer Take out the vector information of position (i, j), whose size is (1, 1, inC*outC*k*k); then according to the feature layer F LR , get (i, j) on the SR image at the feature layer F LR Considering that a single center point will lead to information loss, the local feature information of the center point k*k is taken here, and its size is (1, k, k, inC).

[0182] against (i, j), reshape it from (1, 1, inC*outC*k*k) to (1, k, k, inC, outC), and reshape it from (1, k, k, inC) to (1, k, k, inC, 1) for local feature information. In this way, The feature fusion result of (i, j) and local feature information is (1, outC).

[0183] On the other hand, an embodiment of the present application further provides an image scaling method, the image scaling method comprising:

[0184] Get the image to be processed;

[0185] Extracting features of the image to be processed to obtain features of the image to be processed;

[0186] Determine the image size of the image to be processed after it is enlarged according to the magnification factor to obtain a first image size;

[0187] Obtaining position vector information based on the first image size and the magnification factor, the position vector information being used to describe a mapping relationship between the same point in the object on the magnified image to be processed and on the image to be processed;

[0188] Performing feature extraction on the position vector information to obtain position features;

[0189] A processed image is generated according to the features of the image to be processed and the position features.

[0190] In a possible implementation manner, the position vector information is in, Indicates the position information of the point after the image is enlarged; The pixel position information corresponding to the point after the image is enlarged on the image to be processed; It represents the positional relationship between the pixel position information corresponding to the point after the image is enlarged on the image to be processed and the nearby points of the pixel position information on the image to be processed.

[0191] In a possible implementation, generating a processed image according to the to-be-processed image feature and the position feature includes:

[0192] Determining position feature information of a preset pixel point from the position features;

[0193] A processed image is generated according to the features of the image to be processed and the position feature information of the preset pixel points.

[0194] In a possible implementation, generating a processed image according to the features of the image to be processed and the position feature information of the preset pixel points includes:

[0195] Determine the image feature information corresponding to the preset pixel point in the image feature to be processed;

[0196] Extracting a local image feature with a size of k*k and centered on the image feature information from the image feature to be processed;

[0197] According to the size of the local image feature, the position feature information of the preset pixel point is processed into feature information consistent with the size of the local image feature;

[0198] The local image features and the processed position feature information of the preset pixel points are processed and fused to generate a processed image.

[0199] The present application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute some or all of the steps in the above method embodiments.

[0200] The present application also provides a computer-readable storage medium, including a computer program, which, when executed on an electronic device, enables the electronic device to perform some or all of the steps in the above method embodiment. The above storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0201] like Figure 8As shown, the present application also provides a chip system, which is applied to electronic devices. The chip system includes one or more processors 501. The processor 501 is used to call computer instructions so that the electronic device inputs the data to be processed into the chip system. The chip system processes the data based on the image scaling method provided in the embodiment of the present application and outputs the processing result.

[0202] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0203] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0204] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0205] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software or any combination thereof. The disclosed embodiments can also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, instructions can be distributed over a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to, floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories, random access memories, erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-only memories (Electrically Erasable Programmable Read Only Memory, EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0206] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0207] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0208] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0209] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A super-resolution model training method, characterized in that: The method comprises: Acquire a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are images of the same object with different resolutions, and the first resolution is greater than the second resolution; Performing feature extraction on the second resolution image to obtain second resolution image features; Determine the image size after the second resolution image is enlarged according to the magnification factor to obtain the first image size; Obtaining position vector information based on the first image size and the magnification factor, the position vector information being used to describe a mapping relationship between the same point in the object on the magnified second resolution image and on the second resolution image; Performing feature extraction on the position vector information to obtain position features; generating a third resolution image according to the second resolution image feature and the position feature; A super-resolution model is trained based on the difference between the third resolution image and the first resolution image.

2. The method according to claim 1, characterized in that The obtaining of the sample image pair comprises: Acquire a first resolution image; Downsampling the first resolution image to obtain a downsampled first resolution image; Noise is added to the downsampled first-resolution image to obtain a second-resolution image.

3. The method according to claim 1, characterized in that The method further comprises: Select a magnification factor in the preset magnification factor range; The step of determining the size of the image after the second resolution image is enlarged according to the enlargement factor to obtain the size of the first image includes: The image size after the second resolution image is enlarged is determined according to the currently selected magnification factor to obtain the first image size.

4. The method according to claim 1, characterized in that: The position vector information is in, Indicates the position information of the point after the image is enlarged; Indicates the pixel position information corresponding to the point after the image is enlarged on the second resolution image; Represents the positional relationship between the pixel position information corresponding to the point after image enlargement on the second resolution image and the nearby points of the pixel position information on the second resolution image; H represents the height of the position vector information, W represents the width of the position vector information, and r represents the magnification factor.

5. The method according to claim 1, characterized in that Generating a third resolution image according to the second resolution image feature and the position feature includes: Determining position feature information of a preset pixel point from the position features; A third resolution image is generated according to the second resolution image features and the position feature information of the preset pixel points.

6. The method according to claim 5, characterized in that The step of generating a third resolution image according to the second resolution image feature and the position feature information of the preset pixel point comprises: Determining image feature information corresponding to the preset pixel point in the second resolution image feature; Extracting a local image feature with a size of k*k centered on the image feature information from the second resolution image feature; According to the size of the local image feature, the position feature information of the preset pixel point is processed into feature information consistent with the size of the local image feature; The local image features and the processed position feature information of the preset pixel points are fused to generate a third resolution image.

7. The method according to claim 3, characterized in that The step of training a super-resolution model based on a difference between the third resolution image and the first resolution image comprises: Determine a second pixel point corresponding to a first pixel point on the third resolution image at the same position on the first resolution image, and calculate a difference in pixel values ​​between the first pixel point and the second pixel point; Determine a difference between the third resolution image and the first resolution image based on a difference between pixel values ​​of the first pixel point and the second pixel point; Adjusting parameters of the super-resolution model based on a difference between the third resolution image and the first resolution image; Return to the execution step: select an amplification factor in the preset amplification factor range until the first preset end condition is met; Other sample image pairs are selected to continue training the super-resolution model until the second preset end condition is met, thereby obtaining a trained super-resolution model.

8. An image scaling method, characterized in that: The image scaling method comprises: Get the image to be processed; Extracting features of the image to be processed to obtain features of the image to be processed; Determine the image size of the image to be processed after it is enlarged according to the magnification factor to obtain a first image size; Obtaining position vector information based on the first image size and the magnification factor, the position vector information being used to describe a mapping relationship between the same point in the object on the magnified image to be processed and on the image to be processed; Performing feature extraction on the position vector information to obtain position features; A processed image is generated according to the features of the image to be processed and the position features.

9. The method according to claim 8, characterized in that The position vector information is in, Indicates the position information of the point after the image is enlarged; The pixel position information corresponding to the point after the image is enlarged on the image to be processed; Represents the positional relationship between the pixel position information corresponding to the point after image enlargement on the image to be processed and the nearby points of the pixel position information on the image to be processed; H represents the height of the position vector information, W represents the width of the position vector information, and r represents the magnification factor.

10. The method according to claim 8, characterized in that The step of generating a processed image according to the image feature to be processed and the position feature comprises: Determining position feature information of a preset pixel point from the position features; A processed image is generated according to the features of the image to be processed and the position feature information of the preset pixel points.

11. The method according to claim 10, characterized in that The step of generating a processed image according to the image features to be processed and the position feature information of the preset pixel points comprises: Determine the image feature information corresponding to the preset pixel point in the image feature to be processed; Extracting a local image feature with a size of k*k and centered on the image feature information from the image feature to be processed; According to the size of the local image feature, the position feature information of the preset pixel point is processed into feature information consistent with the size of the local image feature; The local image features and the processed position feature information of the preset pixel points are processed and fused to generate a processed image.

12. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 7.

13. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when being executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 7.

14. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device inputs data into the chip system, and executes the method described in any one of claims 8-11 to process the data and then output the processing results.

Images