Image processing method and device and storage medium

Abstract

The invention provides an image processing method and device and a storage medium. According to the method, super-resolution processing can be carried out on a shot first image with the zoom magnification larger than or equal to the first zoom magnification to obtain a second image. Compared with the first image, the second image obtained after super-resolution processing has more textural features, so that the definition of the image with the overlarge zoom ratio is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] With the development of science and technology, the shooting effect of electronic devices (such as mobile phones and tablet computers with cameras) is getting better and better, and using electronic devices to shoot has become a common choice for people. At present, electronic devices usually support image zoom when shooting images. Among them, image zoom refers to the electronic device changing the size of the subject in the captured image by changing the zoom ratio when shooting an image. However, when the zoom ratio is too large, the zoomed image may not be clear. Summary of the invention

[0003] The embodiment of the present application provides an image processing method, device and storage medium. The image processing method can perform super-resolution processing on a first image captured with a zoom ratio greater than or equal to a first zoom ratio to obtain a clearer second image.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: encoding a first image to obtain a feature extraction matrix of the first image; the first image is captured by a device having a zoom ratio greater than or equal to the first zoom ratio; dividing the feature extraction matrix into blocks to obtain N first feature matrix blocks, where N is a positive integer greater than 1; performing super-resolution processing on each first feature matrix block to obtain a second feature matrix block corresponding to each first feature matrix block; and processing the second feature matrix block corresponding to each first feature matrix block to obtain a second image.

[0005] Optionally, the second feature matrix block corresponding to each first feature matrix block is processed to obtain a second image, including: splicing the second feature matrix blocks corresponding to each first feature matrix block to obtain a processed feature extraction matrix, and decoding the processed feature extraction matrix to obtain the second image.

[0006] In the method described in this aspect, the electronic device can perform super-resolution processing on a first image captured with a zoom ratio greater than or equal to a first zoom ratio to obtain a second image. The second image obtained after the super-resolution processing has more texture features than the first image, thereby improving the clarity of the image with an excessively large zoom ratio.

[0007] In addition, the embodiment of the present application divides the feature extraction matrix of the first image into blocks and then performs super-resolution processing. In this way, even if the size of the first image is large, it can be divided into blocks and then super-resolution processing can be performed. Therefore, the method described in the embodiment of the present application can be applied to a wide range of image sizes. In addition, the embodiment of the present application divides into blocks and then performs super-resolution processing, which can improve the image quality compared to the solution of scaling the first image and then performing super-resolution processing. At the same time, compared to the solution of setting super-resolution processing for the first image with a larger size and the first image with a smaller size respectively, it can effectively save computing resources.

[0008] In addition, since the embodiment of the present application divides the feature extraction matrix in the latent space into blocks, compared with the solution of directly dividing the first image into blocks, it can avoid the stitching marks between the image blocks.

[0009] In combination with the first aspect, in an optional implementation, each first characteristic matrix block in the N characteristic matrix blocks overlaps with its adjacent first characteristic matrix block; super-resolution processing is performed on the kth first characteristic matrix block to obtain a second characteristic matrix block corresponding to the kth first characteristic matrix block, including: selecting a first characteristic matrix block adjacent to the kth first characteristic matrix block from the 1st to k-1th first characteristic matrix blocks; k is a positive integer greater than 1; based on the overlapping area between the adjacent first characteristic matrix blocks and the kth first characteristic matrix block, selecting a corresponding overlapping area from the second characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks; respectively splicing the overlapping areas of the second characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks with the non-overlapping areas of the kth first characteristic matrix block to obtain the processed kth first characteristic matrix block; super-resolution processing is performed on the processed kth first characteristic matrix block to obtain the second characteristic matrix block corresponding to the kth first characteristic matrix block.

[0010] In this embodiment, the electronic device can perform overlapping block division on the feature extraction matrix, and use the overlapping area of ​​the second feature matrix block corresponding to the first feature matrix block adjacent to the kth first feature matrix block to replace the overlapping area of ​​the kth first feature matrix block, which is conducive to retaining the features of the first feature matrix block adjacent to it after super-resolution processing when super-resolution processing is performed on the kth first feature matrix block, so that the texture features of the image block corresponding to the kth first feature matrix block can be the same as the texture features of the image block corresponding to the first feature matrix block adjacent to it. In this way, even if the size of the first image is large, the second image obtained by splicing after block super-resolution processing has continuous and consistent texture features on adjacent image blocks.

[0011] In combination with the first aspect, in an optional embodiment, the second characteristic matrix block corresponding to each first characteristic matrix block is processed to obtain a second image, including: respectively calculating the mean and standard deviation of each first characteristic matrix block, and the mean and standard deviation of the second characteristic matrix block; respectively inputting the mean and standard deviation of each first characteristic matrix block, the mean and standard deviation of the second characteristic matrix block, and the second characteristic matrix block into the distribution transformation formula to obtain the third characteristic matrix block corresponding to each first characteristic matrix block; processing the third characteristic matrix block corresponding to each first characteristic matrix block to obtain a second image.

[0012] Optionally, the third feature matrix block corresponding to each first feature matrix block is processed to obtain a second image, including: splicing the third feature matrix block corresponding to each first feature matrix block to obtain a processed feature extraction matrix, and decoding the processed feature extraction matrix to obtain the second image.

[0013] In this embodiment, based on the first feature matrix block before super-resolution processing, style transfer can be performed on the second feature matrix block after super-resolution processing to obtain a third feature matrix block. Compared with the second feature matrix block, the third feature matrix block obtained after style transfer represents a color closer to the color represented by the first feature matrix block. In this way, when the style transfer step is performed for each second feature matrix block, the color difference between the third feature matrix blocks obtained after style transfer is small. In this way, the blocking traces caused by the color difference between different image blocks can be effectively avoided, which is conducive to the color of the second image obtained after super-resolution processing being the same as the color of the first image before super-resolution processing.

[0014] In combination with the first aspect, in an optional implementation, each first characteristic matrix block in the N characteristic matrix blocks overlaps with its adjacent first characteristic matrix block; the kth first characteristic matrix block is super-resolution processed to obtain a second characteristic matrix block corresponding to the kth first characteristic matrix block, including: selecting a first characteristic matrix block adjacent to the kth first characteristic matrix block from the 1st to k-1th first characteristic matrix blocks; k is a positive integer greater than 1; based on the overlapping area between the adjacent first characteristic matrix block and the kth first characteristic matrix block, selecting a corresponding overlapping area from the third characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks; respectively splicing the overlapping areas of the third characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks with the non-overlapping areas of the kth first characteristic matrix block to obtain the processed kth first characteristic matrix block; super-resolution processing is performed on the processed kth first characteristic matrix block to obtain the second characteristic matrix block corresponding to the kth first characteristic matrix block.

[0015] In combination with the first aspect, in an optional implementation manner, the distribution transformation formula is:

[0016]

[0017] Among them, X is used to represent the second characteristic matrix block, μ2 is used to represent the mean of the second characteristic matrix block, σ2 is used to represent the standard deviation of the second characteristic matrix block, μ1 is used to represent the mean of the first characteristic matrix block, σ1 is used to represent the standard deviation of the first characteristic matrix block, and output is used to represent the third characteristic matrix block.

[0018] In combination with the first aspect, in an optional embodiment, before encoding the first image and obtaining the feature extraction matrix of the first image, the method also includes: detecting whether the shooting mode of shooting the first image is a super-resolution mode; the super-resolution mode is used to trigger the operation of super-resolution processing performed on the first image shot with a zoom ratio greater than or equal to the first zoom ratio; if the shooting mode is the super-resolution mode, then performing the step of encoding the first image and obtaining the feature extraction matrix of the first image.

[0019] In combination with the first aspect, in an optional implementation, before encoding the first image and obtaining the feature extraction matrix of the first image, the method further includes: displaying a first user interface, the first user interface including a first control; the first control is used to enable the super-resolution mode; when a first operation on the first control is detected, the super-resolution mode is enabled; and the first image is captured. In this implementation, the user can directly enable the super-resolution mode through the first control of the first user interface, which is simpler and more convenient.

[0020] In combination with the first aspect, in an optional implementation, before encoding the first image and obtaining the feature extraction matrix of the first image, the method further includes: displaying a second user interface, the second user interface including a second control; the second control is used to trigger the display of the first control; when a second operation on the second control is detected, displaying a third user interface, the third user interface including the first control; when a first operation on the first control is detected, turning on the super-resolution mode; and shooting the first image. In this implementation, the first control can be triggered to display by the second control, and the user interface is simpler.

[0021] In a second aspect, an embodiment of the present application further provides an electronic device, comprising: a central processing unit, a graphics processor, an image signal processor, and one or more memories; the one or more memories are coupled to the central processing unit, the graphics processor, and the image signal processor, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the central processing unit, the graphics processor, or the image signal processor executes the computer instructions, the electronic device executes the method of the first aspect.

[0022] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, including computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application further provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments of the present application will be described below.

[0025] Figure 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 2 is a software structure block diagram of the electronic device provided in the embodiment of the present application;

[0027] FIG. 3A to FIG. 3E is a schematic diagram of a user interface provided in an embodiment of the present application;

[0028] Figure 4 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0029] Figure 5 It is a flowchart of another image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] Some terms in this application are explained below to facilitate understanding by those skilled in the art.

[0031] The application (Application, App) involved in the embodiments of the present application, which can also be called an application program, is a software program that can realize one or more specific functions. Usually, multiple applications can be installed in an electronic device, for example, instant messaging applications, audio applications, image capture applications, and the like. Among them, instant messaging applications, for example, may include text messaging applications, WeChat (WeChat), DingTalk, and the like. Audio applications, for example, may include Kugou Music, Xiami Music, and the like. Image capture applications, for example, may include camera applications (factory setting system camera or third-party camera application). The applications mentioned in the following embodiments may be applications that are installed on the electronic device when it leaves the factory, or they may be applications that the user downloads from the Internet or obtains from other electronic devices during the use of the electronic device.

[0032] It should be understood that, in this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this application is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more, and "more than one" means two or more.

[0033] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0034] At present, electronic devices generally support image zoom when shooting images. During shooting, the position of the object being photographed relative to the electronic device does not change. If the zoom ratio increases, the field of view (FOV) of the electronic device becomes smaller, and the object being photographed in the image may become larger. However, when the zoom ratio is too large, the zoomed image may not be clear. For example, when digital zoom is used to achieve image zoom, the electronic device needs to crop an image corresponding to the zoom ratio from the original image, and then upsample the cropped image to form an image of the same size as the original image as the zoomed image. The resolution of the cropped image decreases, resulting in an unclear zoomed image. It should be noted that the original image refers to an image taken by the electronic device at a zoom ratio of 1.

[0035] Based on this, an embodiment of the present application provides an image processing method. In this method, a first image captured with a zoom ratio greater than or equal to a first zoom ratio can be super-resolution processed to obtain a second image. The second image has more texture features than the first image, and the second image is clearer than the first image.

[0036] Exemplarily, the image processing method mentioned in the embodiments of the present application can be run in an electronic device, which may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, etc. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0037] like Figure 1 As shown, it is a schematic diagram of a hardware structure of an electronic device. Among them, the electronic device may include: a processor, an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, a sensor module, a button, a motor, an indicator, a camera, a display screen, and a SIM card slot, etc. Among them, the audio module may include a speaker, a receiver, a microphone, an earphone interface, etc., and the sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0038] It is to be understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0039] Among them, the processor may include one or more processing units, for example: the processor may include an application processor (Application Processor, AP), a modem processor (Modem), a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), an image signal processor (Image Signal Processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a neural network processor (neural-network processing unit, NPU), etc. Different processing units can be independent devices or integrated in one or more processors. The processor is the nerve center and command center of the electronic device. For example, the CPU, GPU, and ISP in the electronic device can execute the image processing method of the embodiment of the present application.

[0040] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module, wireless communication module, and Modem, etc. In some embodiments, the antenna 1 of the electronic device is coupled with the mobile communication module, and the antenna 2 is coupled with the wireless communication module, so that the electronic device can communicate with the network side device and other electronic devices through wireless communication technology.

[0041] Among them, the touch sensor is also called a "touch control device". The touch sensor can be set on the display screen, and the touch sensor and the display screen form a touch screen, also called a "touch control screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor can transmit the detected touch operation to the AP to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen. In other embodiments, the touch sensor can also be set on the surface of the electronic device, which is different from the position of the display screen. In an embodiment of the present application, the electronic device can detect whether there is a user operation acting on the display screen of the electronic device through the touch sensor. After the touch sensor detects the user operation acting on the camera application, the electronic device can take an image.

[0042] In addition, operating systems are running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft.

[0043] The operating system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device. It should be noted that although the embodiment of the present application is illustrated by taking the Android system as an example, its basic principle is also applicable to electronic devices based on operating systems such as iOS or Windows.

[0044] See also Figure 2 , is a software structure block diagram of an electronic device provided in an embodiment of the present application. The software structure adopts a layered architecture, which divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. Taking the Android system as an example, in some embodiments, the Android system is divided from top to bottom into an application layer, an application framework layer (Framework), a hardware abstraction layer (HAL), and a kernel layer (Kernel).

[0045] The application layer may include a series of application packages, including camera, gallery, calendar, call, map, WLAN, Bluetooth, music, video, short message and other APPs.

[0046] Among them, the application framework layer provides an application programming interface (Application Programming Interface, API) and a programming framework for the application of the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, etc. The embodiment of the present application does not impose any restrictions on this. Exemplarily, the above-mentioned window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The above-mentioned content provider is used to store and obtain data, and make these data accessible to the application. The data may include video, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The above-mentioned view system can be used to build the display interface of the application. Each display interface can be composed of one or more controls. Generally speaking, controls can include interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. The above-mentioned resource manager provides various resources for the application, such as localized strings, icons, images, layout files, video files, etc. The above notification manager uses an application to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, a vibration is made, an indicator light flashes, etc. In some embodiments, the application framework layer may also include a camera service (Camera Service). The camera service (Camera Service) can interact with the hardware abstraction layer.

[0047] Among them, the hardware abstraction layer is an interface layer located between the kernel layer and the hardware, which can be used to abstract the hardware. In some embodiments, the hardware abstraction layer includes a hardware abstraction layer interface definition language (HIDL) interface. Among them, the hardware abstraction layer may include: camera HAL, display HAL, sensor HAL, etc. In some embodiments, the hardware abstraction layer may include an algorithm library. Optionally, the algorithm library provides one or more algorithm models, for example, at least one of an encoding model, a decoding model, a super-resolution model, and a style transfer model. Among them, the encoding model is used to perform encoding processing, the decoding model is used to perform decoding processing, the super-resolution model is used to perform super-resolution processing, and the style transfer model is used to perform style transfer processing (for example, the style transfer model is used to perform style transfer processing based on a distribution transformation formula).

[0048] The kernel layer is the foundation of the Android operating system, responsible for hardware drivers, network, power, system security, memory management and other functions. The kernel layer is an intermediate layer between hardware and software, and its function is to pass application requests to the hardware. The kernel layer contains at least camera drivers, sensor drivers, display drivers and audio drivers.

[0049] The following is a user interface diagram of an electronic device implementing the image processing method provided in an embodiment of the present application.

[0050] The image processing method provided in the embodiment of the present application can be applied in combination with a photographing method provided in the embodiment of the present application. In the photographing method, the electronic device can provide a zoom function. Figure 3A-3E The user interface of the photographing method involved in the embodiment of the present application is described in detail.

[0051] like Figure 3A As shown, the electronic device can detect the touch operation of the user on the camera application 301 in the user interface 31, and in response to the operation, the electronic device can display the following Figure 3B The user interface 32 shown in FIG. 32 may be a camera application shooting interface, and the electronic device may shoot an image in response to a user operation in the user interface 32. The camera application is an image shooting application on the electronic device, and the application does not limit the name of the application. That is, the user can click on the Figure 3A The camera application 301 shown is used to open the user interface 32 of the camera application. Not limited to this, the user can also open the user interface 32 in other applications. For example, the user can click on the shooting control in an instant messaging application (such as WeChat) to open the user interface 32.

[0052] Figure 3B The user interface 32 of the camera application on the electronic device is shown as an example. Figure 3B As shown, the user interface 32 may include: a preview box 311, a shooting mode list 312, a gallery shortcut key 313, a shutter control 314, a camera flip control 315 and a zoom progress bar 316. Among them:

[0053] The preview frame 311 can be used to display the image captured by the camera of the electronic device in real time. The electronic device can refresh the display content therein in real time so that the user can preview the image currently captured by the camera.

[0054] One or more shooting mode options may be displayed in the shooting mode list 312. The one or more shooting mode options may include, but are not limited to, portrait mode option 312A, video mode option 312B, photo mode option 312C, super-resolution mode option 312D, and more options 312E. The one or more shooting mode options may be displayed as text information on the interface, such as "portrait", "video", "photo", "super-resolution", and "more". Not limited to this, the one or more shooting mode options may also be displayed as icons or other forms of interactive elements (IE) on the interface. Among them, the super-resolution mode option 312D is used to turn on the super-resolution mode. In response to a user operation (such as a click operation) acting on the super-resolution mode option 312D, the electronic device may turn on the super-resolution mode. The super-resolution mode can be understood as that when the electronic device shoots an image in a scene where the zoom ratio is greater than the first zoom ratio, the captured image can be super-resolution processed. That is, the super-resolution mode is used to trigger the operation of performing super-resolution processing on the image captured with a zoom ratio greater than or equal to the first zoom ratio. Optionally, the super-resolution mode option 312D may be folded into the more options 312E, such as Figure 3C 10 is a diagram showing another user interface 33 of a camera application on an electronic device. A shooting mode list 312 of the user interface 33 includes a portrait mode option 312A, a video mode option 312B, a photo mode option 312C, and more options 312E. The electronic device can detect a user's touch operation on the more options 312E in the user interface 33, and in response to the operation, display the following Figure 3D The user interface 34 shown includes a super-resolution mode option 312D.

[0055] The gallery shortcut key 313 can be used to open the gallery application. In response to a user operation (such as a click operation) acting on the gallery shortcut key 313, the electronic device can open the gallery application. In this way, the user can conveniently view the photos and videos taken without having to exit the camera application first and then open the gallery application. The gallery application is an image management application on the electronic device, which can also be called an "album". This embodiment does not limit the name of the application. The gallery application can support users to perform various operations on images stored on the electronic device, such as browsing, editing, deleting, selecting, and the like.

[0056] The shutter control 314 can be used to monitor the user operation that triggers the photo shooting. The electronic device can detect the user operation (such as a click operation) acting on the shutter control 314, and in response to the operation, the electronic device can save the image in the preview box 311 as an image in the gallery application. In addition, the electronic device can also display the thumbnail of the saved image in the gallery shortcut key 313. In other words, the user can click the shutter control 314 to trigger the shooting of the image. Among them, the shutter control 314 can be a button or other forms of controls.

[0057] The camera flip control 315 can be used to monitor the user operation that triggers the flip camera. The electronic device can detect the user operation (such as a click operation) acting on the camera flip control 315, and in response to the operation, the electronic device can flip the camera, such as switching the rear camera to the front camera.

[0058] The zoom progress bar 316 is used to adjust the zoom ratio of the shooting. For example, the zoom progress bar can provide a zoom function with a zoom ratio between 1X and 100X. Optionally, the electronic device can detect a user operation (e.g., a drag operation) acting on the zoom progress bar 316, and in response to the operation, the electronic device adjusts the zoom ratio of the shooting. For example, Figure 3B As shown, the electronic device can detect an upward drag operation on the zoom progress bar 316, and in response to the operation, the electronic device increases the zoom ratio of the shooting to 60X, as shown in FIG. Figure 3E As shown in the user interface 35. Under different zoom magnifications, the size of the photographed object displayed in the preview frame 311 is different, for example, Figure 3B As shown, in the scene where the zoom ratio is 1X, the photographed object displayed in the preview frame 311 is small, such as Figure 3EAs shown, in the scenario where the zoom ratio is 60X, the photographed object displayed in the preview frame 311 is relatively large. Optionally, the user interface 32 may also hide the zoom progress bar 316. When the electronic device detects a user operation (such as a double-click operation or a rectangular frame drawing operation) acting on the user interface 32, the user interface 32 displays the zoom progress bar 316. In other embodiments, the electronic device may also adjust the zoom ratio of the shooting by other user operations, such as a gesture operation of zooming (such as sliding two fingers outward to increase the zoom ratio of the shooting, and sliding two fingers inward to reduce the zoom ratio of the shooting), which is not limited in the present application.

[0059] See also Figure 4 , Figure 4 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 3A-3E The user interface shown and Figure 2 The software architecture of the electronic device shown introduces the process of the image processing method provided in the embodiment of the present application.

[0060] S101, the electronic device starts super-resolution mode.

[0061] refer to Figure 3B In the user interface 32 shown, the electronic device can detect a user operation acting on the super-resolution mode option 312D, and in response to the operation, the electronic device determines to turn on the super-resolution mode. Optionally, the user interface 32 can be called a first user interface, the super-resolution mode option 312D can be called a first control, and the user operation acting on the super-resolution mode option 312D can be called a first operation.

[0062] refer to Figure 3C The user interface 33 shown in FIG. 1 may detect a user operation on more options 312E and, in response to the operation, display the Figure 3D The user interface 34 shown includes a super-resolution mode option 312D. The electronic device can detect a user operation acting on the super-resolution mode option 312D. In response to the operation, the electronic device determines to turn on the super-resolution mode. Optionally, the user interface 33 can be called a second user interface, the more options 312E can be called a second control, the user operation acting on the more options 312E can be called a second operation, and the user interface 34 can be called a third user interface.

[0063] It should be noted that step S101 is an optional step. That is to say, in one scenario, the electronic device may directly perform super-resolution processing on the first image captured with a zoom ratio greater than or equal to the first zoom ratio. In another scenario, if the electronic device introduces a super-resolution mode, the electronic device may perform super-resolution processing on the first image captured with a zoom ratio greater than or equal to the first zoom ratio when the super-resolution mode is turned on, and not perform super-resolution processing on the first image captured with a zoom ratio greater than or equal to the first zoom ratio when the super-resolution mode is not turned on.

[0064] S102: The electronic device adjusts the zoom ratio for shooting to a second zoom ratio.

[0065] refer to Figure 3E The user interface 34 shown in FIG. 1 may detect a user operation on the zoom progress bar 316 of the user interface 32, and in response to the operation, the electronic device adjusts the zoom ratio to a second zoom ratio (eg, Figure 3E 60X shown in FIG.

[0066] S103: The electronic device captures the first image at a second zoom ratio.

[0067] It should be noted that the embodiment of the present application is exemplified by an independent first image frame. In other implementations, the first image may also be any frame of an image in an image sequence. The image sequence refers to an image set composed of multiple frames of images collected continuously and arranged in the order of collection; for example, the image sequence may be a video image.

[0068] Optionally, the embodiment of the present application does not limit the format of the captured first image. For example, the first image may be at least one of an RGB image, a YUV image, or a YCrCb image.

[0069] Optionally, the embodiment of the present application does not limit the size of the captured first image. For example, the first image may be a 4K image or a 1080P image.

[0070] It should be noted that the second zoom ratio in the embodiment of the present application refers to the zoom ratio for capturing the first image.

[0071] S104: If the second zoom ratio is greater than or equal to the first zoom ratio, the electronic device calls a coding model to encode the first image to obtain a feature extraction matrix of the first image.

[0072] The encoding model is located in the system library of the hardware abstraction layer.

[0073] It should be noted that the embodiment of the present application does not limit the value of the first zoom ratio, and the first zoom ratio may be 20X, 40X, 60X, 80X, etc. For the sake of convenience, the first zoom ratio is 40X for example. For example, when the second zoom ratio (60X) for shooting the first image is greater than the first zoom ratio (40X), the electronic device calls the encoding model to encode the first image and obtains a feature extraction matrix of the first image.

[0074] In an optional implementation, the electronic device may encode the first image based on a coding model (or referred to as an "encoder") to obtain a feature extraction matrix (or referred to as "implicit coding") of the first image. Optionally, each element in the feature extraction matrix may be used to represent features such as shape and color of a corresponding area of ​​the first image.

[0075] Optionally, the encoding model involved in the embodiment of the present application has a corresponding decoding model (or referred to as a "decoder"). The encoding model and the decoding model are obtained based on paired training of a deep neural network. The encoding model and the decoding model may exist in an autoencoder model (or a variational autoencoder model), which will not be described in detail.

[0076] S105. The electronic device divides the feature extraction matrix into blocks to obtain N first feature matrix blocks, where N is a positive integer greater than 1.

[0077] For example, the feature extraction matrix is ​​a feature extraction matrix with m rows and n columns, including m*n elements, where m and n are positive integers.

[0078] Exemplarily, the feature extraction matrix =

[0079] In an optional implementation, step S103 specifically includes: dividing the feature extraction matrix into blocks according to the number N of blocks of the first image to obtain N first feature matrix blocks. Optionally, the number of blocks N can be determined based on the size of the first image and the size of the input image of the super-resolution model. For example, when the first image is a 4K image (i.e., the size of the first image is 3840×2160) and the input image of the super-resolution model is a 1080P image (i.e., the size of the image is 1920×1080), the number of blocks N=4.

[0080] For the sake of convenience, the present embodiment uses a feature extraction matrix =

[0081]

[0082] If N=2, the feature extraction matrix can be divided into the following two first feature matrix blocks:

[0083] The first characteristic matrix block =

[0084] The second first characteristic matrix block =

[0085] S106: The electronic device calls the super-resolution model to perform super-resolution processing on each first feature matrix block respectively to obtain a second feature matrix block corresponding to each first feature matrix block.

[0086] Optionally, the dimension of the second feature matrix block is the same as that of the first feature matrix block. For example, when the first feature matrix block is a matrix of p rows and q columns, the second feature matrix block is also a matrix of p rows and q columns, where p and q are positive integers. The second feature matrix block after super-resolution represents more texture features than the first feature matrix block.

[0087] In an optional implementation, the super-resolution model may include an input layer, a processing layer, and an output layer. The input layer is used to splice the first feature matrix block and the preset feature matrix block to obtain an intermediate feature matrix block, the convolution layer is used to convolve the intermediate feature matrix block to obtain a second feature matrix block, and the output layer is used to output the second feature matrix block. The preset feature matrix block is a feature extraction matrix corresponding to a fixed noise image. Optionally, the dimension of the preset feature matrix block is the same as that of the first feature matrix block. For example, when the first feature matrix block is a matrix of p rows and q columns, the preset feature matrix block is also a matrix of p rows and q columns. Optionally, the first feature matrix block and the preset feature matrix block may be spliced ​​on the channel. Optionally, the processing layer may be constructed based on the Unet model without limitation.

[0088] S107: The electronic device processes the second characteristic matrix block corresponding to each first characteristic matrix block to obtain a second image.

[0089] In an optional implementation, the second feature matrix blocks corresponding to each first feature matrix block are spliced ​​to obtain a processed feature extraction matrix, and the processed feature extraction matrix is ​​decoded to obtain a second image. Optionally, the dimension of the processed feature extraction matrix is ​​the same as the dimension of the feature extraction matrix. For example, when the feature extraction matrix is ​​a matrix of s rows and z columns, the processed feature extraction matrix is ​​also a matrix of s rows and z columns, where s and z are positive integers. The processed feature extraction matrix represents more texture features than the feature extraction matrix.

[0090] Optionally, the splicing process of step S107 corresponds to the block processing of step S105. Following the example of step S105, if the second characteristic matrix block corresponding to the first first characteristic matrix block =

[0091] The second characteristic matrix block corresponding to the second first characteristic matrix block =

[0092] Then, the second feature matrix blocks corresponding to the two first feature matrix blocks are concatenated to obtain the processed feature extraction matrix =

[0093] Optionally, the decoding of step S107 corresponds to the encoding of step S104. For example, step S107 performs decoding based on a decoding model, and the decoding model corresponds to the encoding model. Optionally, the decoding model may also be located in an algorithm library of a hardware abstraction layer.

[0094] In another optional implementation, the mean and standard deviation of each first feature matrix block, and the mean and standard deviation of the second feature matrix block are calculated respectively, and the mean and standard deviation of each first feature matrix block, the mean and standard deviation of the second feature matrix block, and the second feature matrix block are input into the distribution transformation formula respectively to obtain the third feature matrix block corresponding to each first feature matrix block; the third feature matrix block corresponding to each first feature matrix block is processed to obtain the second image. Optionally, the distribution transformation formula can be deployed in a style transfer model, which is located in the algorithm library of the hardware abstraction layer.

[0095] Optionally, the above distribution transformation formula may be:

[0096]

[0097] Among them, X is used to represent the second characteristic matrix block, μ2 is used to represent the mean of the second characteristic matrix block, σ2 is used to represent the standard deviation of the second characteristic matrix block, μ1 is used to represent the mean of the first characteristic matrix block, σ1 is used to represent the standard deviation of the first characteristic matrix block, and output is used to represent the third characteristic matrix block.

[0098] Exemplarily, a related embodiment of obtaining the third characteristic matrix block corresponding to the first first characteristic matrix block is described in detail below.

[0099] The mean of the first characteristic matrix block

[0100] Standard deviation of the first characteristic matrix block

[0101] The mean of the second characteristic matrix block corresponding to the first characteristic matrix block

[0102] The standard deviation of the second characteristic matrix block corresponding to the first characteristic matrix block

[0103] The third characteristic matrix block corresponding to the first characteristic matrix block =

[0104] in 1≤i≤3, 1≤j≤3.

[0105] It should be noted that the embodiment of the present application performs style transfer based on the distribution transformation formula. In other implementations, other style transfer methods may also be used, and the present application does not limit this.

[0106] It should be noted that the specific implementation method of processing the third characteristic matrix block corresponding to the first characteristic matrix block to obtain the second image can refer to the relevant embodiment of processing the second characteristic matrix block corresponding to the first characteristic matrix block to obtain the second image, which will not be repeated here.

[0107] In this embodiment, based on the first feature matrix block before super-resolution processing, style transfer can be performed on the second feature matrix block after super-resolution processing to obtain a third feature matrix block. Compared with the second feature matrix block, the third feature matrix block obtained after style transfer represents a color closer to the color represented by the first feature matrix block. In this way, when the style transfer step is performed for each second feature matrix block, the color difference between the third feature matrix blocks obtained after style transfer is small. In this way, the blocking traces caused by the color difference between different image blocks can be effectively avoided, which is conducive to the color of the second image obtained after super-resolution processing being the same as the color of the first image before super-resolution processing.

[0108] In order to better explain the image processing process involved in step S104 to step S107, Figure 5 The flow chart of the process is described in detail. Figure 5 A flowchart of another image processing method provided in an embodiment of the present application is shown below. Figure 5 As shown, the electronic device can encode the first image to obtain a feature extraction matrix of the first image. After the feature extraction matrix is ​​divided into blocks to obtain N first feature matrix blocks, the electronic device can perform super-resolution processing on each first feature matrix block respectively to obtain a second feature matrix block corresponding to each first feature matrix block. Then, style transfer processing is performed on each second feature matrix block respectively to obtain a third feature matrix block corresponding to each first feature matrix block. After splicing the N third feature matrix blocks, a processed feature extraction matrix can be obtained, and the processed feature extraction matrix can be decoded to obtain a second image. For specific implementation, please refer to steps S104 to S107, which will not be repeated here.

[0109] S108: The electronic device stores the second image.

[0110] S109: If the second zoom ratio is smaller than the first zoom ratio, the electronic device stores the first image.

[0111] It should be noted that the above embodiment is described by taking the example of an electronic device performing super-resolution processing on a first image taken by itself with a zoom ratio greater than or equal to a first zoom ratio. In other embodiments, the electronic device can also perform super-resolution processing on a first image taken by other devices with a zoom ratio greater than or equal to the first zoom ratio. For example, the electronic device can obtain the first image from other devices through an instant messaging application, a Bluetooth application, or a device sharing function. When the electronic device detects that the first image is taken by a device with a zoom ratio greater than or equal to the first zoom ratio, the electronic device can execute steps S104 to S108, which will not be described in detail. Optionally, the electronic device can determine the zoom ratio of the device that took the first image based on the focal length of the first image. For example, when the focal length of the first image is 542mm, the zoom ratio of the device that took the first image is 20X; for another example, when the focal length of the first image is 810mm, the zoom ratio of the device that took the first image is 30X.

[0112] It should be noted that before the electronic device performs super-resolution processing on the first image shot by other devices with a zoom ratio greater than or equal to the first zoom ratio, the method further includes: the electronic device detects whether the shooting mode of shooting the first image is a super-resolution mode.

[0113] In an embodiment of the present application, the electronic device can perform super-resolution processing on a first image captured with a zoom ratio greater than or equal to a first zoom ratio to obtain a second image. Compared with the first image, the second image obtained after the super-resolution processing has more texture features, thereby improving the clarity of the image with an excessively large zoom ratio.

[0114] In addition, the embodiment of the present application divides the feature extraction matrix of the first image into blocks and then performs super-resolution processing. In this way, even if the size of the first image is large, it can be divided into blocks and then super-resolution processing can be performed. Therefore, the method described in the embodiment of the present application can be applied to a wide range of image sizes. In addition, the embodiment of the present application divides into blocks and then performs super-resolution processing, which can improve the image quality compared to the solution of scaling the first image and then performing super-resolution processing. At the same time, compared to the solution of setting super-resolution processing for the first image with a larger size and the first image with a smaller size respectively, it can effectively save computing resources.

[0115] In addition, since the embodiment of the present application divides the feature extraction matrix in the latent space into blocks, compared with the solution of directly dividing the first image into blocks, it can avoid the stitching marks between the image blocks.

[0116] In addition, an embodiment of the present application further provides another image processing method, in which step S105 includes: the electronic device divides the feature extraction matrix into blocks according to the number of blocks N of the first image according to a preset overlap ratio to obtain N first feature matrix blocks. Among them, each first feature matrix block in the N first feature matrix blocks overlaps with its adjacent first feature matrix blocks. It should be noted that the preset overlap ratio refers to the overlap ratio between every two adjacent first feature matrix blocks in the N first feature matrix blocks, and the preset overlap ratio can be an overlap ratio input by the user as needed.

[0117] Optionally, the overlap ratio may refer to the ratio between the elements included in the overlap region and the total elements included in the first feature matrix block. Exemplarily, taking the overlap ratio of 50% as an example, the feature extraction matrix may be divided into blocks to obtain two first feature matrix blocks.

[0118] The first characteristic matrix block =

[0119] The second first characteristic matrix block =

[0120] Among them, the first characteristic matrix block includes a 13 、a 14 、a 23 、a 24 、a 33 and a 34 The second first characteristic matrix block also includes a 13 、a 14 、a 23 、a 24 、a 33 and a 34 There is an overlap between the first characteristic matrix block and the second first characteristic matrix block. 13 、a 14 、a 23 、a 24 、a 33 and a 34 The elements are the overlapping elements between the first first characteristic matrix block and the second first characteristic matrix block, or in other words, the overlapping area (solid line rectangular box area) between the first first characteristic matrix block and the second first characteristic matrix block includes a 13 、a 14 、a 23 、a 24 、a 33 and a 34 And other elements.

[0121] Optionally, when the first feature matrix block is located at different positions of the feature extraction matrix, the overlapping area of ​​the first feature matrix block is different. For example, when the first feature matrix block is located on the left side of the feature extraction matrix, the first feature matrix block has an overlapping area with the first feature matrix block located on the right side of the feature extraction matrix; for another example, when the first feature matrix block is located on the upper left side of the feature extraction matrix, the first feature matrix block has an overlapping area with the first feature matrix block located on the upper right side of the feature extraction matrix and the first feature matrix block located on the lower left side of the feature extraction matrix.

[0122] Exemplarily, taking N=4 as an example, the feature extraction matrix may be divided into blocks to obtain four first feature matrix blocks.

[0123] The first characteristic matrix block =

[0124] The second first characteristic matrix block =

[0125] The third first characteristic matrix block =

[0126] The 4th first characteristic matrix block =

[0127] Among them, for the first first characteristic matrix block, there is overlap between the first first characteristic matrix block and the second first characteristic matrix block and the third first characteristic matrix block, and the overlapping area (solid line rectangular frame area) between the first first characteristic matrix block and the second first characteristic matrix block includes a 13 、a 14 、a 23 and a 24 The overlapping area (dotted rectangular frame area) between the first characteristic matrix block and the third characteristic matrix block includes a 21 、a 22 、a 23 and a 24 And other elements.

[0128] For the second first characteristic matrix block, there is overlap between the second first characteristic matrix block and the first first characteristic matrix block and the fourth first characteristic matrix block, and the overlapping area (solid line rectangular frame area) between the second first characteristic matrix block and the first first characteristic matrix block includes a 13 、a 14 、a 23 and a 24 The overlapping area between the second first characteristic matrix block and the fourth first characteristic matrix block (the dotted rectangular frame area) includes a 23 、a 24 、a25 and a 26 And other elements.

[0129] For the third first characteristic matrix block, there is overlap between the third first characteristic matrix block and the first first characteristic matrix block and the fourth first characteristic matrix block. The overlapping area (dotted rectangular frame area) between the third first characteristic matrix block and the first first characteristic matrix block includes a 21 、a 22 、a 23 and a 24 The overlapping area between the third first characteristic matrix block and the fourth first characteristic matrix block (long dashed rectangular frame area) includes a 23 、a 24 、a 33 and a 34 And other elements.

[0130] For the 4th first characteristic matrix block, there is overlap between the 4th first characteristic matrix block and the 2nd first characteristic matrix block and the 3rd first characteristic matrix block, and the overlapping area (dotted line rectangular frame area) between the 4th first characteristic matrix block and the 2nd first characteristic matrix block includes a 23 、a 24 、a 25 and a 26 The overlapping area between the fourth first characteristic matrix block and the third first characteristic matrix block (long dashed rectangular frame area) includes a 23 、a 24 、a 33 and a 34 And other elements.

[0131] Correspondingly, in the image processing method, step S106 includes: super-resolution processing is performed on the kth first characteristic matrix block to obtain a second characteristic matrix block corresponding to the kth first characteristic matrix block. Specifically: from the 1st to k-1th first characteristic matrix blocks, a first characteristic matrix block adjacent to the kth first characteristic matrix block is selected; based on the overlapping area between the adjacent first characteristic matrix block and the kth first characteristic matrix block, the corresponding overlapping area is selected from the second characteristic matrix block corresponding to the adjacent first characteristic matrix block; the overlapping area of ​​the second characteristic matrix block corresponding to the adjacent first characteristic matrix block is respectively spliced ​​with the non-overlapping area of ​​the kth first characteristic matrix block to obtain the processed kth first characteristic matrix block; the processed kth first characteristic matrix block is super-resolution processed to obtain the second characteristic matrix block corresponding to the kth first characteristic matrix block. Wherein, k is a positive integer greater than 1.

[0132] Exemplarily, taking k=2 as an example, for the second first characteristic matrix block, the electronic device can obtain the second characteristic matrix block corresponding to the first first characteristic matrix block adjacent to the second first characteristic matrix block. Then, the electronic device can select the corresponding overlapping area from the second characteristic matrix block corresponding to the first first characteristic matrix block based on the overlapping area between the second first characteristic matrix block and the first first characteristic matrix block.

[0133] Exemplarily, the second characteristic matrix block corresponding to the first first characteristic matrix block= For example, the overlapping area (i.e., the solid line distance box area) corresponding to the second feature matrix block corresponding to the first feature matrix block includes b 13 、b 14 、b 23 and b 24 The overlapping area corresponding to the second characteristic matrix block corresponding to the first first characteristic matrix block is spliced ​​with the non-overlapping area of ​​the second first characteristic matrix block to obtain the processed second first characteristic matrix block.

[0134] The second first feature matrix block after processing =

[0135] Exemplarily, taking k=4 as an example, for the 4th first characteristic matrix block, the electronic device can obtain the second characteristic matrix block corresponding to the 2nd first characteristic matrix block adjacent to the 4th first characteristic matrix block, and the second characteristic matrix block corresponding to the 3rd first characteristic matrix block adjacent to the 4th first characteristic matrix block. Then, the electronic device can select the corresponding overlapping area from the second characteristic matrix block corresponding to the 2nd first characteristic matrix block based on the overlapping area between the 4th first characteristic matrix block and the 2nd first characteristic matrix block. And based on the overlapping area between the 4th first characteristic matrix block and the 3rd first characteristic matrix block, select the corresponding overlapping area from the second characteristic matrix block corresponding to the 3rd first characteristic matrix block.

[0136] Exemplarily, the second characteristic matrix block corresponding to the second first characteristic matrix block= For example, the overlapping area (i.e., the dotted line distance box area) corresponding to the second feature matrix block corresponding to the second first feature matrix block includes b 23 、b 24 、b 25 and b 26 And other elements.

[0137] Exemplarily, the second characteristic matrix block corresponding to the third first characteristic matrix block= For example, the overlapping area (i.e., the long dash distance frame area) corresponding to the second characteristic matrix block corresponding to the third first characteristic matrix block includes b 23、b 24 、b 33 and b 34 And other elements.

[0138] The overlapping area corresponding to the second characteristic matrix block corresponding to the second first characteristic matrix block, the overlapping area corresponding to the second characteristic matrix block corresponding to the third first characteristic matrix block, and the non-overlapping area of ​​the fourth first characteristic matrix block are spliced ​​to obtain the processed fourth first characteristic matrix block.

[0139] The fourth first feature matrix block after processing =

[0140] In an embodiment of the present application, the electronic device can perform overlapping block division on the feature extraction matrix (that is, each of the N first feature matrix blocks overlaps with its adjacent first feature matrix block), and use the overlapping area of ​​the second feature matrix block corresponding to the first feature matrix block adjacent to the kth first feature matrix block to replace the overlapping area of ​​the kth first feature matrix block, which is beneficial for retaining the features of the first feature matrix block adjacent to the kth first feature matrix block after super-resolution processing, thereby facilitating ensuring that the texture features of the image block corresponding to the kth first feature matrix block are the same as the texture features of the image block corresponding to the adjacent first feature matrix block. In this way, even if the size of the first image is large, the second image obtained by block super-resolution processing and then splicing has continuous and consistent texture features on adjacent image blocks.

[0141] It should be noted that when the image processing method involved in the embodiment of the present application includes a style transfer operation, step S106 performs super-resolution processing on the kth first feature matrix block to obtain the second feature matrix block corresponding to the kth first feature matrix block, including: selecting the first feature matrix block adjacent to the kth first feature matrix block from the 1st to k-1th first feature matrix blocks; based on the overlapping area between the adjacent first feature matrix block and the kth first feature matrix block, selecting the corresponding overlapping area from the third feature matrix block corresponding to the adjacent first feature matrix block; respectively splicing the overlapping area of ​​the third feature matrix block corresponding to the adjacent first feature matrix block with the non-overlapping area of ​​the kth first feature matrix block to obtain the processed kth first feature matrix block; super-resolution processing is performed on the processed kth first feature matrix block to obtain the third feature matrix block corresponding to the kth first feature matrix block. Wherein, k is a positive integer greater than 1. No further details are given.

[0142] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0143] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. An image processing method, characterized in that: The method comprises: Encoding a first image to obtain a feature extraction matrix of the first image; the first image is taken by a device having a zoom ratio greater than or equal to a first zoom ratio; Divide the feature extraction matrix into blocks to obtain N first feature matrix blocks, where N is a positive integer greater than 1; Performing super-resolution processing on each first characteristic matrix block respectively to obtain a second characteristic matrix block corresponding to each first characteristic matrix block; The second characteristic matrix block corresponding to each first characteristic matrix block is processed to obtain a second image.

2. The method according to claim 1, characterized in that Each first characteristic matrix block in the N characteristic matrix blocks overlaps with its adjacent first characteristic matrix block; super-resolution is performed on the kth first characteristic matrix block to obtain a second characteristic matrix block corresponding to the kth first characteristic matrix block, including: From the 1st to the k-1th first characteristic matrix blocks, select a first characteristic matrix block adjacent to the kth first characteristic matrix block; k is a positive integer greater than 1; Based on the overlapping area between the adjacent first characteristic matrix blocks and the kth first characteristic matrix block, selecting a corresponding overlapping area from the second characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks; Respectively splicing the overlapping areas of the second characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks with the non-overlapping areas of the k-th first characteristic matrix block to obtain a processed k-th first characteristic matrix block; The processed k-th first characteristic matrix block is subjected to super-resolution processing to obtain a second characteristic matrix block corresponding to the k-th first characteristic matrix block.

3. The method according to claim 1, characterized in that The step of processing the second characteristic matrix block corresponding to each first characteristic matrix block to obtain the second image includes: Respectively calculating the mean and standard deviation of each of the first characteristic matrix blocks, and the mean and standard deviation of the second characteristic matrix blocks; Inputting the mean and standard deviation of each first characteristic matrix block, the mean and standard deviation of the second characteristic matrix block, and the second characteristic matrix block into a distribution transformation formula respectively to obtain a third characteristic matrix block corresponding to each first characteristic matrix block; The third characteristic matrix block corresponding to each first characteristic matrix block is processed to obtain a second image.

4. The method according to claim 3, characterized in that Each first characteristic matrix block in the N characteristic matrix blocks overlaps with its adjacent first characteristic matrix block; super-resolution is performed on the kth first characteristic matrix block to obtain a second characteristic matrix block corresponding to the kth first characteristic matrix block, including: From the 1st to the k-1th first characteristic matrix blocks, select a first characteristic matrix block adjacent to the kth first characteristic matrix block; k is a positive integer greater than 1; Based on the overlapping area between the adjacent first characteristic matrix blocks and the kth first characteristic matrix block, selecting a corresponding overlapping area from the third characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks; Respectively splicing the overlapping areas of the third characteristic matrix blocks corresponding to the adjacent first characteristic matrix blocks with the non-overlapping areas of the k-th first characteristic matrix block to obtain a processed k-th first characteristic matrix block; The processed k-th first characteristic matrix block is subjected to super-resolution processing to obtain a second characteristic matrix block corresponding to the k-th first characteristic matrix block.

5. The method according to claim 3, characterized in that The distribution transformation formula is: Among them, X is used to represent the second characteristic matrix block, used to represent the mean of the second characteristic matrix block, used to represent the standard deviation of the second characteristic matrix block, used to represent the mean of the first characteristic matrix block, used to represent the standard deviation of the first characteristic matrix block, and used to represent the third characteristic matrix block.

6. The method according to any one of claims 1 to 5, characterized in that: Before encoding the first image and obtaining the feature extraction matrix of the first image, the method further includes: Detecting whether the shooting mode for shooting the first image is a super-resolution mode; the super-resolution mode is used to trigger the super-resolution processing operation performed on the first image shot with a zoom ratio greater than or equal to a first zoom ratio; If the shooting mode is the super-resolution mode, the step of encoding the first image to obtain a feature extraction matrix of the first image is performed.

7. The method according to any one of claims 1 to 5, characterized in that: Before encoding the first image and obtaining the feature extraction matrix of the first image, the method further includes: Displaying a first user interface, wherein the first user interface includes a first control; the first control is used to enable super-resolution mode; When a first operation on the first control is detected, starting a super-resolution mode; The first image is captured.

8. The method according to any one of claims 1 to 5, characterized in that Before encoding the first image and obtaining the feature extraction matrix of the first image, the method further includes: Displaying a second user interface, wherein the second user interface includes a second control; the second control is used to trigger the display of the first control; When a second operation on the second control is detected, displaying a third user interface, wherein the third user interface includes the first control; When a first operation on the first control is detected, starting a super-resolution mode; The first image is captured.

9. An electronic device, characterized in that: include: A central processing unit, a graphics processing unit, an image signal processor, and one or more memories; the one or more memories are coupled to the central processing unit, the graphics processing unit, and the image signal processor, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the central processing unit, the graphics processing unit, or the image signal processor executes the computer instructions, the electronic device executes the method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 9.

11. A computer program product, characterized in that When the computer program product is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 8.

Images