Image resolution adjusting method and electronic device

Abstract

The invention provides an image resolution adjusting method and an electronic device. The method comprises the following steps: running a target application program to obtain a target image with a first resolution based on a first frame rate; selecting a target memory from the plurality of buffer memories according to the utilization rate of each of the plurality of buffer memories during the running of the target application program; processing the target image through an image processing model and a target memory to adjust the resolution of the target image from the first resolution to a second resolution, the second resolution being higher than the first resolution; and controlling the display interface to present the target image with the second resolution based on the first frame rate. Therefore, the user experience and the image processing efficiency of the electronic device can be effectively improved.

Description

Technical Field

[0001] The present invention relates to an image processing technology, and more particularly to an image resolution adjustment method and an electronic device. Background Art

[0002] With the progress of technology, the resolution and frame rate that can be supported by the camera modules of most smartphones, tablet computers or notebook computers have gradually increased. However, for the vast majority of camera modules, if the resolution of the acquired image is increased, the frame rate of the acquired image will inevitably decrease correspondingly. On the contrary, if the frame rate of the acquired image is increased, the resolution of the acquired image will also inevitably decrease correspondingly. Therefore, when switching to the slow motion shooting mode supported by some types of camera modules, since the frame rate of the acquired image is increased, the resolution of the acquired image will also be correspondingly reduced. Subsequently, when the user views the image previously captured in the slow motion shooting mode through a larger screen, the user will significantly feel the degradation of the image quality. Summary of the Invention

[0003] The present invention provides an image resolution adjustment method and an electronic device, which can improve the above problems.

[0004] An embodiment of the present invention provides an image resolution adjustment method, which includes: running a target application program to obtain a target image with a first resolution based on a first frame rate; during the running of the target application program, selecting a target memory from the plurality of buffer memories according to the respective usage rates of the plurality of buffer memories; processing the target image through an image processing model and the target memory to adjust the resolution of the target image from the first resolution to a second resolution, where the second resolution is higher than the first resolution; and controlling a display interface to present the target image with the second resolution based on the first frame rate.

[0005] Another embodiment of the present invention provides an electronic device, which includes an image acquisition interface, a display interface, a plurality of buffer memories and a processor. The processor is connected to the image acquisition interface, the display interface and the plurality of buffer memories. The processor is configured to: run a target application program to instruct the image acquisition interface to obtain a target image with a first resolution based on a first frame rate; during the running of the target application program, select a target memory from the plurality of buffer memories according to the respective usage rates of the plurality of buffer memories; process the target image through an image processing model and the target memory to adjust the resolution of the target image from the first resolution to a second resolution, where the second resolution is higher than the first resolution; and control the display interface to present the target image with the second resolution based on the first frame rate.

[0006] Based on the above, by running the target application, a target image with a first resolution can be obtained based on a first frame rate. On the other hand, during the running of the target application, the target memory can be selected from multiple buffer memories according to the respective usage rates of the multiple buffer memories. Through the image processing model and the target memory, the target image can be processed to increase the resolution of the target image from the first resolution to a second resolution. Subsequently, the display interface can be controlled to present the target image with the second resolution based on the first frame rate. Thus, the requirement of the user to capture and present high-resolution images based on a higher frame rate can be satisfied on the premise of maximizing the image processing efficiency of the image processing model. Description of the Drawings

[0007] Figure 1 is a schematic diagram of an electronic device shown according to an embodiment of the present invention;

[0008] Figure 2 is a schematic diagram of the workflow of an image processing model shown according to an embodiment of the present invention;

[0009] Figure 3 is a flowchart of an image resolution adjustment method shown according to an embodiment of the present invention. Detailed Embodiments

[0010] Reference will now be made in detail to exemplary embodiments of the present invention. The examples of the exemplary embodiments are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0011] Figure 1 is a schematic diagram of an electronic device shown according to an embodiment of the present invention. Please refer to Figure 1 , the electronic device 10 can be various electronic devices such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a game console or a server that support image acquisition and image processing functions, and the type of the electronic device 10 is not limited thereto.

[0012] The electronic device 10 includes an image acquisition interface 11, a display interface 12, a storage circuit 13, buffer memories 14(1) to 14(n), and a processor 15. The image acquisition interface 11 is used to acquire an external image. For example, the image acquisition interface 11 may include an image acquisition module. This image acquisition module at least includes a lens and a photosensitive element. This image acquisition module (including the lens and the photosensitive element) can be used to capture an external image and generate corresponding image data.

[0013] The display interface 12 is used to display images. For example, the display interface 12 may include a plasma display, a liquid-crystal display (LCD), a thin-film transistor liquid-crystal display (TFT-LCD), an organic light-emitting diode (OLED), an LED display, etc., and the type of the display interface 12 is not limited thereto.

[0014] The storage circuit 13 is used to store data. For example, the storage circuit 13 may include a read-only memory (ROM), a solid-state disk (SSD), a traditional hard disk drive (HDD), a flash memory module, an embedded multimedia card (eMMC), a universal flash storage (UFS) device, or other types of non-volatile storage media.

[0015] The buffer memories 14(1) to 14(n) are used to temporarily store data. For example, the buffer memories 14(1) to 14(n) may include dynamic random access memory (DRAM) or other types of volatile storage media. In addition, the number of the buffer memories 14(1) to 14(n) is multiple (i.e., n is greater than 1).

[0016] The processor 15 is connected to the image acquisition interface 11, the display interface 12, the storage circuit 13, and the buffer memories 14(1) to 14(n). The processor 15 can be used to be responsible for the overall or partial operation of the electronic device 10. For example, the processor 15 may include a Central Processing Unit (CPU), or other programmable general-purpose or special-purpose microprocessors, Digital Signal Processors (DSPs), programmable controllers, Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), or other similar devices, or a combination of these devices. In one embodiment, the processor 15 may further include a Graphic Processing Unit (GPU), a Vision Processing Unit (VPU), a Neural network Processing Unit (NPU), or other processors dedicated to (or facilitating) the execution of image processing or neural network processing.

[0017] In one embodiment, the application program 101 (also referred to as the target application program) may be stored in the storage circuit 13. The application program 101 is used to control the image acquisition interface 11. For example, the processor 15 may perform operations such as parameter setting for the image acquisition interface 11, operation mode selection, and / or starting the image acquisition interface 11 to acquire an external image through the application program 101. For example, the application program 101 may include a camera application program or other types of application programs that can be used to control the image acquisition interface 11.

[0018] In one embodiment, the image processing model 102 may also be stored in the storage circuit 13. The image processing model 102 may include a machine learning model, a deep learning model, and / or other computer operation engines. In addition, the image processing model 102 may adopt neural network architectures such as Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), and / or Convolutional Neural Networks (CNNs), or various operation architectures such as Artificial Neural Networks (ANNs), and the present invention is not limited thereto.

[0019] In one embodiment, the processor 15 may run the application 101 to obtain an image (also referred to as the target image) with a specific resolution (also referred to as the first resolution) based on a certain frame rate (also referred to as the first frame rate). For example, after starting the application 101, the processor 15 may control the image acquisition interface 11 through the application 101. For example, the processor 15 may instruct the image acquisition interface 11 to acquire (e.g., capture) the target image based on the first frame rate and the first resolution through the application 101.

[0020] In one embodiment, the first frame rate may be 120 frames per second (FPS), and the first resolution may be 640×360. In one embodiment, the first frame rate must be at least 120 FPS. In one embodiment, the first frame rate may be or close to the highest frame rate supported by the image acquisition interface 11. In one embodiment, the first frame rate and the first resolution may also be adjusted according to practical requirements.

[0021] In one embodiment, the processor 15 may control the image acquisition interface 11 to operate in a specific shooting mode through the application 101 to acquire (e.g., capture) the target image based on the first frame rate and the first resolution. For example, this specific shooting mode may include a slow-motion shooting (or recording) mode.

[0022] In one embodiment, limited by the hardware limitations of the image acquisition interface 11, in the specific shooting mode (e.g., slow-motion shooting (or recording) mode), since the image acquisition interface 11 acquires the target image based on the first frame rate, the resolution of the target image (i.e., the first resolution) may not be able to reach (or even be far lower than) another resolution (also referred to as the second resolution) supported by the image acquisition interface 11. In one embodiment, the second resolution must at least meet the full high definition (FHD) standard. For example, the second resolution may be at least 1920×1080 or 2560×1440. In one embodiment, the second resolution may be or close to the highest resolution supported by the image acquisition interface 11. In one embodiment, the second resolution may also be adjusted according to practical requirements.

[0023] In one embodiment, during the running of application 101, processor 15 may monitor the usage rate of each of buffer memories 14(1) to 14(n). Processor 15 may select at least one memory (also referred to as the target memory) from buffer memories 14(1) to 14(n) according to the monitored usage rate of each of buffer memories 14(1) to 14(n). For example, the usage rate of each of buffer memories 14(1) to 14(n) may reflect (e.g., be positively correlated with) the frequency at which the buffer memory is used or accessed. For example, if during the running of application 101, the usage rate of a certain buffer memory among buffer memories 14(1) to 14(n) is relatively high, it means that during the running of application 101, this buffer memory is used or accessed more frequently (e.g., more often) than other buffer memories.

[0024] In one embodiment, processor 15 may compare the usage rate of each of buffer memories 14(1) to 14(n) during the running of application 101. Then, processor 15 may select at least one buffer memory from buffer memories 14(1) to 14(n) as the target memory according to the comparison result. In one embodiment, the target memory may be the buffer memory with the highest or relatively high usage rate among buffer memories 14(1) to 14(n) during the running of application 101.

[0025] In one embodiment, after selecting the target memory, processor 15 may process the target image through image processing model 102 and the target memory to adjust (e.g., increase) the resolution of the target image from a first resolution to a second resolution. In particular, the second resolution may be higher than the first resolution. For example, assuming the first resolution is 640×360, the second resolution may be 1920×1080 or 2560×1440, and the present invention is not limited thereto.

[0026] In one embodiment, during the process of processing the target image, image processing model 102 may increase the resolution of each image frame in the target image by 4 times (or other multiples) to increase the resolution of the target image from the first resolution to the second resolution. In one embodiment, during the process of processing the target image, image processing model 102 may also perform various image processing means such as pixel value interpolation and / or edge smoothing on the target image after the resolution is increased to improve the image quality of the target image after the resolution is increased.

[0027] In one embodiment, after selecting the target memory, the processor 15 may configure one or more reserved spaces in the target memory. This reserved space can be dedicated to storing data used by the image processing model 102 during the process of processing the target image. For example, the processor 15 may configure this reserved space to be dedicated to staging various (transient) data such as tensors, gradients, biases, and / or weightings used, generated, and / or updated by the image processing model 102 during the process of processing the target image to assist in adjusting the resolution of the target image from the first resolution to the second resolution. Subsequently, during the process of the image processing model 102 processing the target image, the image processing model 102 can quickly write, read, and / or update the above-mentioned various (transient) data by accessing the reserved space in the target memory.

[0028] In one embodiment, in the target memory, the total number of reserved spaces configured by the processor 15 can correspond or match the types of data used by the image processing model 102 during the process of processing the target image. For example, in one embodiment, assuming that the data used by the image processing model 102 during the process of processing the target image includes the above 4 types of data (i.e., tensors, gradients, biases, and weightings), the total number of reserved spaces configured in the target memory can be 4. Thus, subsequently, during the process of the image processing model 102 processing the target image, each reserved space in the target memory can be dedicated to storing the specific type of data corresponding to that reserved space. In addition, the total number of reserved spaces configured in the target memory can also be increased or decreased according to practical requirements, and the present invention does not limit this.

[0029] In one embodiment, the target memory is the buffer memory with the highest or relatively high usage rate among the buffer memories 14(1) to 14(n) during the running of the application program 101. Therefore, subsequently, during the process of the image processing model 102 processing the target image, it can be ensured as much as possible that the buffer memory accessed by the application program 101 and the image processing model 102 is the same buffer memory among the buffer memories 14(1) to 14(n), thereby reducing the occurrence of data replication across buffer memories (data replication across buffer memories will reduce the working efficiency of the image processing model 102). Thus, the processing efficiency of the image processing model 102 for the target image can be effectively improved.

[0030] In one embodiment, compared with randomly using any buffer memory by the image processing model 102 during the execution of image processing, by selecting the target memory in the above manner and configuring a reserved space in the target memory for exclusive access by the image processing model 102, the working efficiency of the image processing model 102 can be increased by about 20% or even higher.

[0031] Figure 2 It is a schematic diagram of the workflow of an image processing model shown in an embodiment of the present invention. Please refer to Figure 2 , after obtaining a target image with a first resolution (i.e., a low-resolution image), the application 101 can transmit this target image with the first resolution to the pre-processing interface 21. The pre-processing interface 21 can be responsible for inputting the image to be processed (i.e., the target image with the first resolution) and a parameter (such as upscaling factor) used to describe the expected magnification of the target image into the image processing model 102. The image processing model 102 can process the target image by matching the target memory 23 and the operation core 24 according to the received target image and the magnification, such as performing resolution magnification on the target image.

[0032] It should be noted that, in the Figure 2 embodiment, the buffer memory 14(i) can be set as the target memory 23. For example, the buffer memory 14(i) can be the buffer memory with the highest or relatively high usage rate among the buffer memories 14(1) to 14(n) during the running of the application 101. Then, during the process of processing the target image, the image processing model 102 can access the target memory 23 through the parameter exchange interface 22 to update various data used by the image processing model 102 during the process of processing the target image.

[0033] On the other hand, during the process of processing the target image, the hardware device where the operation core 24 is located (such as GPU, VPU, and / or NPU, etc.) can also access the target memory 23 to assist the image processing model 102 in performing related neural network operations (such as parameter update), so as to generate a target image with a second resolution (i.e., a high-resolution image). After generating the target image with the second resolution, the operation core 24 can transmit this target image with the second resolution back to the application 101.

[0034] In an embodiment, compared with arbitrarily using the buffer memories 14(1) to 14(n) to temporarily store data during the process of processing the target image, by configuring the buffer memory 14(i) as the target memory 23, the processing efficiency of the image processing model 102 (and the operation core 24) for the target image can be effectively improved.

[0035] In an embodiment, it is assumed that previously, due to the hardware limitation of the image acquisition interface 11, it was only possible to obtain a target image with a relatively low resolution (such as 640×360) based on a relatively high frame rate (such as 120 FPS) in the slow-motion shooting (or video recording) mode provided by the application 101. After a very fast resolution magnification program (such as Figure 2After the target image is as shown, without changing the frame rate of the target image, the resolution of this target image can be significantly increased (for example, increased to 1920×1080 or 2560×1440).

[0036] Please return to Figure 1 , in an embodiment, after generating a target image with a second resolution, the processor 15 can control the display interface 12 to present the target image with the second resolution based on the first frame rate. For example, in the slow-motion playback mode provided by the application 101, the processor 15 can control the display interface 12 to present the target image with the second resolution based on the first frame rate.

[0037] In an embodiment, compared with the traditional situation where only low-resolution images (such as images with a resolution of 640×360) can be presented based on a relatively high frame rate (such as 120 FPS) in slow-motion video recording and playback modes due to the hardware limitations of the image acquisition interface 11, in the embodiments of the present invention, in the slow-motion playback mode, the display interface 12 can still present high-resolution images (such as images with a resolution of 1920×1080 or 2560×1440) based on the aforementioned relatively high frame rate (such as 120 FPS). Thus, the user experience can be effectively improved.

[0038] Figure 3 is a flowchart of an image resolution adjustment method shown according to an embodiment of the present invention. Please refer to Figure 3 , in step S301, run the target application to obtain a target image with a first resolution based on the first frame rate. In step S302, during the running of the target application, select a target memory from the multiple buffer memories according to the respective usage rates of the multiple buffer memories. In step S303, process the target image through an image processing model and the target memory to adjust the resolution of the target image from the first resolution to a second resolution, where the second resolution is higher than the first resolution. In step S304, control the display interface to present the target image with the second resolution based on the first frame rate.

[0039] However, Figure 3 the steps in Figure 3 have been described in detail above and will not be elaborated here. It should be noted that Figure 3 the steps in

[0040] In summary, the image resolution adjustment method and electronic device proposed in the embodiments of the present invention can break through the hardware limitations of the image acquisition interface. In a specific shooting mode (such as the slow motion video mode), the resolution of the low-resolution target image obtained based on a high frame rate can be quickly increased, and the target image after the resolution is increased can be immediately played based on the high frame rate (such as slow motion playback). Thus, the user experience and the image processing efficiency of the electronic device can be effectively improved.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image resolution adjustment method, characterized in that, Comprising: Running a target application to obtain a target image with a first resolution based on a first frame rate; During the running of the target application, selecting a target memory from the plurality of buffer memories according to the respective usage rates of the plurality of buffer memories; Processing the target image through an image processing model and the target memory to adjust the resolution of the target image from the first resolution to a second resolution, wherein the second resolution is higher than the first resolution; And Controlling a display interface to present the target image with the second resolution based on the first frame rate.

2. The image resolution adjustment method according to claim 1, wherein the step of selecting the target memory from the plurality of buffer memories according to the respective usage rates of the plurality of buffer memories during the running of the target application comprises: Comparing the respective usage rates of the plurality of buffer memories during the running of the target application; And Selecting the target memory from the plurality of buffer memories according to the comparison result.

3. The image resolution adjustment method according to claim 1, wherein the target memory is the buffer memory with the highest usage rate among the plurality of buffer memories during the running of the target application.

4. The image resolution adjustment method according to claim 1, further comprising: After selecting the target memory, configuring a reserved space in the target memory, wherein the reserved space is dedicated to storing data used by the image processing model during the process of processing the target image.

5. The image resolution adjustment method according to claim 1, wherein the step of controlling the display interface to present the target image with the second resolution based on the first frame rate comprises: In a slow motion playback mode, controlling the display interface to present the target image with the second resolution based on the first frame rate.

6. The image resolution adjustment method according to claim 1, wherein the first frame rate reaches at least 120 frames per second, and the second resolution reaches at least the full high definition standard.

7. An electronic device, characterized in that, Comprising: An image acquisition interface; A display interface; A plurality of buffer memories; And A processor, connected to the image acquisition interface, the display interface and the plurality of buffer memories, wherein the processor is configured to: Run a target application to instruct the image acquisition interface to obtain a target image with a first resolution based on a first frame rate; During the running of the target application, select a target memory from the plurality of buffer memories according to the respective usage rates of the plurality of buffer memories; Process the target image through an image processing model and the target memory to adjust the resolution of the target image from the first resolution to a second resolution, wherein the second resolution is higher than the first resolution; And Control the display interface to present the target image with the second resolution based on the first frame rate.

8. The electronic device according to claim 7, wherein during the running of the target application, the operation of the processor to select the target memory from the plurality of buffer memories according to the respective usage rates of the plurality of buffer memories includes: comparing the usage rates of the plurality of buffer memories during the running of the target application; and selecting the target memory from the plurality of buffer memories according to the comparison result.

9. The electronic device according to claim 7, wherein the target memory is the buffer memory with the highest usage rate among the plurality of buffer memories during the running of the target application.

10. The electronic device according to claim 7, wherein the processor is further configured to: after selecting the target memory, configure a reserved space in the target memory, where the reserved space is specifically used to store data used by the image processing model during the processing of the target image.

11. The electronic device according to claim 7, wherein the operation of the processor to control the display interface to present the target image with the second resolution based on the first frame rate includes: in the slow motion playback mode, controlling the display interface to present the target image with the second resolution based on the first frame rate.

12. The electronic device according to claim 7, wherein the first frame rate reaches at least 120 frames per second, and the second resolution reaches at least the full high definition specification.

Images