A processing method and electronic device

Abstract

The application discloses a processing method and an electronic device, and relates to the technical field of computers. The method disclosed in the application comprises the following steps: obtaining first image data output by an image acquisition device to an operating system through driving of the image acquisition device; performing target processing on the first image data to obtain second image data; the image resolution corresponding to the second image data is higher than the image resolution corresponding to the first image data; and outputting the second image data to the operating system, so that an application program obtains the second image data from the operating system to perform picture output.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a processing method and an electronic device. Background Technology

[0002] Current electronic devices such as PCs (Personal Computers) and all-in-one PCs are limited by the size of the camera module, and generally use low-resolution cameras such as those below 5M. The resolution of the captured images / videos is relatively low. When low-resolution images / videos are enlarged, there will be problems with image quality blurring and distortion. This leads to a fundamental contradiction between the physical limitations of electronic devices and the requirements for image quality. How to solve this problem has become a research hotspot in this field. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] A processing method includes:

[0005] The first image data output by the image acquisition device to the operating system is obtained through the driving of the image acquisition device;

[0006] The first image data is processed to obtain the second image data; the image resolution of the second image data is higher than the image resolution of the first image data.

[0007] The second image data is output to the operating system so that the application can obtain the second image data from the operating system for screen output.

[0008] Optionally, the first image data output by the image acquisition device to the operating system is obtained through the driving of the image acquisition device, including:

[0009] The data interception unit set in the driver of the image acquisition device intercepts the first image data output by the image acquisition device to the first interface in the operating system.

[0010] The first image data includes at least one of the images or video streams acquired by the image acquisition device, and the first interface is used to output image data that can be used by the application layer.

[0011] Optionally, target processing is performed on the first image data, including:

[0012] Super-resolution processing is performed on the first image data based on an image super-resolution model.

[0013] Optionally, performing target processing on the first image data further includes at least one of the following:

[0014] Before performing super-resolution processing on the first image data, the first image data is denoised, so that super-resolution processing is performed on the denoised first image data.

[0015] After performing super-resolution processing on the first image data, image enhancement processing is performed on the super-resolution processed first image data to obtain the second image data.

[0016] Optionally, before performing super-resolution processing on the first image data based on the image super-resolution model, the method further includes:

[0017] Determine the target hardware for running the image super-resolution model and the maximum size of the image super-resolution model that the target hardware can currently support;

[0018] Super-resolution processing is performed on the first image data based on an image super-resolution model, including:

[0019] The target hardware performs super-resolution processing on the first image data based on the maximum-size image super-resolution model.

[0020] Optionally, determining the target hardware for running the image super-resolution model and the maximum size of the image super-resolution model that the target hardware can currently support includes:

[0021] If the electronic device includes a processor capable of running an image super-resolution model, the processor is identified as the target processor for running the image super-resolution model, and the maximum size of the image super-resolution model that the target processor can currently support is determined based on the current load information of the target processor.

[0022] If an electronic device includes multiple processors capable of running image super-resolution models, a target processor for running the image super-resolution model is determined based on the current load information and response rate of each processor through an adjustment strategy, so as to migrate the image super-resolution model from the current processor to the target processor, and the maximum size of the image super-resolution model that the target processor can currently support is determined based on the current load information of the target processor.

[0023] The adjustment strategy is used to adjust at least one of the running position and model size of the image super-resolution model; the priority of the adjustment strategy in adjusting the running position of the image super-resolution model is higher than the priority of the adjustment strategy in adjusting the model size of the image super-resolution model; each processor includes a graphics processor and a neural network processor, and different running positions indicate different processors.

[0024] Optionally, the second image data is output to the operating system so that the application can obtain the second image data from the operating system for screen output, including:

[0025] The second image data is output to the second interface in the operating system so that the application can obtain the second image data output by the second interface based on the corresponding application interface in the operating system;

[0026] The second interface differs from the first interface in that it is used to output image data that can be used by the application layer.

[0027] Optionally, before performing target processing on the first image data, the method further includes:

[0028] Determine the processing mode of the electronic device for image acquisition data;

[0029] In response to the processing mode being the first mode, a step of performing target processing on the first image data is triggered;

[0030] In response to the processing mode being the second mode, the first image data is output to the operating system so that the application can obtain the first image data from the operating system for screen output.

[0031] The first mode is used to improve the resolution of the image output after acquisition, while the second mode maintains the resolution of the image output after acquisition.

[0032] An electronic device, comprising:

[0033] Image acquisition device;

[0034] A processor, which runs an operating system, is used for:

[0035] The first image data output by the image acquisition device to the operating system is obtained through the driving of the image acquisition device;

[0036] The first image data is processed to obtain the second image data; the image resolution of the second image data is higher than the image resolution of the first image data.

[0037] The second image data is output to the operating system so that the application can obtain the second image data from the operating system for screen output.

[0038] Optionally, when the processor obtains the first image data output by the image acquisition device to the operating system, it is configured to: intercept the first image data output by the driver of the image acquisition device to a first interface in the operating system;

[0039] When the processor outputs the second image data to the operating system so that the application can obtain the second image data from the operating system for screen output, it is configured to: output the second image data to a second interface in the operating system so that the application can obtain the second image data output by the second interface based on the corresponding application interface in the operating system;

[0040] The first interface and the second interface are respectively used to output image acquisition data that can be used by the application layer, and the first interface is different from the second interface.

[0041] A storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to perform any of the processing methods provided above. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating one of the processing methods provided in this application;

[0044] Figure 2 This is another flowchart illustrating the processing method provided in this application;

[0045] Figure 3 This is a schematic diagram of the dynamic adaptive dual-path architecture provided in this application;

[0046] Figure 4 This is a flowchart of image acquisition data processing based on the super-resolution processing channel provided in this application;

[0047] Figure 5 This is a schematic diagram of the three-level processing of noise reduction, super-resolution, and post-processing provided in this application;

[0048] Figure 6 This is a flowchart illustrating the processing of image acquisition data in the application example provided in this application;

[0049] Figures 7(a)-7(d) are examples of different super-resolution processing scenarios provided in this application;

[0050] Figure 8 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a processing method and an electronic device for improving the resolution of image data output by an image acquisition device to the system layer without modifying the application (App) or replacing the hardware, thereby improving the image quality of a low-end camera on a high-resolution display.

[0053] The provided method can be applied to electronic devices in a variety of general-purpose or special-purpose computing environments or configurations, such as personal computers, smart terminals, handheld or portable devices, tablet devices, etc. The electronic device has an image acquisition device and a display device (such as a screen) or can be connected to an external display device to display the image data acquired by the image acquisition device.

[0054] See Figure 1 The flowchart shown illustrates the processing method provided in this application embodiment, which may include the following steps 101 to 103, which are described in detail below.

[0055] Step 101: Obtain the first image data output by the image acquisition device to the operating system through the driving of the image acquisition device.

[0056] Image acquisition devices can be, but are not limited to, cameras on electronic devices, such as low-resolution cameras (below 5M) installed on electronic devices, and have image data acquisition capabilities.

[0057] In this embodiment of the application, the first image data output by the image acquisition device to the operating system is intercepted at the hardware driver layer of the electronic device's operating system.

[0058] Optionally, a data interception unit can be set in the driver of the image acquisition device, such as setting a split hook in the camera driver, and intercepting the first image data output by the image acquisition device to the first interface in the operating system based on the set split hook and other data interception units.

[0059] The first image data includes at least one of the images or video streams acquired by the image acquisition device, typically in YUV (Luminance-Chrominance) format. The first interface is the native camera interface in the electronic device system, used to output image data that can be acquired and used by the application layer.

[0060] The operating system pre-allocates corresponding buffer areas in memory for image acquisition devices such as cameras. Image data acquired by the image acquisition device, such as YUV images or video streams, is first output to this buffer area in memory for caching, and then the cached image data is sent to the system layer interface for upper-layer applications to access.

[0061] Based on this, in implementation, the kernel-mode YUV buffer can be directly manipulated through memory mapping to intercept the first image data output by the data acquisition device. Specifically, the address information of the buffer area can be written into the data interception unit, such as the hook code, so that the first image data of the image acquisition device can be directly intercepted from the memory buffer area based on the written address information. After the interception is completed, the first image data is no longer sent to the system layer interface.

[0062] Step 102: Perform target processing on the first image data to obtain second image data; the image resolution corresponding to the second image data is higher than the image resolution corresponding to the first image data.

[0063] The target processing performed on the first image data may include performing super-resolution (super-resolution) processing on the first image data based on an image super-resolution model to improve the resolution of the first image data.

[0064] Image super-resolution models can be large or small AI (Artificial Intelligence) models. The large model can be a general-purpose large model, such as a general-purpose visual large model, a multimodal large model, etc., or it can be an expert large model obtained by fine-tuning a general-purpose large model based on specific needs. This application does not limit this. The large model learns the features and patterns of the input information by training on a large amount of diverse data. It typically has hundreds of millions to trillions of model parameters (such as the weights corresponding to the model network layers, used to control model behavior), and can capture complex relationships and patterns in the input information for accurate and efficient inference / computation.

[0065] Step 103: Output the second image data to the operating system so that the application can obtain the second image data from the operating system for screen output.

[0066] After performing target processing such as super-resolution on the first image data to obtain high-resolution (compared to the original first image data) second image data, the embodiments of this application output the second image data to the operating system. The application can directly obtain the high-resolution second image data from the operating system for screen output or storage and other related applications.

[0067] Optionally, in this step, the second image data can be output to the second interface in the operating system so that the application can obtain the second image data output by the second interface based on the corresponding application programming interface (API) in the operating system.

[0068] The second interface and the first interface are different interfaces, both used to output image data that can be used by the application layer. The first interface is the native camera interface provided by the system, used to receive and output the native image acquisition data from the camera. The second interface is a virtual camera interface added to the system layer in this embodiment, used to receive and output the second image data after super-resolution or other target processing.

[0069] The system layer also includes an application programming interface (API) that can be accessed by applications. This API is connected to the first and second interfaces and is used to output the image data provided by the first / second interfaces to the application layer for applications to acquire and use.

[0070] In summary, the processing method provided in this application obtains first image data output by the image acquisition device to the operating system through the driving of the image acquisition device, performs target processing on the first image data to at least improve the resolution of the first image data, and then outputs the second image data obtained after target processing to the operating system. This allows the application to directly obtain high-resolution second image data from the operating system for screen output. Thus, this application achieves the improvement of the resolution of image data output by the image acquisition device to the system layer without modifying the application or replacing the hardware, resolving the contradiction between the physical limitations and image quality requirements of electronic devices, and effectively improving the image quality of low-end cameras on high-resolution displays.

[0071] In an optional embodiment, the target processing performed on the first image data may further include at least one of the following steps "1-1"-"1-2":

[0072] 1-1: Before performing super-resolution processing on the first image data, the first image data is denoised, so that super-resolution processing can be performed on the denoised first image data.

[0073] The noise reduction processing performed on the first image data may include, but is not limited to, 3D (three-dimensional) noise reduction.

[0074] 3D noise reduction includes temporal, spatial, and color gamut noise reduction. Its key feature is utilizing information from multi-frame image sequences (i.e., the temporal dimension) for noise reduction, identifying and suppressing noise by comparing adjacent frames.

[0075] 1-2: After performing super-resolution processing on the first image data, image enhancement processing is performed on the super-resolution processed first image data, and the enhanced image data is used as the second image data.

[0076] The image enhancement processing may include, but is not limited to, image brightness / color adjustment, image restoration and / or detail refinement, etc., to improve the quality of the super-resolution image.

[0077] For example, the image enhancement processing includes HDR (High Dynamic Range) post-processing, a technique used to achieve a greater dynamic range of exposure (i.e., greater difference between light and dark) than ordinary digital imaging techniques. HDR is used to expand the brightness and color range of an image, making it closer to the visual effects perceived by the human eye in the real world.

[0078] This embodiment improves the quality of image data output to the system layer by denoising the first image data before super-resolution processing and / or performing image enhancement processing after super-resolution processing. This allows for further improvement of the image quality of the screen output by the upper-layer application without modifying the application or replacing the hardware.

[0079] In an optional embodiment, the processing method provided in this application may further include the following processing steps before performing super-resolution processing on the first image data based on the image super-resolution model:

[0080] Determine the target hardware for running the image super-resolution model and the maximum size of the image super-resolution model that the target hardware can currently support.

[0081] The target hardware may be, but is not limited to, processors such as GPUs (Graphics Processing Units) or NPUs (Neural Network Processing Units).

[0082] The GPU may further include at least one of dGPU (Discrete Graphics Processing Unit) and iGPU (Integrated Graphics Processing Unit).

[0083] The process of determining the target hardware for running the image super-resolution model and the maximum size of the image super-resolution model that the target hardware can currently support can be further implemented as step "2-1" or "2-2" below.

[0084] 2-1: If the electronic device includes a processor capable of running an image super-resolution model, that processor is identified as the target processor for running the image super-resolution model, and the maximum size of the image super-resolution model that the target processor can currently support is determined based on the current load information of the target processor.

[0085] If an electronic device includes only one processor capable of running an image super-resolution model, such as only one GPU or one NPU, then that single processor is directly identified as the target processor. Based on the target processor's current load information, its current available resource rate is determined, and then based on its current available resource rate, the maximum size of the image super-resolution model that the target processor can currently support is determined.

[0086] The current load information of the target processor can include multiple dimensions of metrics, including but not limited to the target processor's utilization (such as GPU utilization), cache utilization, number of running processes, etc.

[0087] "Model size" can refer to the physical space occupied by the model file in storage or memory. The model size is positively correlated with the amount of resources required to run the model. The larger the model size, the higher the amount of resources required to run it.

[0088] 2-2: If the electronic device includes multiple processors capable of running image super-resolution models, the target processor for running the image super-resolution model is determined by an adjustment strategy based on the current load information and response rate of each processor, so as to migrate the image super-resolution model from the current processor to the target processor, and the maximum size of the image super-resolution model that the target processor can currently support is determined based on the current load information of the target processor.

[0089] The aforementioned adjustment strategy is used to adjust at least one of the running position and model size of the image super-resolution model. Optionally, the adjustment strategy prioritizes adjusting the running position of the image super-resolution model over adjusting the model size.

[0090] Different operating positions indicate different processors, and different processors represent different operating positions of the image super-resolution model.

[0091] In electronic devices that include multiple processors capable of running image super-resolution models, a strategy can be adjusted to prioritize processors with low load (high available resources) and fast response rates as the target processors for running the image super-resolution models. Optionally, a priority can be set between processor load and response rate, and the selection of the target processor can be combined with this priority.

[0092] For example, suppose processor load takes precedence over response rate. And suppose the electronic device includes two processors, an NPU and a GPU, capable of running image super-resolution models. In this case, the processor with the lower load can be selected as the target processor to run the image super-resolution model. If the loads of the NPU and GPU are the same or the load difference is less than a threshold, then the processor with the faster response rate (such as the NPU) can be further considered as the target processor to run the image super-resolution model.

[0093] In practical applications, the target processor can be determined in real time or periodically by adjusting the strategy based on the real-time load information and response rate of each processor. If the target processor is different from the processor currently used to run the image super-resolution model, the running position of the image super-resolution model can be migrated and the model can be moved to the target processor for running.

[0094] When relocation is not possible (e.g., all processors are under high load, at full or near full load), and the processor load currently used to run the image super-resolution model has reached its limit, the image super-resolution model can be replaced to reduce its size. Since reducing the model size affects the output quality of the image super-resolution, the model size should only be reduced when relocation is not possible. In other words, adjusting the running location of the image super-resolution model has a higher priority than adjusting its size.

[0095] Once the target processor is identified, the maximum size of the image super-resolution model that it can currently support can be determined based on the current load information of the target processor, and the image super-resolution model of the maximum size that it can support can be run to improve the image super-resolution performance as much as possible.

[0096] In this embodiment, super-resolution processing is performed on the first image data based on an image super-resolution model. This can be implemented as follows: the target hardware, such as the target processor, performs super-resolution processing on the first image data based on the largest image super-resolution model it can support.

[0097] Based on this embodiment, a processor with low load and high response rate can be dynamically selected as the target processor, and the image super-resolution model with the largest size it can support can be used for image super-resolution processing, thereby further improving the image super-resolution performance, accelerating the image super-resolution rate and / or improving the image super-resolution quality.

[0098] In an alternative embodiment, see Figure 2 The flowchart shown illustrates that the processing method provided in this application may further include the following processing before performing target processing on the first image data:

[0099] Step 201: Determine the processing mode of the electronic device for the image acquisition data, and in response to the processing mode being the first mode, trigger the step of performing target processing on the first image data.

[0100] The processing modes of image acquisition data by electronic devices may include, but are not limited to, a first mode and a second mode. The first mode is used to improve the resolution of the output image after acquisition, while the second mode maintains the resolution of the output image after acquisition.

[0101] Users can set the image acquisition data processing mode on the electronic device according to their actual needs, and the electronic device can determine the image acquisition data processing mode based on the user's mode setting information.

[0102] If the electronic device processes the image acquisition data in the first mode, the processing method of this application can be triggered to perform target processing and subsequent processing on the first image data, so as to improve the image resolution of the first image data and obtain high-resolution second image data, so that the application can directly obtain the high-resolution second image data from the operating system for screen output and other related applications.

[0103] In this mode, after performing target processing such as super-resolution on the first image data to obtain the second image data, the second image data can be output to the second interface in the operating system, and then further transmitted to the application interface in the operating system via the second interface, so that the application can obtain and use the second image data by accessing the application interface at the system layer, for example, storing the second image data or outputting the screen based on the second image data.

[0104] Step 202: In response to the processing mode being the second mode, the first image data is output to the operating system so that the application can obtain the first image data from the operating system for screen output.

[0105] If the electronic device processes the image acquisition data in the second mode, it avoids triggering the target processing of the first image data to maintain its existing resolution, and can output the first image data to the first interface in the operating system, that is, the native camera interface of the system layer. Through this interface, the first image data is further transmitted to the application interface in the system layer, so that the application can acquire and use the second image data through the application interface.

[0106] Based on this embodiment, in practical applications, a dynamic adaptive dual-path architecture that runs in parallel with the super-resolution processing channel and the traditional rendering channel can be established to process image acquisition data.

[0107] See Figure 3The diagram shows the dynamic adaptive dual-path architecture. The working process of this dual-path architecture is as follows.

[0108] For data acquired by image acquisition devices such as cameras, the data is first converted from analog to digital by an ADC (Analog-to-Digital Converter) to obtain the first image data in digital signal form. Then, the data is processed at the system level by components such as an ISP (Image Signal Processor), and the processed first image data, such as YUV image data, is output through the camera driver.

[0109] Based on this, if the system needs to perform super-resolution processing on image data, for example, if the electronic device is currently in the first mode, then the first image data can be processed based on the super-resolution processing channel, combined with... Figure 3 and Figure 4 As shown, specifically, the first image data can be intercepted through a hook set in the Driver, and then processed by a target processor such as an NPU for noise reduction, image super-resolution, and HDR to obtain the second image data (in... Figure 3 (represented as YUV* in Chinese), then the second image data is sent to the second interface of the system layer, such as... Figure 3 The virtual camera filter in the middle, and then transmits the second image data to the system-level application interface via the second interface, such as Figure 3 The DirectShow API is a system-level API that can be called by upper-layer applications. It is responsible for sending image data output from the camera interface (such as the first interface / second interface) to the application. After the application obtains the second image data through this system-level application API and renders it, it can output the image in RGB (Red-Green-Blue) format.

[0110] See Figure 5 , Figure 4 The three-level processing shown includes denoising, super-resolution, and post-processing (such as HDR) of the image data in a target processor such as an NPU, so as to process the first image data into high-resolution second image data based on the three-level processing.

[0111] Table 1 provides examples of the relevant technologies, resource configurations, and performance indicators involved in the above three-level processing.

[0112] Table 1

[0113]

[0114] If the system does not require super-resolution processing of the image data, for example, if the electronic device is currently in a second mode, the first image data can be processed based on the traditional rendering pipeline. Figure 3 As shown, in a traditional rendering pipeline, the first image data can be directly transmitted to the first interface of the system layer, such as... Figure 3 The system-level native filter transmits the first image data to the application via the system-level application interface Direct Show API, so that the application can render and output the image as needed.

[0115] In implementation, a three-level processing engine can be embedded in the operating system: driver layer hook → super-resolution channel → virtual camera interface (second interface). This allows low-end cameras to display higher image quality on high-resolution displays without modifying upper-layer applications or replacing hardware. Compared with existing technologies, this application has the technical advantages shown in Table 2 below.

[0116] Table 2

[0117]

[0118] Based on the solution in this embodiment, the electronic device can process the image data acquired by the image acquisition device on demand according to its current mode, and provide the image data to the system layer through the corresponding channel in the dynamic adaptive dual-path architecture for the upper-layer application to obtain and use. This further improves the flexibility of image processing while meeting the user's high-resolution image quality requirements.

[0119] The following is an application example of the method of this application.

[0120] In this example, the system layer processes the image data captured by the camera through a dynamic adaptive mechanism. The overall processing flow is as follows: Figure 6 As shown.

[0121] The dynamic adaptive working mechanism can dynamically adjust the running position (such as NPU or dGPU / iGPU) and model size (such as large model A or small model B) of the image super-resolution model based on the system resource usage (mainly the utilization rate of NPU and GPU) and the response rate of different processors.

[0122] The adjustment strategy prioritizes relocating the model's runtime location because changing the model size would affect the image super-resolution output quality. Therefore, the model size is only changed when relocation is not feasible. This adjustment strategy can utilize a multi-dimensional decision matrix to cover various image processing scenarios. This multi-dimensional decision matrix includes relocation decisions based on processor load and response rate, and adjustments to the model size based on available processor resources.

[0123] The following are examples of different scenarios for the above processing flow.

[0124] 1. The device runs a single app that requires image output.

[0125] For example, a single video conferencing app runs on the device.

[0126] In this scenario, as shown in Figure 7(a), processor resources are abundant, so processors with high available resources and fast response speed, such as the NPU, can be prioritized to run large model A based on the adjustment strategy, so as to output high-quality super-resolution images.

[0127] 2. Load migration optimization

[0128] In this scenario, referring to Figure 7(b), assuming that the NPU utilization rate is high (e.g., 70%), the availability rate drops to a set lower limit (e.g., 30%), and the GPU has sufficient available resources, then model location migration can be performed to migrate model A from the NPU to the GPU so that the super-resolution processing does not degrade.

[0129] 3. Extreme scenarios: NPU overload without dGPU

[0130] In this scenario, as shown in Figure 7(c), the NPU overload causes its utilization rate to exceed the set limit, such as 90%, and there is no GPU in the system, such as no dGPU. In this case, the model size can be reduced and a small-sized model B can be used to skip the quality assurance of image super-resolution and effectively consider system stability.

[0131] 4. Extreme scenarios: Sudden peak GPU rendering

[0132] In this scenario, as shown in Figure 7(d), if there is a sudden surge in GPU rendering, the image super-resolution model can be migrated from the GPU to the NPU. If the available resources of the NPU are insufficient, the model size can be temporarily reduced by using a smaller model B to balance the GPU load.

[0133] This application also provides an electronic device, the composition of which is as follows: Figure 8 As shown, it includes an image acquisition device 10 and a processor 20.

[0134] Image acquisition device 10.

[0135] Image acquisition devices may be, but are not limited to, cameras of electronic devices.

[0136] The processor runs an operating system and can be used to implement any of the processing methods provided in the above method embodiments.

[0137] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a neural network processor (NPU), or other programmable logic devices.

[0138] Optionally, the electronic device may also include a display device for displaying information.

[0139] Optionally, electronic devices may also include storage resources such as memory, cache, etc., for data storage or caching.

[0140] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other similar components. Memory, processor, image acquisition device, and communication interface can communicate with each other via, but are not limited to, a communication bus.

[0141] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0142] This application also discloses a storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement the processing method provided in any of the above method embodiments.

[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0144] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0145] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0146] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0147] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A processing method, comprising: The first image data output by the image acquisition device to the operating system is obtained through the driving of the image acquisition device; The first image data is processed to obtain the second image data; the image resolution of the second image data is higher than the image resolution of the first image data. The second image data is output to the operating system so that the application can obtain the second image data from the operating system for screen output.

2. The processing method according to claim 1, wherein obtaining the first image data output by the image acquisition device to the operating system through the driving of the image acquisition device includes: The data interception unit set in the driver of the image acquisition device intercepts the first image data output by the image acquisition device to the first interface in the operating system. The first image data includes at least one of the images or video streams acquired by the image acquisition device, and the first interface is used to output image data that can be used by the application layer.

3. The processing method according to claim 1, wherein target processing is performed on the first image data, comprising: Super-resolution processing is performed on the first image data based on an image super-resolution model.

4. The processing method according to claim 3, further comprising performing target processing on the first image data, including at least one of the following: Before performing super-resolution processing on the first image data, the first image data is denoised, so that super-resolution processing is performed on the denoised first image data. After performing super-resolution processing on the first image data, image enhancement processing is performed on the super-resolution processed first image data to obtain the second image data.

5. The processing method according to claim 3 or 4, further comprising, before performing super-resolution processing on the first image data based on the image super-resolution model: Determine the target hardware for running the image super-resolution model and the maximum size of the image super-resolution model that the target hardware can currently support; Super-resolution processing is performed on the first image data based on an image super-resolution model, including: The target hardware performs super-resolution processing on the first image data based on the maximum-size image super-resolution model.

6. The processing method according to claim 5, determining the target hardware for running the image super-resolution model and the maximum size of the image super-resolution model currently supported by the target hardware, includes: If the electronic device includes a processor capable of running an image super-resolution model, the processor is identified as the target processor for running the image super-resolution model, and the maximum size of the image super-resolution model that the target processor can currently support is determined based on the current load information of the target processor. If an electronic device includes multiple processors capable of running image super-resolution models, a target processor for running the image super-resolution model is determined based on the current load information and response rate of each processor through an adjustment strategy, so as to migrate the image super-resolution model from the current processor to the target processor, and the maximum size of the image super-resolution model that the target processor can currently support is determined based on the current load information of the target processor. The adjustment strategy is used to adjust at least one of the running position and model size of the image super-resolution model; the priority of the adjustment strategy in adjusting the running position of the image super-resolution model is higher than the priority of the adjustment strategy in adjusting the model size of the image super-resolution model; each processor includes a graphics processor and a neural network processor, and different running positions indicate different processors.

7. The processing method according to claim 2, wherein the second image data is output to the operating system so that the application program obtains the second image data from the operating system for screen output, includes: The second image data is output to the second interface in the operating system so that the application can obtain the second image data output by the second interface based on the corresponding application interface in the operating system; The second interface differs from the first interface in that it is used to output image data that can be used by the application layer.

8. The processing method according to any one of claims 1-7, further comprising, before performing target processing on the first image data: Determine the processing mode of the electronic device for image acquisition data; In response to the processing mode being the first mode, a step of performing target processing on the first image data is triggered; In response to the processing mode being the second mode, the first image data is output to the operating system so that the application can obtain the first image data from the operating system for screen output. The first mode is used to improve the resolution of the image output after acquisition, while the second mode maintains the resolution of the image output after acquisition.

9. An electronic device, comprising: Image acquisition device; A processor, which runs an operating system, is used for: The first image data output by the image acquisition device to the operating system is obtained through the driving of the image acquisition device; The first image data is processed to obtain the second image data; the image resolution of the second image data is higher than the image resolution of the first image data. The second image data is output to the operating system so that the application can obtain the second image data from the operating system for screen output.

10. The electronic device according to claim 9, wherein when the processor obtains the first image data output by the image acquisition device to the operating system, it is configured to: intercept the first image data output by the driver of the image acquisition device to a first interface in the operating system; When the processor outputs the second image data to the operating system so that the application can obtain the second image data from the operating system for screen output, it is configured to: output the second image data to a second interface in the operating system so that the application can obtain the second image data output by the second interface based on the corresponding application interface in the operating system; in, The first interface and the second interface are respectively used to output image acquisition data that can be used by the application layer, and the first interface is different from the second interface.

Images