Image processing system and method, electronic device, readable storage medium, and chip

Abstract

The present application provides an image processing system and method, an electronic device, a readable storage medium, and a chip. The image processing system comprises: an image processing chip and a hardware abstraction layer which are communicatively connected to each other; and a manufacturer framework layer and an application framework layer, wherein the manufacturer framework layer is communicatively connected to the hardware abstraction layer and the application framework layer, separately, the manufacturer framework layer is also communicatively connected to the image processing chip, and the manufacturer framework layer is used for performing communication control on the image processing chip.

Description

Image processing system, method, electronic device, readable storage medium and chip

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 27, 2023, with application number 202310938352.0 and invention name “Image processing system, method, electronic device, readable storage medium and chip”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing, and in particular to an image processing system, method, electronic device, readable storage medium and chip. Background Art

[0003] With the development of communication technology, electronic devices are becoming increasingly powerful. For example, users can use electronic devices to capture images. However, when using existing electronic devices to capture images, the quality of the captured images depends to a certain extent on the external shooting environment. When there are many external interference factors affecting the shooting, the captured images are usually not ideal.

[0004] In order to improve the shooting quality of electronic devices, the related art proposes that an image processing chip can be added to the electronic device. In the related art, the control layer of the image processing chip is usually set in the camera hardware abstraction layer of the system on chip (SoC), so that the SoC can call the image processing chip to perform image processing. When the image processing chip is applied to an electronic device, it is usually necessary to write interactive logic code to facilitate normal interaction between the SoC and the image processing chip. However, when the same image processing chip needs to be applied to SoCs provided by different platforms, due to different types of SoCs, there are usually certain differences in the internal camera hardware abstraction layer. As a result, users have a large difference in user experience when using different electronic device models.

[0005] Summary of the Invention

[0006] The present application provides an image processing system, method, electronic device, readable storage medium and chip, which can solve the problem that there are large differences in user experience when using different electronic device models.

[0007] In a first aspect, an embodiment of the present application provides an image processing system, the system comprising:

[0008] An image processing chip and a hardware abstraction layer, wherein the image processing chip is communicatively connected with the hardware abstraction layer;

[0009] A manufacturer framework layer and an application framework layer, wherein the manufacturer framework layer is communicatively connected with the hardware abstraction layer and the application framework layer respectively;

[0010] The manufacturer framework layer is also in communication connection with the image processing chip, and the manufacturer framework layer is used to perform communication control on the image processing chip.

[0011] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to the image processing system described in the first aspect, and the method includes:

[0012] When the vendor framework layer receives the first image data stream output by the camera hardware abstraction module, controlling the vendor framework layer to transmit the first image data stream to the image processing chip;

[0013] controlling the image processing chip to perform image processing on the first image data stream to obtain a second image data stream; and

[0014] The image processing chip is controlled to transmit the second image data stream to the manufacturer framework layer.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image processing method described in the second aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the image processing method described in the second aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the image processing method described in the second aspect.

[0018] The image processing system in the embodiment of the present application includes: an image processing chip, a hardware abstraction layer, a manufacturer framework layer and an application framework layer, wherein the image processing chip is communicated with the hardware abstraction layer; the manufacturer framework layer is communicated with the hardware abstraction layer and the application framework layer respectively; the manufacturer framework layer is also communicated with the image processing chip, and the manufacturer framework layer is used to communicate and control the image processing chip. Therefore, by adding a manufacturer framework layer to the Android framework system, the manufacturer framework layer is communicated with the hardware abstraction layer and the application framework layer, and some functional modules of the hardware abstraction layer are moved up to the manufacturer framework layer, so that the functional modules after being moved up can be independently called by the application framework layer without being bound to the camera hardware abstraction module, and do not rely on the development of the camera hardware abstraction module. After decoupling, the chip can be upgraded independently, and can be quickly adapted when a new platform is introduced. Thus, the problem of large differences in user experience when using different electronic device models can be solved, so that users can have a unified experience when using different electronic device models, instead of having differences in memory usage and power consumption due to differences in processing solutions caused by platform differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a schematic structural diagram of an image processing system provided by the present application;

[0020] FIG2 is a flow chart of an image processing method provided by the present application;

[0021] FIG3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0022] FIG4 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0024] The terms "first," "second," and so on, in the specification and claims of this application are used to distinguish similar objects, not to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein. Furthermore, the objects distinguished by "first," "second," and so on generally refer to a class and do not limit the number of objects. For example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the connected objects are in an "or" relationship.

[0025] The image processing system, method, electronic device, readable storage medium and chip provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0026] To facilitate understanding, the Android system framework structure involved in the embodiments of the present application is first introduced.

[0027] Application layer (Applications): The application layer consists of all applications running on Android devices, including pre-installed system applications and third-party applications installed by users.

[0028] The Application Framework layer is the core of Android application development and provides a basic Application Programming Interface (API) framework for developers to develop applications.

[0029] The system runtime layer, libraries or Android Runtime (ART), libraries encapsulate a series of function libraries for use by the upper layer; ART is composed of Java core libraries (Core Libraries) and Android virtual machine (Dalvik).

[0030] The Hardware Abstraction Layer (HAL) is an interface layer between the operating system kernel and the hardware circuit. Its purpose is to abstract the hardware.

[0031] The Linux kernel layer is the foundation of the Android system, mainly implementing memory management, schedule scheduling, inter-process communication, device drivers, etc.

[0032] Please refer to FIG1 below, which is a schematic diagram of the structure of an image processing system provided by an embodiment of the present application. The image processing system includes:

[0033] An image processing chip and a hardware abstraction layer, wherein the image processing chip is communicatively connected with the hardware abstraction layer;

[0034] A manufacturer framework layer and an application framework layer, wherein the manufacturer framework layer is communicatively connected with the hardware abstraction layer and the application framework layer respectively;

[0035] The manufacturer framework layer is also in communication connection with the image processing chip, and the manufacturer framework layer is used to perform communication control on the image processing chip.

[0036] The above-mentioned image processing system may be an image processing system in an electronic device, and the electronic device may be various types of user terminals, such as a mobile phone, a tablet computer, etc., or the electronic device may also be a device developed by a manufacturer.

[0037] The above-mentioned image processing system may also include an SoC in an electronic device. The above-mentioned image processing chip is a chip in the electronic device other than the SoC for processing images. The image processing chip may also be called an image enhancement chip or a plug-in chip. Specifically, in the process of image processing by the image processing system, on the basis of processing the image based on the SoC, it may be selected to further perform image enhancement processing on the image output by the SoC based on the image processing chip to improve the image processing effect. For example, the above-mentioned image processing chip can be used to perform the following processing on the image: denoising processing, clarity adjustment processing, and white balance processing, etc. The SoC can be used to perform the following processing on the image: stylization processing, watermarking processing, denoising processing, etc.

[0038] The hardware abstraction layer may include a camera hardware abstraction module. Specifically, the hardware abstraction layer may be communicatively connected with the manufacturer framework layer through the camera hardware abstraction module.

[0039] The camera hardware abstraction module is a standard module provided by Google. In related technologies, in order to control the image processing chip, the SoC of each platform usually stores the control logic of the image processing chip in the camera hardware abstraction module. Since the image processing logic of different platforms may be different, the camera hardware abstraction modules of different platforms may be different. In the embodiment of the present application, the hardware abstraction layer in the SoC corresponding to each platform can communicate with the image processing chip based on the manufacturer framework layer. That is, there is no need to set the control logic of the corresponding image processing chip in the camera hardware abstraction module in the hardware abstraction layer of the SoC corresponding to each platform. That is, there is no need to modify the camera hardware abstraction module during the development process, so that the SoC of each platform has the same camera hardware abstraction module.

[0040] Because the SoCs on each platform share the same camera hardware abstraction module, they can share a common set of interaction code with the vendor framework layer. When developing an image processing chip across platforms, the logic interaction code between the image processing chip and the first camera service module only needs to be written once. This allows the camera hardware abstraction modules on different platforms to interact with the image processing chip through the vendor framework layer, thus avoiding the need to rewrite a single set of interaction logic multiple times. This solves the significant differences in user experience across different electronic device models, ensuring a consistent user experience across different electronic device models, rather than differences in memory usage and power consumption due to differences in processing solutions caused by platform differences.

[0041] The above-mentioned manufacturer framework layer can also be called the Original Equipment Manufacturer (OEM) framework layer. The manufacturer framework layer is a functional module in the electronic device that is independent of the SoC. The manufacturer framework layer can pre-store customized code from each platform. Specifically, such as the High Dynamic Range (HDR) photography algorithm, different platform manufacturers will provide self-developed HDR algorithms, and the code of the self-developed HDR algorithm can be transplanted to the manufacturer framework layer.

[0042] This allows the manufacturer's framework layer to establish communication connections between the SoC and image processing chips of each platform.

[0043] The aforementioned platforms may specifically include various types of SoC development platforms.

[0044] The application framework layer may be an application framework layer in an Android system framework, for example, the application framework layer may be a Google Framework layer. It is understood that the application framework layer may establish a communication connection between the manufacturer framework layer and the application layer.

[0045] It should be noted that the communication control of the image processing chip by the manufacturer framework layer may refer to the control logic of the image processing chip or the data transmission control of the input / output (I / O) channel.

[0046] The vendor framework layer connects to the application framework layer via the Android Interface Definition Language (AIDL) data path and defines a common interface for the camera module, enabling configuration of data streams and processing of capture requests. The vendor framework layer establishes an internal mechanism for transferring image data streams and metadata, forwarding image data streams from the hardware abstraction layer to the application framework layer.

[0047] In an embodiment of the present application, by adding a manufacturer framework layer to the Android framework system, the manufacturer framework layer is communicated with the hardware abstraction layer and the application framework layer, so that some functional modules of the hardware abstraction layer are moved up to the manufacturer framework layer, so that the functional modules after moving up can be called independently by the application framework layer, without being bound to the camera hardware abstraction module, and are not dependent on the development of the camera hardware abstraction module. After decoupling, the chip can be upgraded independently, and can be quickly adapted when a new platform is introduced. Thus, the problem that there are large differences in the user experience when using different electronic device models is solved, so that the user experience of using different electronic device models is unified, instead of having differences in memory usage and power consumption due to differences in processing solutions caused by platform differences. At the same time, when developing image processing chips across platforms, it is only necessary to write the logical interaction code between the image processing chip and the manufacturer framework layer once, and the hardware abstraction layers of different platforms can control the image processing chip based on the manufacturer framework layer, thereby improving the efficiency of cross-platform development of image processing chips.

[0048] Optionally, the manufacturer framework layer includes a first camera service module and a display hardware synthesizer;

[0049] The first camera service module is used to communicate and control the image processing chip through the hardware abstraction layer;

[0050] The display hardware synthesizer is used to transmit the preview image stream to the image processing chip under the control of the application framework layer.

[0051] The first camera service module may be a vendor camera service module (Vendor Camera Server). The display hardware compositor may be a hardware composition abstraction layer (HWComposer, HWC).

[0052] In an embodiment of the present application, a communication control path is formed between the first camera service module of the manufacturer framework layer, the hardware abstraction layer, and the image processing chip, which can realize that the image data stream is output from the image sensor and then input to the first camera service module through the hardware abstraction layer, or that the data stream is output from the image processing chip and then input to the first camera service module through the hardware abstraction layer.

[0053] In the embodiments of the present application, the preview image stream may refer to image data to be sent to a display screen for display during the image capture process of an electronic device. When an application at the application layer requires camera preview services, the preview image stream is sent down to the display hardware synthesizer via the application framework layer. The preview image stream is then sent out to the image processing chip by the display hardware synthesizer, and then transmitted to the display device for preview display by the image processing chip.

[0054] In this embodiment, by including the first camera service module and the display hardware synthesizer in the vendor framework layer, the vendor framework layer can, on the one hand, control communication with the image processing chip through the hardware abstraction layer, and, on the other hand, transmit a preview image stream to the image processing chip through the vendor framework layer. This enables communication control between the image sensor and image processing chip and the vendor framework layer, as well as transmission of preview requests from upper-layer applications to the image processing chip via the vendor framework layer.

[0055] Please refer to Figure 1. The image processing system may also include an image sensor, a hardware layer and a kernel layer. The kernel layer includes a video device driver (video for linux 2, V4L2). The hardware layer includes the image sensor's port CSI0, and the image sensor's port CSI0 is connected to the camera hardware abstraction module via V4L2. The image sensor can sequentially transmit the image data stream collected by the image sensor to the camera hardware abstraction module via the ports CSI0 and V4L2. After receiving the image, the camera hardware abstraction module can transmit the image data stream to the manufacturer framework layer. The manufacturer framework layer can transmit the received image data stream to the image processing chip so that the image processing chip can process the image. In addition, the manufacturer framework layer can also transmit the image data stream received from the image processing chip to the application framework layer so that the image can be displayed or stored based on the application framework layer.

[0056] Optionally, the hardware abstraction layer includes a camera hardware abstraction module, and the first camera service module is connected to the image processing chip through the display hardware synthesizer to form a first data transmission path;

[0057] Optionally, the image processing chip is connected to the first camera service module through a camera hardware abstraction module to form a second data transmission path.

[0058] The camera hardware abstraction module may be a camera hardware abstraction layer (Camera Hardware Abstraction Layer, Camera HAL).

[0059] As shown in Figure 1 , the kernel layer also includes a Direct Rendering Manager Driver (DRM Driver). The first camera service module is connected to the image processing chip's DSI1 port via the display hardware compositor and the DRM Driver, thereby forming a first data transmission path. The image processing chip is connected to the first camera service module via the image sensor's CSI1 port, the kernel layer's V4L2, and the Camera Hal, thereby forming a second data transmission path.

[0060] In this embodiment, by forming the aforementioned first data transmission path, the first camera service module can transmit the image data stream to be processed to the image processing chip via the first data transmission path for image processing. Correspondingly, by forming the aforementioned second data transmission path, the image processing chip can transmit the processed image data back to the first camera service module via the second data transmission path.

[0061] Optionally, the system further includes a display screen, a memory, and an application layer;

[0062] Optionally, the application framework layer includes a second camera service module, a display framework module and a media codec;

[0063] The above-mentioned second camera service module can be a Google Camera Server; the above-mentioned display framework module can be Surface Flinger, which runs in the system process of the Android framework and is responsible for managing the frame buffer (Frame Buffer) of the Android system; the above-mentioned media codec can be a media codec (Media Codec).

[0064] Optionally, the first camera service module is connected to the display screen through the second camera service module, the application layer, the display framework module, the display hardware synthesizer, and the image processing chip in sequence to form a third data transmission path;

[0065] In an embodiment of the present application, referring to FIG1 , the first camera service module is connected to the display screen through the second camera service module, the application layer, the display framework module, the display hardware synthesizer, the DRM Driver, the image processing chip, and the display processor (DPU) in sequence to form a third data transmission path.

[0066] In this embodiment, the third data transmission path is formed so that the first camera service module can transmit the processed image to the display screen through the third data transmission path for preview display.

[0067] Optionally, the first camera service module is connected to the memory via the second camera service module and the media codec in sequence to form a fourth data transmission path.

[0068] In an embodiment of the present application, referring to FIG1 , the first camera service module is connected to the memory interface (UFS / sMMC) in sequence through the second camera service module, the media codec, the open multimedia acceleration module (Open Media Acceleration, OMX), and the data block driver module (Block Driver) to form a fourth data transmission path.

[0069] In this embodiment, the fourth data transmission path is formed so that the first camera service module can transmit the processed image to the memory for storage through the fourth data transmission path.

[0070] Optionally, the image processing system also includes an application layer, the application framework layer includes a display enhancement service module, and the manufacturer framework layer also includes an image processing chip hardware abstraction module. The application layer is connected to the image processing chip through the display enhancement service module and the image processing chip hardware abstraction module in sequence to form a first control path.

[0071] In this embodiment, the display enhancement service module may be a display enhancement service (Display Enhanced Service) of the Google framework layer, and the image processing chip hardware abstraction module may be an application specific integrated circuit hardware abstraction layer (ASIC HAL).

[0072] Referring to FIG. 1 , the application layer is connected to the SPI of the image processing chip through the display enhancement service module, the image processing chip hardware abstraction module, and the serial peripheral interface driver (SPI Driver) in sequence to form the first control path.

[0073] Specifically, the first control path can be conveniently used to control the state of the image processing chip based on corresponding controls in the user interface. For example, when the electronic device displays a specific interface, a display enhancement switch can be set in the display interface. The user can turn the display enhancement switch on or off as needed to improve the image quality of the displayed interface. The specific interface can specifically be a game interface or a shooting preview interface.

[0074] It can be understood that when the display enhancement function of the image processing chip is activated, the first camera service module can first transmit the image that needs to be transmitted to the display screen for display to the image processing chip through the above-mentioned first data transmission path for display enhancement, and the image processing chip then transmits the image after display enhancement to the first camera service module through the second data transmission path, and the first camera service module then transmits the image after display enhancement to the display screen based on the third data transmission path for display.

[0075] In this embodiment, by forming the above-mentioned first control path, the user can control the state of the image processing chip through the control in the user interface.

[0076] Optionally, the manufacturer framework layer further includes an image processing chip hardware abstraction module, and the first camera service module is connected to the image processing chip through the image processing chip hardware abstraction module to form a second control path.

[0077] Referring to FIG. 1 , the first camera service module may be connected to the SPI of the image processing chip via the image processing chip hardware abstraction module and the SPI Driver in sequence to form the second control path.

[0078] In this embodiment, the second control path is formed so that the first camera service module can control the state of the image processing chip through the second control path.

[0079] In the embodiment of the present application, when the image processing chip is in the display enhancement mode, the image processing chip is notified through the first control path and the second control path to perform color enhancement on the preview image stream.

[0080] In the embodiment of the present application, when the video recording is finished, the image processing chip is notified to finish the operation through the first control path and the second control path, thereby saving the operating power consumption of the electronic device.

[0081] Please refer to FIG2 , which is a flow chart of an image processing method provided in an embodiment of the present application. The image processing method is applied to the image processing system described in the above embodiment. The method includes the following steps:

[0082] Step 201: When the manufacturer framework layer receives a first image data stream output by a camera hardware abstraction module, control the manufacturer framework layer to transmit the first image data stream to the image processing chip;

[0083] Step 202: Control the image processing chip to perform image processing on the first image data stream to obtain a second image data stream;

[0084] Step 203: Control the image processing chip to transmit the second image data stream to the manufacturer framework layer.

[0085] The above-mentioned image processing chip can specifically perform image processing on the first image data stream by using common image processing means in related technologies. For example, the first image data stream can be processed as follows: denoising processing, clarity adjustment processing, and white balance processing.

[0086] In an embodiment of the present application, the image sensor transmits the collected first image data stream to the hardware abstraction module through the kernel layer, and then transmits it to the manufacturer framework layer through the hardware abstraction module. The manufacturer framework layer is then controlled to transmit the first image data stream to the image processing chip through the kernel layer; the image processing chip is controlled to perform image processing on the first image data stream to obtain a second image data stream; and the image processing chip is then controlled to transmit the second image data stream to the manufacturer framework layer through the kernel layer and the hardware abstraction layer.

[0087] In the embodiment of the present application, the vendor framework layer can be controlled to implement control over the output data of the image sensor and the input and output of the image processing chip, thereby enabling the control logic and data path of the image processing chip to be controlled through the vendor framework layer. In this implementation, since the image processing method provided in this embodiment corresponds to the above-mentioned image processing system, the image processing method has all the beneficial effects of the above-mentioned image processing system.

[0088] Optionally, the manufacturer framework layer includes a first camera service module and a display hardware synthesizer, and the first camera service module is connected to the image processing chip through the display hardware synthesizer to form a first data transmission path;

[0089] The controlling the manufacturer framework layer to transmit the first image data stream to the image processing chip includes:

[0090] Control the first camera service module in the manufacturer framework layer to transmit the first image data stream to the image processing chip through the first data transmission path.

[0091] In this embodiment, by forming the above-mentioned first data transmission path, the first camera service module can transmit the first image data stream to the image processing chip through the first data transmission path for image processing.

[0092] In the embodiment of the present application, with reference to FIG1 , the first image data stream may be image data captured by the image sensor, and the image sensor may sequentially transmit the first image data stream to the Camera Hal via ports CSI0 and V4L2. Upon receiving the first image data stream, the Camera Hal may transmit the first image data stream to the vendor framework layer. The vendor framework layer transmits the first image data stream to the image processing chip via the DMR Driver and DSI1.

[0093] Optionally, the image processing chip is connected to the first camera service module through the camera hardware abstraction module to form a second data transmission path;

[0094] The controlling the image processing chip to transmit the second image data stream to the manufacturer framework layer includes:

[0095] The image processing chip is controlled to transmit the second image data stream to the first camera service module in the manufacturer framework layer through the second data transmission path.

[0096] Accordingly, by forming the aforementioned second data transmission path, the image processing chip can transmit the second image data stream back to the first camera service module via the second data transmission path. Specifically, the image processing chip outputs the second image data stream to the vendor camera server at the vendor framework layer via CSI1, V4L2, and Camera HAL.

[0097] Optionally, the system further includes a display screen, a memory and an application layer, and the application framework layer includes a second camera service module, a display framework module and a media codec.

[0098] Optionally, the first camera service module is connected to the display screen through the second camera service module, the application layer, the display framework module, the display hardware synthesizer, and the image processing chip in sequence to form a third data transmission path;

[0099] After controlling the image processing chip to transmit the second image data stream to the manufacturer framework layer, the method further includes:

[0100] The first camera service module is controlled to transmit the second image data stream to the display screen through the third data transmission path for preview display.

[0101] In this embodiment, the first camera service module is controlled to transmit the second image data stream to the display screen for preview display via a third data transmission path. Specifically, the second image data stream is output by the first camera service module in the vendor framework layer, passes through the second camera service module in the application framework layer, and then passes through the application layer to the display framework module in the hardware framework layer. It is then output to the display hardware compositor in the vendor framework layer, driven by the display direct rendering manager in the kernel layer, and then input to the image processing chip via DSI0. Finally, it passes through the display processor and is finally input to the display screen for display, thereby achieving preview display of the image data stream.

[0102] Optionally, the first camera service module is connected to the memory via the second camera service module and the media codec in sequence to form a fourth data transmission path;

[0103] After controlling the image processing chip to transmit the second image data stream to the manufacturer framework layer, the method further includes:

[0104] The first camera service module is controlled to transmit the second image data stream to the memory for storage through the fourth data transmission path.

[0105] In this embodiment, the first camera service module can also be controlled to transmit the second image data stream to memory for storage via a fourth data transmission path. Specifically, the second image data stream is output by the first camera service module at the vendor framework layer, passes through the second camera service module at the application framework layer, and then through the media decoder at the application framework layer, the open multimedia acceleration module at the hardware abstraction layer, and the data block driver at the kernel layer before being output to memory for storage via the UFS / sMMC. This allows for simultaneous photo or video storage of the image data stream.

[0106] Optionally, the system further includes an application layer, the application framework layer includes a display enhancement service module, and the manufacturer framework layer further includes an image processing chip hardware abstraction module;

[0107] The application layer is connected to the image processing chip through the display enhancement service module and the image processing chip hardware abstraction module in sequence to form a first control path;

[0108] The method further comprises:

[0109] When the first input is received, the application layer is controlled to send a first control instruction to the image processing chip through the first control path, where the first control instruction is used to control the image processing chip to perform image enhancement processing on the image data stream.

[0110] Specifically, when an electronic device displays a specific interface, a display enhancement switch can be set in the display interface. In this way, the user can turn the display enhancement switch on or off as needed to facilitate the user to improve the image quality of the displayed interface. The first input can refer to the user's input to turn on the display enhancement switch in the display interface, and the first input can also be the user's voice input. The specific interface can be a game interface or a shooting interface, etc.

[0111] It can be understood that the first camera service module can first transmit the image that needs to be transmitted to the display screen for display to the image processing chip through the above-mentioned first data transmission path for display enhancement, and the image processing chip then transmits the image data stream after display enhancement to the first camera service module through the second data transmission path, and the first camera service module then transmits the image data stream after display enhancement to the display screen based on the third data transmission path for display.

[0112] In this embodiment, by forming the first control path, the user can control the state of the image processing chip through the user interface displayed by the electronic device.

[0113] Optionally, the manufacturer framework layer further includes a first camera service module and an image processing chip hardware abstraction module;

[0114] The first camera service module is connected to the image processing chip through the image processing chip hardware abstraction module to form a second control path;

[0115] The method further comprises:

[0116] When the system is in a video data stream processing state and receives a video recording end input, the first camera service module is controlled to send a second control instruction to the image processing chip through the second control path, and the second control instruction is used to control the image processing chip to end the video data stream processing.

[0117] It can be understood that when the electronic device receives the start input for starting the recording function, it can send a third control instruction to the image processing chip through the second control path based on the second image data stream processing unit, wherein the third control instruction is used to control the image processing chip to start up.

[0118] The first camera service module may be pre-configured with the control logic for triggering the second control instruction and the third control instruction.

[0119] The above-mentioned input for ending the recording and input for starting the recording function may be inputs made by the user in an application interface displayed on the electronic device.

[0120] In this embodiment, since the quality of the image recorded by the video data is usually high, when the user starts recording, the image processing chip can be controlled to start based on the third control instruction, so as to perform image enhancement on the video data stream based on the image processing chip, thereby improving the video quality; correspondingly, when the electronic device only displays a preview image and does not need to record, the image processing chip can be disabled to save power consumption of the electronic device.

[0121] As shown in Figure 3, an embodiment of the present application also provides an electronic device 300, including a processor 301, a memory 302, and a program or instruction stored in the memory 302 and executable on the processor 301. When the program or instruction is executed by the processor 301, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0122] It should be noted that the interface display device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device mentioned above.

[0123] It should be understood that the processors involved in this application may include general-purpose processors, special-purpose processors, etc., such as central processing units (CPUs), microprocessors, digital signal processors (DSPs), artificial intelligence (AI) processors, graphics processors (GPUs), application-specific integrated circuits (ASICs), network processors (NPs), field programmable gate arrays (FPGAs), or other programmable logic devices, gate circuits, transistors, discrete hardware components, etc.

[0124] FIG4 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0125] The electronic device 400 includes but is not limited to components such as a radio frequency unit 401 , a network module 402 , an audio output unit 403 , an input unit 404 , a sensor 405 , a display unit 406 , a user input unit 407 , an interface unit 408 , a memory 409 , and a processor 410 .

[0126] The processor 410 is configured to, when the vendor framework layer receives the first image data stream output by the camera hardware abstraction module, control the vendor framework layer to transmit the first image data stream to the image processing chip;

[0127] controlling the image processing chip to perform image processing on the first image data stream to obtain a second image data stream; and

[0128] The image processing chip is controlled to transmit the second image data stream to the manufacturer framework layer.

[0129] Optionally, the manufacturer framework layer includes a first camera service module and a display hardware synthesizer, and the first camera service module is connected to the image processing chip through the display hardware synthesizer to form a first data transmission path;

[0130] Optionally, the image processing chip is connected to the first camera service module through the camera hardware abstraction module to form a second data transmission path;

[0131] Optionally, the processor 410 is specifically configured to control the first camera service module in the manufacturer framework layer to transmit the first image data stream to the image processing chip through the first data transmission path;

[0132] Optionally, the processor 410 is specifically configured to control the image processing chip to transmit the second image data stream to the first camera service module in the manufacturer framework layer through the second data transmission path.

[0133] Optionally, it further includes a display unit 406, a memory 409 and an application layer, wherein the application framework layer includes a second camera service module, a display framework module and a media codec;

[0134] Optionally, the first camera service module is connected to the display unit 406 in sequence through the second camera service module, the application layer, the display framework module, the display hardware synthesizer, and the image processing chip to form a third data transmission path;

[0135] Optionally, the first camera service module is connected to the memory 409 in sequence through the second camera service module and the media codec to form a fourth data transmission path;

[0136] Optionally, the processor 410 is further configured to control the first camera service module to transmit the second image data stream to the display unit 406 through the third data transmission path for display;

[0137] Optionally, the processor 410 is further configured to control the first camera service module to transmit the second image data stream to the memory 409 for storage through the fourth data transmission path.

[0138] Optionally, it further includes an application layer, the application framework layer includes a display enhancement service module, and the manufacturer framework layer also includes an image processing chip hardware abstraction module;

[0139] Optionally, the application layer is connected to the image processing chip through the display enhancement service module and the image processing chip hardware abstraction module in sequence to form a first control path;

[0140] Optionally, the processor 410 is further used to, upon receiving a first input, control the application layer to send a first control instruction to the image processing chip through the first control path, wherein the first control instruction is used to control the image processing chip to perform image enhancement processing on the image data stream.

[0141] Optionally, the manufacturer framework layer further includes a first camera service module and an image processing chip hardware abstraction module;

[0142] Optionally, the first camera service module is connected to the image processing chip through the image processing chip hardware abstraction module to form a second control path;

[0143] Optionally, the processor 410 is also used to control the first camera service module to send a second control instruction to the image processing chip through the second control path when the system is in a video data stream processing state and receives a video recording end input, and the second control instruction is used to control the image processing chip to end video data stream processing.

[0144] Those skilled in the art will appreciate that the electronic device 400 may further include a power source (e.g., a battery) to power various components. The power source may be logically connected to the processor 410 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG4 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or arrange the components differently, which will not be described in detail here.

[0145] It should be understood that in an embodiment of the present application, the input unit 404 may include a graphics processing unit (GPU) 4041 and a microphone 4042, and the graphics processor 4041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 406 may include a display panel 4061, and the display panel 4061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 407 includes a touch panel 4071 and at least one of other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 may include two parts: a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0146] The memory 409 can be used to store software programs and various data. The memory 409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 409 may include a volatile memory or a non-volatile memory, or the memory 409 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0147] Processor 410 may include one or more processing units. Optionally, processor 410 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 410.

[0148] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0149] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0150] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0151] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0152] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0153] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can essentially or in other words, the contributing part can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0155] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing system, wherein: The system comprises: An image processing chip and a hardware abstraction layer, wherein the image processing chip is communicatively connected with the hardware abstraction layer; A manufacturer framework layer and an application framework layer, wherein the manufacturer framework layer is communicatively connected with the hardware abstraction layer and the application framework layer respectively; The manufacturer framework layer is also connected to the image processing chip for communication, and the manufacturer framework layer is used to perform communication control on the image processing chip.

2. The system according to claim 1, wherein: The manufacturer framework layer includes a first camera service module and a display hardware synthesizer; The first camera service module is used to perform communication control on the image processing chip through the hardware abstraction layer; The display hardware synthesizer is used to transmit the preview image stream to the image processing chip under the control of the application framework layer.

3. The system according to claim 2, wherein: The hardware abstraction layer includes a camera hardware abstraction module, and the first camera service module is connected to the image processing chip through the display hardware synthesizer to form a first data transmission path; The image processing chip is connected to the first camera service module through a camera hardware abstraction module to form a second data transmission path.

4. The system according to claim 3, wherein: It also includes display screen, memory and application layer; The application framework layer includes a second camera service module, a display framework module and a media codec; The first camera service module is connected to the display screen through the second camera service module, the application layer, the display framework module, the display hardware synthesizer, and the image processing chip in sequence to form a third data transmission path; The first camera service module is connected to the memory via the second camera service module and the media codec in sequence to form a fourth data transmission path.

5. The system according to any one of claims 2 to 4, wherein: It also includes an application layer, the application framework layer includes a display enhancement service module, and the manufacturer framework layer also includes an image processing chip hardware abstraction module. The application layer is connected to the image processing chip through the display enhancement service module and the image processing chip hardware abstraction module in sequence to form a first control path.

6. A system according to any one of claims 2 to 5, wherein: The manufacturer framework layer also includes an image processing chip hardware abstraction module, and the first camera service module is connected to the image processing chip through the image processing chip hardware abstraction module to form a second control path.

7. An image processing method, wherein: Applied to the image processing system according to any one of claims 1 to 6, the method comprising: When the manufacturer framework layer receives the first image data stream output by the camera hardware abstraction module, the manufacturer framework layer controls the The commercial framework layer transmits the first image data stream to the image processing chip; controlling the image processing chip to perform image processing on the first image data stream to obtain a second image data stream; and, The image processing chip is controlled to transmit the second image data stream to the manufacturer framework layer.

8. The method according to claim 7, wherein: The manufacturer framework layer includes a first camera service module and a display hardware synthesizer, and the first camera service module is connected to the image processing chip through the display hardware synthesizer to form a first data transmission path; The image processing chip is connected to the first camera service module through the camera hardware abstract module to form a second data transmission path; The controlling the manufacturer framework layer to transmit the first image data stream to the image processing chip comprises: Controlling the first camera service module in the manufacturer framework layer to transmit the first image data stream to the image processing chip through the first data transmission path; The controlling the image processing chip to transmit the second image data stream to the manufacturer framework layer comprises: The image processing chip is controlled to transmit the second image data stream to the first camera service module in the manufacturer framework layer through the second data transmission path.

9. The method according to claim 8, wherein: The system also includes a display screen, a memory and an application layer, wherein the application framework layer includes a second camera service module, a display framework module and a media codec; The first camera service module is connected to the display screen through the second camera service module, the application layer, the display framework module, the display hardware synthesizer, and the image processing chip in sequence to form a third data transmission path; The first camera service module is connected to the memory via the second camera service module and the media codec in sequence to form a fourth data transmission path; After controlling the image processing chip to transmit the second image data stream to the manufacturer framework layer, the method further includes: Control the first camera service module to transmit the second image data stream to the display screen for preview display through the third data transmission path, or control the first camera service module to transmit the second image data stream to the memory for storage through the fourth data transmission path.

10. The method according to any one of claims 7 to 9, wherein: The system further comprises an application layer, the application framework layer comprises a display enhancement service module, and the manufacturer framework layer further comprises an image processing chip hardware abstraction module; The application layer is connected to the image processing chip through the display enhancement service module and the image processing chip hardware abstraction module in sequence to form a first control path; The method further comprises: When the first input is received, the application layer is controlled to send a first control instruction to the image processing chip through the first control path, where the first control instruction is used to control the image processing chip to perform image enhancement processing on the image data stream.

11. The method according to any one of claims 7 to 10, wherein: The manufacturer framework layer also includes a first camera service module and an image processing chip hardware abstraction module; The first camera service module is connected to the image processing chip through the image processing chip hardware abstraction module to form a second control path; The method further comprises: When the system is in a video data stream processing state and receives a video recording end input, the first camera service module is controlled to send a second control instruction to the image processing chip through the second control path, and the second control instruction is used to control the image processing chip to end the video data stream processing.

12. An electronic device, wherein: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 7 to 11 are implemented.

13. A readable storage medium, wherein: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 7 to 11 are implemented.

14. A chip, wherein: The chip includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the image processing method according to any one of claims 7 to 11.

15. A computer program product, wherein: The program product is executed by at least one processor to implement the steps of the image processing method according to any one of claims 7 to 11.

16. An image processing device, wherein: The apparatus is configured to perform the steps of the image processing method according to any one of claims 7 to 11.

Images