Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a storage medium and a program product, and the image processing method comprises the steps: transmitting a preview request to a hardware abstraction layer of an operating system in response to a preview request initiated by a camera application and received by a middle layer of the operating system of the electronic equipment; wherein the middle layer is used for packaging the hardware abstraction layer, and the camera application performs data interaction with the hardware abstraction layer based on the middle layer; in response to preview image data returned by the hardware abstraction layer according to the preview request and received by the middle layer, caching the preview image data to a first queue in the middle layer; wherein the preview image data is collected by a hardware abstraction layer based on a preview request driving image sensor; in response to the photographing request received by the intermediate layer, the image data is selected from the first queue, and the photographed image is generated on the intermediate layer based on the image data selected from the first queue, so that the duration of generating the photographed image can be shortened.

Description

Technical Field

[0001] This disclosure relates to the field of image technology, and in particular to an image processing method and apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of electronic devices, they have gradually become indispensable tools in people's work and life, and most electronic devices are equipped with cameras to enable photo-taking. Currently, electronic devices rely on the system architecture of their native operating systems to achieve this photo-taking function.

[0003] However, different operating systems have different hardware platform architectures. Since image processing software cannot be decoupled from the hardware platform architecture, adjusting the hardware platform architecture requires corresponding adjustments to the software. In other words, the software functionality for generating captured images is not universally applicable across different operating systems. Furthermore, while setting up an intermediate layer within the native operating system to improve compatibility can be problematic, the cross-layer transmission between the hardware platform and the intermediate layer requires the Central Processing Unit (CPU) to copy the transmitted image data, thus introducing a delay in image generation. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image processing method and apparatus, electronic device, storage medium, and program product that can shorten the time to generate photographic images while improving software design compatibility.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0006] In response to the middleware layer of the operating system of the electronic device receiving a preview request initiated by the camera application, the preview request is transmitted to the hardware abstraction layer of the operating system; wherein, the middleware layer is used to encapsulate the hardware abstraction layer, and the camera application interacts with the hardware abstraction layer based on the middleware layer;

[0007] In response to the intermediate layer receiving preview image data returned by the hardware abstraction layer based on the preview request, the preview image data is cached in the first queue in the intermediate layer; wherein, the preview image data is acquired by the image sensor driven by the hardware abstraction layer based on the preview request;

[0008] In response to the intermediate layer receiving a photo-taking request, image data is selected from the first queue, and a photo image is generated in the intermediate layer based on the image data.

[0009] In some embodiments, the method further includes:

[0010] In response to the middle layer receiving the preview request, the cache task corresponding to the preview request is added to the second queue in the middle layer, and the cache task is transmitted to the hardware abstraction layer;

[0011] The step of caching the preview image data to the first queue in the intermediate layer includes:

[0012] In response to the hardware abstraction layer transmitting the preview image data corresponding to the cached task to the intermediate layer, the preview image data is cached in the first queue; or,

[0013] The preview image data is transmitted to the second queue, and the preview image data in the second queue that meets the preset conditions is cached in the first queue.

[0014] In some embodiments, caching preview image data that meets preset conditions in the second queue to the first queue includes:

[0015] In response to transmitting the preview image data to the second queue, a first data volume and a second data volume corresponding to the preview image data are determined; wherein, the first data volume is determined by the intermediate layer, and the second data volume is determined by the hardware abstraction layer;

[0016] When the difference between the first data volume and the second data volume is less than a preset data volume threshold, the preview image data is transmitted to the first queue.

[0017] In some embodiments, the method further includes:

[0018] If the actual number of frames of the preview image data in the first queue is greater than or equal to the required number of frames indicated by the photo-taking request, the time difference between the time when the photo-taking request is initiated and the time when the preview image data is started to be collected is determined.

[0019] The preview image data corresponding to the time difference that is less than a preset time threshold is determined as the image data corresponding to the photo-taking request.

[0020] In some embodiments, the method further includes:

[0021] If the actual number of frames of the preview image data in the first queue is less than the required number of frames indicated by the photo-taking request, the first queue is updated based on the preview image data transmitted from the second queue to the first queue until the actual number of frames of the preview image data in the updated first queue reaches the required number of frames.

[0022] In some embodiments, the method further includes:

[0023] At the first moment, in response to the intermediate layer receiving the photo-taking request, the image data corresponding to the photo-taking request is selected from the first queue of the intermediate layer;

[0024] In response to the fact that the first preview image data does not meet the preset conditions corresponding to the photo-taking request at the first moment, the execution of the photo-taking request is suspended; wherein, the first preview image data is the preview image data returned based on the hardware abstraction layer before the first moment;

[0025] The first queue is updated based on the second preview image data returned by the hardware abstraction layer at the second time point; wherein, the second time point is after the first time point.

[0026] In response to the completion of the update for the first queue, the photo-taking request is re-executed, and image data that meets the preset conditions corresponding to the photo-taking request is selected from the updated first queue.

[0027] In some embodiments, generating a photographed image based on the image data in the intermediate layer includes:

[0028] In response to successfully acquiring the image data, the intermediate layer transmits the image data from the first queue to the image processor for image processing to generate the captured image;

[0029] The image processor is connected to the intermediate layer via a data transmission interface.

[0030] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0031] A transmission module is configured to, in response to a preview request initiated by a camera application received by the middleware layer of the operating system of an electronic device, transmit the preview request to the hardware abstraction layer of the operating system; wherein, the middleware layer is used to encapsulate the hardware abstraction layer, and the camera application interacts with the hardware abstraction layer based on the middleware layer;

[0032] A caching module is configured to cache the preview image data in a first queue in the intermediate layer in response to the intermediate layer receiving preview image data returned by the hardware abstraction layer based on the preview request; wherein the preview image data is acquired by the image sensor driven by the hardware abstraction layer based on the preview request.

[0033] The generation module is configured to, in response to the intermediate layer receiving a photo-taking request, select image data from the first queue and generate a photo image based on the image data in the intermediate layer.

[0034] In some embodiments, the image processing apparatus further includes:

[0035] The task addition module is configured to, in response to the middle layer receiving the preview request, add the cache task corresponding to the preview request to the second queue in the middle layer, and transmit the cache task to the hardware abstraction layer;

[0036] The caching module further includes:

[0037] The first caching module is configured to cache the preview image data in the first queue in response to the hardware abstraction layer transmitting the preview image data corresponding to the caching task to the intermediate layer; or...

[0038] The preview image data is transmitted to the second queue, and the preview image data in the second queue that meets the preset conditions is cached in the first queue.

[0039] In some embodiments, the first cache module is further configured to:

[0040] In response to transmitting the preview image data to the second queue, a first data volume and a second data volume corresponding to the preview image data are determined; wherein, the first data volume is determined by the intermediate layer, and the second data volume is determined by the hardware abstraction layer;

[0041] When the difference between the first data volume and the second data volume is less than a preset data volume threshold, the preview image data is transmitted to the first queue.

[0042] In some embodiments, the image processing apparatus further includes:

[0043] The first determining module is configured to determine the time difference between the time of initiating the photo-taking request and the time of starting to collect the preview image data when the actual number of frames of the preview image data in the first queue is greater than or equal to the required number of frames indicated by the photo-taking request.

[0044] The second determining module is configured to determine the preview image data corresponding to the time difference that is less than a preset duration threshold as the image data corresponding to the photo taking request.

[0045] In some embodiments, the image processing apparatus further includes:

[0046] The first update module is configured to update the first queue based on the preview image data transmitted from the second queue to the first queue when the actual number of frames of the preview image data in the first queue is less than the required number of frames indicated by the photo capture request, until the actual number of frames of the preview image data in the updated first queue reaches the required number of frames.

[0047] In some embodiments, the image processing apparatus further includes:

[0048] The selection module is configured to, at a first moment, in response to the intermediate layer receiving the photo-taking request, select the image data corresponding to the photo-taking request from the first queue;

[0049] The pause module is configured to pause the execution of the photo-taking request in response to the first preview image data not meeting the preset conditions corresponding to the photo-taking request at the first moment; wherein, the first preview image data is the preview image data returned based on the hardware abstraction layer before the first moment;

[0050] The second update module is configured to update the first queue based on the second preview image data returned by the hardware abstraction layer at a second time; wherein the second time is after the first time.

[0051] The execution module is configured to re-execute the photo-taking request in response to completing the update of the first queue, and select image data from the updated first queue that meets the preset conditions corresponding to the photo-taking request based on the photo-taking request.

[0052] In some embodiments, the generation module is further configured to:

[0053] In response to successfully acquiring the image data, the intermediate layer transmits the image data from the first queue to the image processor for image processing to generate the captured image;

[0054] The image processor is connected to the intermediate layer via a data transmission interface.

[0055] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0056] processor;

[0057] Memory used to store processor-executable instructions;

[0058] The processor is configured to execute any one of the image processing methods described in the first aspect above.

[0059] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising:

[0060] When the computer program or instructions in the storage medium are executed by a processor, the steps of any one of the image processing methods in the first aspect described above are implemented.

[0061] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising:

[0062] When the computer program or instructions are executed by a processor, they implement the steps of any of the image processing methods described in the first aspect above.

[0063] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0064] In this embodiment of the disclosure, an intermediate layer is set in the system architecture of the operating system of the electronic device, which makes the operating system of the electronic device more modular. If the interaction between the camera application and the hardware is changed, only the intermediate layer needs to be modified, reducing the need to modify the more complex hardware abstraction layer in the system architecture. This makes it easier to expand and maintain the system architecture of the electronic device, and also improves the compatibility of the camera application in different operating systems.

[0065] Furthermore, since the electronic device has already cached the preview image data in the first queue of the intermediate layer based on the preview request before initiating the photo-taking request, when the photo-taking request is received, it can select image data from the cached preview image data to generate the photo image. On the one hand, the camera application can obtain the photo image by interacting with the intermediate layer, which shortens the time for the photo-taking request to be sent to the hardware abstraction layer and interact with the hardware abstraction layer, thereby shortening the time for returning the photo image. On the other hand, compared with the intermediate layer and the hardware abstraction layer consuming CPU to copy the preview image data across layers, directly transmitting the image data in the intermediate layer can further shorten the time for generating the photo image.

[0066] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0067] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0068] Figure 1 This is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 .

[0069] Figure 2 This is a schematic diagram illustrating the framework of a native operating system according to an exemplary embodiment.

[0070] Figure 3 This is a schematic diagram of the framework of an operating system according to an exemplary embodiment.

[0071] Figure 4 This is a schematic diagram of a preview stage according to an exemplary embodiment.

[0072] Figure 5 This is a schematic diagram of a frame for a photographing stage according to an exemplary embodiment.

[0073] Figure 6a This is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 .

[0074] Figure 6b This is a comparison chart of various time periods related to an image processing method according to an exemplary embodiment.

[0075] Figure 7 This is a structural diagram of an image processing apparatus according to an exemplary embodiment.

[0076] Figure 8 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0077] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0078] Figure 1 This is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 ,like Figure 1 As shown, the image processing method mainly includes the following steps:

[0079] In step 101, in response to the middleware layer of the operating system of the electronic device receiving the preview request initiated by the camera application, the preview request is transmitted to the hardware abstraction layer of the operating system; wherein, the middleware layer is used to encapsulate the hardware abstraction layer, and the camera application interacts with the hardware abstraction layer based on the middleware layer;

[0080] In step 102, in response to the intermediate layer receiving the preview image data returned by the hardware abstraction layer based on the preview request, the preview image data is cached in the first queue in the intermediate layer; wherein, the preview image data is acquired by the image sensor driven by the hardware abstraction layer based on the preview request;

[0081] In step 103, in response to the intermediate layer receiving a photo-taking request, the intermediate layer generates a photo image based on the image data selected from the first queue.

[0082] It should be noted that the image processing method can be applied to electronic devices. Here, electronic devices can include terminal devices, such as mobile terminals or fixed terminals. Mobile terminals can include devices such as mobile phones, tablets, laptops, and wearable electronic devices. Fixed terminals can include desktop computers, smart TVs, and in-vehicle systems. In other embodiments, the image processing method can also be applied to applications installed on the electronic device.

[0083] In other embodiments, the image processing method can be configured in an image processing device, which can be located in an electronic device; this disclosure does not limit this. It should be noted that the execution entity of this disclosure can be the central processing unit (CPU) in the electronic device in hardware, and related background services in the electronic device in software; this is not limited.

[0084] In step 101, the native operating system of the electronic device may include an application layer, a framework layer, and a hardware abstraction layer (HAL). The operating system in this embodiment includes an application layer, a framework layer, a hardware abstraction layer, and an intermediate layer set between the framework layer and the platform hardware abstraction layer (i.e., the hardware abstraction layer). The intermediate layer provides abstraction of the control logic and implementation details of the related software and hardware resources involved in the general camera control method, simplifying the application business design of the application layer, ignoring platform differences, and enabling portability on multiple platforms. Here, different platforms refer to different native operating systems, including Android, Hyper OS, or Symbian, etc.

[0085] The application layer can include a camera application, which can include third-party camera applications and native camera applications. Both third-party and native camera applications can provide user interface controls, such as buttons, text boxes, or sliders, and users can interact with the camera application by manipulating these user interface controls.

[0086] The framework layer can include a camera application interface, a camera service, and a framework layer interface. The camera application interface is used for data interaction between the framework layer and the application layer. The camera service is used for data interaction between the camera application interface and the framework layer interface. The framework layer interface is used for data interaction between the framework layer and the middle layer.

[0087] The middleware layer encapsulates the hardware abstraction layer (HAL) program, providing an interface to the framework layer. It transmits preview or capture requests from the camera application in the application layer to the HAL and receives image data from the HAL based on these requests. The middleware layer can also process the image data cached within it and transmit the processed image data to the application layer through the framework layer's interface.

[0088] For example, the middle layer interacts with the framework layer through a first interface (e.g., HAL3.0) and with the hardware abstraction layer through a second interface (e.g., HAL3.0).

[0089] The Hardware Abstraction Layer (HAL) is the system platform's hardware abstraction layer. It encapsulates hardware drivers, abstracting the hardware and providing interfaces to the middleware layer, thus providing a virtual hardware platform for the operating system. The HAL acts as a proxy, forwarding requests from the middleware layer to the corresponding hardware drivers to execute actions and returning the image data acquired by the hardware to the middleware layer.

[0090] For example, the hardware abstraction layer includes a platform interface and an image sensor. The image sensor is used to acquire RAW image data. RAW format is an unprocessed and uncompressed format that directly records the raw data from the image sensor. Here, the image sensor can be a semiconductor image sensor (Complementary Metal-Oxide Semiconductor, CMOS) or a charge-coupled device (CCD).

[0091] In the preview process of an electronic device, the user opens the camera application on the electronic device. The camera application's Android application package (APK) sends a preview request. The framework layer receives the preview request through the camera application interface and transmits the preview request to the middle layer through the framework layer interface. After receiving the preview request through the first interface, the middle layer transmits the preview request to the hardware abstraction layer through the second interface. The hardware abstraction layer drives the image sensor to acquire images through the camera hardware driver and caches the acquired preview image data in the middle layer. Then, it performs post-processing operations on the acquired preview image data to generate a preview image and returns the preview image to the camera application for preview.

[0092] The preview image data is RAW image data, and the preview image format can be YUV. Here, Y represents grayscale value, U represents hue, and V represents saturation.

[0093] In step 102, the hardware abstraction layer returns preview image data based on the preview request, including: the hardware abstraction layer drives the corresponding image sensor to collect preview image data based on the image identifier, preview format, preview parameters, and required frame number carried in the preview request.

[0094] For example, when the preview parameters include a preset image size, the hardware abstraction layer drives the image sensor to acquire preview image data of the width and height corresponding to the preset image size; when the required number of frames includes multiple preview images, the hardware abstraction layer drives the image sensor to acquire preview image data of each frame corresponding to the preset number of frames.

[0095] It is understandable that the preview image data returned by the hardware abstraction layer based on the image identifier carried in the preview request also carries the image identifier corresponding to the preview request. In this way, when the middle layer returns a preview image to the application, it can return the preview image corresponding to the preview request based on the image identifier.

[0096] In this embodiment of the disclosure, the electronic device includes memory for temporarily storing data and program instructions, and has high-speed access and read / write capabilities. The electronic device creates a first queue in the memory, and an intermediate layer encapsulates the memory interface, enabling the electronic device to interact with the first queue through the intermediate layer.

[0097] It should be noted that the middleware layer can provide a unified interface to manage multiple queues, or it can manage each queue separately through different interfaces. Here, in addition to the first queue, the middleware layer can also include a second queue, a third queue, and so on.

[0098] For example, the intermediate layer directly caches the preview image data to the first queue; or, the intermediate layer first caches the preview image data to the second queue, and then transmits the preview image data that meets the preset conditions in the second queue to the first queue; or, the intermediate layer directly caches one frame of preview image data collected by the first image collector to the first queue, and at the same time caches another frame of preview image data collected by the second image collector to the third queue.

[0099] The first queue is a cache queue used to cache image data returned by preview requests executed by the hardware abstraction layer. The data structure of the first queue includes: a linked list structure, an array structure, or a queue structure; or a composite data structure formed by any combination of one or more of the above data structures.

[0100] For example, when the first queue is a linked list structure, the data nodes in the first queue are connected by pointers, which can dynamically adjust the memory space size of each data node. This is suitable for scenarios where frequent insertion and deletion operations are required in the first queue. When the first queue is a queue structure, elements are added to one end of the first queue (enqueue) and removed from the other end (dequeue). This is suitable for scenarios where cached elements in the first queue are processed sequentially.

[0101] In step 103, the photo-taking process of the electronic device includes: the user clicks the photo-taking control in the camera application on the electronic device, the camera application's APK sends a photo-taking request, the intermediate layer selects image data from the cached preview image data based on the received photo-taking request, performs post-processing operations on the image data to generate a photo image, and returns the photo image to the camera application for the user to view or store in the album application.

[0102] Figure 2 This is a schematic diagram illustrating the framework of a native operating system according to an exemplary embodiment, such as... Figure 2 As shown, in the related technology, before the photo capture request is initiated, the hardware abstraction layer 22 caches the RAW image data acquired by the image sensor based on the preview request into the cache queue of the hardware abstraction layer 22. After the camera application 20 initiates the photo capture request, the camera service in the framework layer 21 communicates directly with the hardware abstraction layer across processes. Based on the photo capture request, the hardware abstraction layer selects the RAW image data from the cache queue and transmits the RAW image data to the image processor 24 in the hardware abstraction layer 22 for image processing to generate the captured image. Then, the captured image is returned to the camera application 20.

[0103] Understandably, since the hardware abstraction layer in the native operating system connects to the image sensor and the cache queue is located in the hardware abstraction layer, it is difficult to decouple from the hardware platform whether it is caching the RAW image data collected by the image sensor or retrieving RAW image data from the cache queue. In cases where different operating systems correspond to different hardware platforms, it is necessary to manually design software architectures that adapt to different hardware platforms in order to achieve the photography solution.

[0104] Even with an intermediate layer to improve software architecture compatibility, the transmission of captured images still introduces a delay in image generation because the central processing unit needs to copy the images during cross-layer transmission. For example, Figure 3 This is a schematic diagram of the framework of an operating system according to an exemplary embodiment, such as... Figure 3 As shown, in related technologies, an intermediate layer 32 is set between the framework layer 31 and the hardware abstraction layer 33, so that the hardware abstraction layer 33 is decoupled from the upper layer. When the hardware platform changes, the intermediate layer 32 is modified accordingly without changing the upper software architecture.

[0105] However, since the cache queue is located in the hardware abstraction layer 33, after the camera application 20 sends a photo capture request, it needs to retrieve RAW image data from the cache queue of the hardware abstraction layer 33. It is understandable that the cache queue in the hardware abstraction layer is isolated from the intermediate layer; therefore, when performing cross-layer communication between the intermediate layer and the hardware abstraction layer, the image data needs to be copied by the CPU to the image processor 34 in the intermediate layer 32 for image processing to generate the captured image.

[0106] Understandably, since the cache queue is located in the hardware abstraction layer, the photo-taking request needs to be sent down to the hardware abstraction layer, and the intermediate layer needs to use the CPU to obtain the RAW image data in the hardware abstraction layer, which leads to higher performance and memory overhead. The time to return the photo-taking image will also be extended due to the interaction between the intermediate layer and the hardware abstraction layer.

[0107] Based on this, in this embodiment of the present disclosure, when the camera application initiates a photo-taking request, since the first queue is located in the intermediate layer, the photo-taking request of the camera application does not need to be sent down to the hardware abstraction layer. The intermediate layer selects preview image data from the first queue based on the photo-taking request, generates a photo image from the preview image data in the intermediate layer, and then returns the photo image to the camera application.

[0108] Understandably, since the captured image is generated based on the preview image data in the intermediate layer, the interaction time between the intermediate layer and the hardware abstraction layer is shortened, thereby shortening the time to return the captured image; and since image data transmission in the intermediate layer does not require cross-layer transmission, the process of CPU participating in image data transmission is shortened, the speed of obtaining image data from the first queue is improved, and the time to return the captured image is further shortened.

[0109] In this embodiment of the disclosure, selecting image data from the first queue includes: selecting image data that meets the photo-taking request based on the required number of frames carried in the photo-taking request, image data parameters, and / or the time the photo-taking request was initiated. Here, image data parameters may include pixel size, exposure time, and ISO sensitivity, etc.

[0110] For example, when the image data parameter carried in the photo-taking request is a preset pixel size, preview image data corresponding to the preset pixel size is selected from the preview image data in the first queue as the image data; or, when the photo-taking request carries the time of initiating the photo-taking request, preview image data corresponding to the time of starting to collect preview image data and the time of initiating the photo-taking request is selected from the preview image data in the first queue as the image data.

[0111] In some embodiments, to achieve the shooting function, the image data selected from the preview image data needs to be post-processed using a corresponding image processing algorithm to generate the captured image corresponding to the shooting function. For some image post-processing, it is necessary to fuse multiple frames of image data into a single captured image. This type of shooting scene requiring the fusion of multiple frames into one image can be called a multi-frame shooting scene, such as the shooting scene corresponding to the High Dynamic Range (HDR) shooting function. The post-processing corresponding to the HDR shooting function requires generating a final captured image from three images: normal exposure, short exposure, and long exposure.

[0112] Therefore, multiple frames of image data need to be selected from the preview image data as input for post-processing. After post-processing, the final captured image is generated and returned to the camera application, so that the camera application can display the captured image.

[0113] In this embodiment of the disclosure, an intermediate layer is added to the system architecture of the native operating system of the electronic device. This makes the operating system of the electronic device more modular. If the interaction between the camera application and the hardware is changed, only the intermediate layer needs to be modified, reducing the need to modify the more complex hardware abstraction layer in the native system architecture. This makes it easier to expand and maintain the system architecture of the electronic device and also improves the compatibility of the camera application in different operating systems.

[0114] Furthermore, since the electronic device has already cached the preview image data in the first queue of the intermediate layer based on the preview request before initiating the photo-taking request, when the photo-taking request is received, it can select image data from the cached preview image data to generate the photo image. On the one hand, the camera application can obtain the photo image by interacting with the intermediate layer, which shortens the time for the photo-taking request to be sent to the hardware abstraction layer and interact with the hardware abstraction layer, thereby shortening the time for returning the photo image. On the other hand, compared with the intermediate layer and the hardware abstraction layer consuming CPU to copy the preview image data across layers, directly transmitting the image data in the intermediate layer can further shorten the time for generating the photo image.

[0115] In some embodiments, the image processing method further includes:

[0116] In response to the middle layer receiving a preview request, the cache task corresponding to the preview request is added to the second queue in the middle layer, and the cache task is transmitted to the hardware abstraction layer;

[0117] Cache the preview image data to the first queue in the intermediate layer, including:

[0118] In response to the hardware abstraction layer transmitting the preview image data corresponding to the cached task to the intermediate layer, the preview image data is cached in the first queue; or,

[0119] The preview image data is transmitted to the second queue, and the preview image data in the second queue that meets the preset conditions is cached in the first queue.

[0120] In this embodiment of the disclosure, the number of cached tasks corresponding to a preview request corresponds to the number of frames required for the preview request. For example, when the number of frames required for a preview request is 5, 5 cached tasks are added to the second queue for the preview request.

[0121] Understandably, since the speed of generating cache tasks exceeds the speed at which the hardware abstraction layer returns preview image data, adding cache tasks to the second queue in the first hardware abstraction layer allows the intermediate layer to more clearly understand that the hardware abstraction layer is collecting preview image data based on preview requests, but the preview image data has not yet been returned to the intermediate layer. This provides the hardware abstraction layer with buffer time to return preview image data and makes it easier for the intermediate layer to manage preview image data based on cache tasks.

[0122] The second queue is a storage queue used to store cached tasks that have not returned preview image data and / or to store preview image data.

[0123] In one embodiment, the second queue is used to store cached tasks. The second hardware abstraction layer directly transmits the preview image data to the first queue in the intermediate layer based on the address carried in the cached task. After confirming that the first queue has received the preview image data, the intermediate layer removes the corresponding cached task from the second queue.

[0124] In this way, the second queue of the intermediate layer can add and remove cache tasks at the first transmission rate, and the first queue can receive preview image data at the second rate, thereby completing the separate management between cache tasks and preview image data. Compared with using only one queue to add cache tasks and receive preview image data, it is clearer whether the preview image data corresponding to the cache task is returned to the intermediate layer.

[0125] In another embodiment, the second queue is used to store cached tasks and temporary preview image data. Figure 4 This is a schematic diagram of a preview stage according to an exemplary embodiment, such as... Figure 4As shown, after receiving a preview request, the intermediate layer 40 adds the corresponding cache task to the second queue 41. At this time, the frame number of the preview image data to be cached corresponding to the newly added cache task is f14. The second queue 41 also contains cache tasks corresponding to frames f9 to f13. The cache tasks corresponding to frames f9 to f14 are sequentially sent to the hardware abstraction layer 42. The hardware abstraction layer 42 drives the image sensor to sequentially collect the preview image data corresponding to frames f9 to f14. When the preview image data corresponding to frame f9 is collected, it is transmitted from the hardware abstraction layer 42 to the intermediate layer and cached in the cache task corresponding to frame f9 in the second queue 41.

[0126] In the case where the preset capacity of the first queue 43 is the space size that can cache 9 frames of preview image data, and the first queue 43 has already cached the preview image data corresponding to frame number f0 to frame number f8, if the preview image corresponding to frame number f9 is cached in the first queue 43, then the preview image data corresponding to frame number f0 that was cached first in the first queue 43 needs to be removed from the first queue 43.

[0127] It should be noted that caching preview image data that meets preset conditions in the second queue to the first queue includes: after the preview image data is transmitted to the second queue, the integrity and accuracy of the preview image data can be verified, and preview image data whose integrity meets preset conditions and / or whose accuracy meets preset conditions can be cached to the first queue. After the intermediate layer receives the preview image data, it calculates the checksum of the received preview image data and compares it with the checksum carried when the hardware abstraction layer sent the preview image data. If the checksums do not match, it indicates that the preview image data may be incomplete or contain erroneous data.

[0128] For example, verification can be performed using Cyclic Redundancy Check (CRC) or Message-Digest Algorithm 5 (MD5). Here, CRC is triggered by a polynomial. When the hardware abstraction layer sends preview image data, it divides the bit string (information field) of the preview image data by a polynomial of a predetermined length (generator polynomial), and then divides the intermediate layer by the same generator polynomial. If the remainder is 0, then no error occurred during the transmission of the preview image data. MD5 calculates a fixed-length output value, such as a 128-bit (16-byte) output value, from the preview image data using a hash algorithm. This output value is also called a message digest, representing the key information in the preview image data. After receiving the preview image data, the intermediate layer calculates the output value in the same way and compares it with the output value carried by the hardware abstraction layer in the preview image data sent. If the output values ​​are the same, then no error occurred during the transmission of the preview image data.

[0129] In this embodiment of the disclosure, the cached tasks and preview image data are stored in different queues, which makes it easier for the electronic device to manage cached tasks and preview image data with different transmission speeds. When the preview image data is directly cached in the first queue, the speed of caching the preview image data is faster. When the preview image data is indirectly cached in the first queue through the second queue, the integrity and accuracy of the cached data in the first queue can be guaranteed, providing a foundation of higher quality preview image data for subsequent selection of image data from the preview image data.

[0130] In some embodiments, caching preview image data that meets preset conditions in the second queue to the first queue includes:

[0131] In response to transmitting the preview image data to the second queue, a first data volume and a second data volume corresponding to the preview image data are determined; wherein, the first data volume is determined by the intermediate layer, and the second data volume is determined by the hardware abstraction layer;

[0132] When the difference between the first data volume and the second data volume is less than a preset data volume threshold, the preview image data is transmitted to the first queue.

[0133] In this embodiment of the disclosure, after the hardware abstraction layer drives the image sensor to acquire a frame of preview image data, it can calculate the data volume of the frame of preview image data. Here, the second data volume is the size (in bytes) of the frame of preview image data transmitted by the hardware abstraction layer to the intermediate layer; or, the total data volume of multiple frames of preview image data transmitted by the hardware abstraction layer to the intermediate layer each time.

[0134] When the intermediate layer receives preview image data transmitted from the hardware abstraction layer, it can calculate the first data volume of the received preview image data.

[0135] It is understandable that during data transmission, due to instability in the data transmission interface between the middleware layer and the hardware abstraction layer, or errors in the transmission protocol, data loss may occur when transmitting preview image data between the middleware layer and the hardware abstraction layer. Therefore, there may be a difference between the second amount of preview image data sent by the hardware abstraction layer and the first amount of preview image data received by the middleware layer.

[0136] When the difference between the first data volume and the second data volume is less than the preset data volume threshold, it indicates that the possibility of data loss between the preview image data received by the intermediate layer and the preview image data sent by the hardware abstraction layer is low.

[0137] Thus, when the difference between the first data volume and the second data volume is less than a preset data volume threshold, the preview image data is transferred from the second queue to the first queue, which can improve the integrity of the preview image data cached in the first queue.

[0138] In this embodiment of the disclosure, by comparing the first data volume determined by the intermediate layer and the second data volume determined by the hardware abstraction layer, it is possible to detect data loss or corruption that may occur during data transmission, ensuring the integrity of the preview image data cached in the first queue. This reduces the possibility of image loading failure due to data errors when subsequently obtaining image data from the first queue based on the photo-taking request to generate a photo image, thereby improving the accuracy and stability of the generated photo image.

[0139] In other embodiments, if the difference between the first data volume and the second data volume is greater than or equal to a preset data volume threshold, it indicates that the preview image data received by the intermediate layer may be incomplete. In this case, the intermediate layer can send an error report or retransmission request to the hardware abstraction layer based on the retransmission request mechanism.

[0140] The intermediate layer's request for retransmission of preview image data from the hardware abstraction layer can be immediate retransmission, delayed retransmission, or partial retransmission. Immediate retransmission means the intermediate layer immediately requests retransmission from the hardware abstraction layer upon detecting lost preview image data; delayed retransmission means the intermediate layer waits a preset time before requesting retransmission when operating system resources are insufficient; partial retransmission means the intermediate layer requests retransmission of a portion of the preview image data from the hardware abstraction layer to improve efficiency when partial data loss is detected.

[0141] In some embodiments, in response to receiving a photo-taking request from the intermediate layer, image data is selected from the first queue, including:

[0142] If the actual number of frames of preview image data in the first queue is greater than or equal to the required number of frames indicated by the photo capture request, determine the time difference between the time when the photo capture request is initiated and the time when the preview image data collection begins.

[0143] Preview image data with a time difference less than a preset duration threshold is identified as the image data corresponding to the photo-taking request.

[0144] In this embodiment of the present disclosure, the electronic device detects whether the actual number of frames of the preview image data in the first queue is greater than or equal to the required number of frames indicated by the photo capture request, and ensures that there is enough preview image data in the first queue to be available for selection and processing if the actual number of frames of the preview image data is greater than or equal to the required number of frames.

[0145] It should be noted that if the actual number of frames of the preview image data in the first queue is equal to the number of frames required by the photo-taking request, all the preview image data in the first queue can be directly identified as the image data corresponding to the photo-taking request.

[0146] Furthermore, determining the time when the photo-taking request is initiated is equivalent to determining the time when the user interacts with the user interface controls, and determining the time when the acquisition of preview image data begins is equivalent to the start time when the image sensor begins to convert light signals into electrical signals.

[0147] Understandably, if the preview image data is not cached in the first queue, the photo-taking request initiated by the application layer needs to be transmitted to the second queue before the image sensor can acquire the captured image data. However, there is a transmission time between the photo-taking request and the hardware abstraction layer, and the image sensor also needs time to wait for the acquisition to complete. Therefore, not only is the returned image inconsistent with the image requested when the photo-taking request was initiated, but there is also a delay in the returned image.

[0148] Therefore, when the time difference between the time of the photo-taking request and the time of starting to collect preview image data is less than the preset time threshold, it indicates that the cached preview image data contains the image data required when the photo-taking request is initiated. This can reduce the situation where the photo-taking image generated when the photo-taking request is initiated is a photo-taking image generated after a delay, and improve the consistency between the photo-taking image generated based on the image data and the photo-taking image required by the photo-taking request.

[0149] In this embodiment of the present disclosure, the electronic device can filter and determine the image data based on the time difference between the time of initiating the photo-taking request and the time of starting to collect the preview image data, provided that the preview image data in the first queue meets the required number of frames indicated by the photo-taking request. In this way, it can ensure that the image data accurately reflects the scene at the time of the photo-taking request, thereby improving the accuracy of generating the photo image based on the image data.

[0150] In some embodiments, the image processing method further includes:

[0151] If the actual number of frames of the preview image data in the first queue is less than the required number of frames indicated by the photo capture request, the first queue is updated based on the preview image data transmitted from the second queue to the first queue until the actual number of frames of the preview image data in the updated first queue reaches the required number of frames.

[0152] In this embodiment of the disclosure, since the second queue is still sending cache tasks to the hardware abstraction layer, the hardware abstraction layer continues to cache preview image data to the second queue based on the cache tasks.

[0153] Therefore, if the actual number of frames of the preview image data in the first queue is less than the required number of frames indicated by the photo-taking request, the first queue can be updated based on the preview image data transmitted from the second queue to the first queue until the actual number of frames of the preview image data in the updated first queue reaches the required number of frames, and then the image data can be determined from the preview image data in the updated first queue.

[0154] In some embodiments, the electronic device may calculate the difference between the actual number of frames of the preview image data in the current first queue and the required number of frames indicated by the photo capture request, and update the first queue based on the preview image data transmitted from the second queue to the first queue until the actual number of frames of the preview image data in the updated first queue makes up the difference.

[0155] For example, the actual number of frames of the preview image data in the first queue is 3 frames, and the required number of frames indicated by the photo-taking request is 5 frames. The first queue is updated based on the preview image data transmitted from the second queue to the first queue. After the second queue transmits 2 frames of preview image data to the first queue, the electronic device can determine the image data from the preview image data in the first queue based on the photo-taking request.

[0156] In this embodiment of the present disclosure, regardless of whether the preview image data in the initial first queue meets the required number of frames for the photo-taking request, the electronic device can update the first queue through the preview image data transmitted from the second queue to the first queue, so that the preview image data in the first queue meets the required number of frames for the photo-taking request, thereby improving the reliability of determining the image data and improving the completeness of generating the photo image based on the image data.

[0157] In some embodiments, the image processing method further includes:

[0158] At the first moment, in response to the middle layer receiving a photo-taking request, the image data corresponding to the photo-taking request is selected from the first queue of the middle layer;

[0159] In response to the fact that the first preview image data does not meet the preset conditions corresponding to the photo capture request at the first moment, the photo capture request is suspended; wherein, the first preview image data is the preview image data returned based on the hardware abstraction layer before the first moment;

[0160] The first queue is updated based on the second preview image data returned by the hardware abstraction layer at the second time step; wherein the second time step is after the first time step.

[0161] In response to the completion of the update for the first queue, the photo-taking request is re-executed, and image data that meets the preset conditions corresponding to the photo-taking request is selected from the updated first queue.

[0162] In some embodiments, the camera service is used for communication between the camera application interface and the framework layer interface, and the camera service communicates directly with the hardware abstraction layer across processes. The camera service has a request queue. When the electronic device is in the preview stage, preview requests can be added to the request queue, and a separate request thread sequentially sends the preview requests from the request queue to the hardware abstraction layer. When a user-triggered photo-taking operation is detected, a photo-taking request is generated based on the camera application's camera service. This photo-taking request can be added to the request queue first, and then the separate request thread sends the photo-taking request from the request queue to the hardware abstraction layer. This forms a standard camera request loop system.

[0163] Here, the standard camera request loop strictly follows the sequence of preview request - capture request - preview request. When the user clicks the capture control on the electronic device, the camera application generates a capture request. For image processing algorithms corresponding to multi-frame shooting scenarios, if the image data corresponding to the capture request is not selected in the cache queue of the hardware abstraction layer, the hardware abstraction layer can only execute the capture request. Since a capture request corresponds to multiple capture tasks, and each capture task processes one frame of image, and the hardware abstraction layer processes them frame by frame in sequence, the preview request following the capture request is delayed.

[0164] For example, when a user clicks the camera control once, a camera request is generated, numbered 1. The subsequent preview request is numbered 2. To handle post-processing for this photo, the hardware abstraction layer (HAL) needs to process three camera tasks before processing the preview request. Therefore, from the HAL's perspective, the preview request is the fourth request to be processed. This means that the acquisition of the preview image data for this frame can only occur after the previous three camera tasks have been completed. At this point, the user will experience a noticeable preview lag in the preview interface (which can be understood as the image display interface), thus affecting the display frame rate.

[0165] Based on this, in this embodiment of the present disclosure, in response to the first preview image data not meeting the preset conditions corresponding to the photo-taking request at the first moment, the photo-taking request can be paused and the first queue can be updated. The image data corresponding to the preset conditions of the photo-taking request can be selected from the updated first queue. In this way, the effect of delaying the selection of image data can be achieved without hindering the subsequent preview requests from the intermediate layer to the hardware abstraction layer.

[0166] In some embodiments, the electronic device includes memory for temporarily storing data and program instructions, and has high-speed access and read / write capabilities. The electronic device creates a first queue in the memory, and an intermediate layer encapsulates the memory interface, enabling the electronic device to interact through the intermediate layer and the first queue within it. Here, the first queue is a cache queue used to cache preview image data returned by preview requests executed by the hardware abstraction layer.

[0167] The preset conditions include at least one of the following: preset resolution, preset size, preset required frame rate, or preset exposure time.

[0168] In some embodiments, when the actual number of frames of the first preview image data in the first queue is less than the preset required number of frames corresponding to the photo capture request, it is determined that the first preview image data does not meet the preset conditions corresponding to the photo capture request. For example, when the first preview image data has 5 frames and the preset required number of frames is 6 frames, it is determined that the first preview image data does not meet the preset conditions corresponding to the photo capture request.

[0169] In other embodiments, when the resolution corresponding to the first preview image data does not match the preset resolution corresponding to the photo-taking request, it is determined that the first preview image data does not meet the preset conditions corresponding to the photo-taking request. For example, before the first moment, the camera application sends a preview task to the hardware abstraction layer based on a default resolution (e.g., 4*3). After the hardware abstraction layer executes the preview task, the resolution corresponding to the first preview image data returned by the hardware abstraction layer to the first queue is the default resolution. However, after the user adjusts the camera application's default resolution to a specified resolution (e.g., 16*9), the preset resolution corresponding to the photo-taking request becomes the specified resolution, thus it can be determined that the first preview image data does not meet the preset conditions corresponding to the photo-taking request.

[0170] In other embodiments, exposure time affects the quality of image data. For example, a longer exposure time allows more light to enter the camera, resulting in excessive brightness and loss of detail in the image data, leading to image distortion. When the exposure time corresponding to the first preview image data is greater than the preset exposure time, it is determined that the first preview image data does not meet the preset conditions corresponding to the photo-taking request. For example, before the first moment, the user accidentally touches the camera application, causing the camera's exposure time to be adjusted from the default exposure time to the exposure time indicated by the accidental touch, and the exposure time indicated by the accidental touch is greater than the preset exposure time. The camera application sends a preview task to the hardware abstraction layer based on the exposure time indicated by the accidental touch. After the hardware abstraction layer executes the preview task, the exposure time corresponding to the first preview image data returned by the hardware abstraction layer to the first queue is the exposure time indicated by the accidental touch. It can be determined that the image data quality of the first preview image data is low, and thus it is determined that the first preview image data does not meet the preset conditions corresponding to the photo-taking request.

[0171] In other embodiments, when the actual number of frames of the first preview image data is greater than or equal to the preset required number of frames corresponding to the photo capture request, it can be further determined whether the exposure time of the first preview image data is greater than the preset exposure time, so as to evaluate the image data quality of the first preview image data. When the exposure time of the first preview image data is greater than the preset exposure time, it is determined that the first preview image data does not meet the preset conditions corresponding to the photo capture request.

[0172] In other embodiments, when the exposure time of the first preview image data is less than or equal to a preset exposure time, it can be further determined whether the resolution corresponding to the first preview image data matches the preset resolution. When the resolution corresponding to the first preview image data does not match the preset resolution, it is determined that the first preview image data does not meet the preset conditions corresponding to the photo-taking request.

[0173] It is understandable that if the first preview image data does not meet the preset conditions corresponding to the photo-taking request, it is determined that selecting the image data corresponding to the photo-taking request at the first moment will fail.

[0174] In this embodiment of the disclosure, if the shooting scene is a multi-frame shooting scene, after the intermediate layer receives the shooting request, it can parse the shooting request into multiple shooting tasks according to the required number of frames indicated by the shooting request, so as to select the image data of each frame corresponding to each shooting task from the first queue.

[0175] Here, the first preview image data not meeting the preset conditions corresponding to the photo-taking request may mean that all of the first preview image data does not meet the preset conditions corresponding to the photo-taking request, or it may mean that some of the first preview image data does not meet the preset conditions corresponding to the photo-taking request. This disclosure does not limit this.

[0176] It should be noted that pausing the execution of the photo-taking request means controlling the photo-taking request to be in the middle layer.

[0177] In some embodiments, the intermediate layer provides a first request queue. In response to the failure to select image data corresponding to the photo-taking request at the first moment, the photo-taking request can be added to the first request queue to control that the photo-taking request is not sent from the intermediate layer to the hardware abstraction layer. Here, the first request queue is a storage queue used to store tasks; the specific implementation of the storage queue can be varied, such as sequential storage (e.g., array) or linked storage (e.g., linked list).

[0178] In other embodiments, the intermediate layer includes a stack that can be added to the stack in response to a failure to select image data corresponding to the photo-taking request at the first moment. Since there is only one push operation for the photo-taking request and no other push operations for other tasks (or requests), the photo-taking request can be kept on the stack to prevent it from being sent from the intermediate layer to the hardware abstraction layer.

[0179] It should be noted that, in order for paused photo capture requests to continue selecting image data from the first queue, while the photo capture request is paused, the hardware abstraction layer will continue to execute the preview task that failed to execute at the first moment. The camera application will also send the preview request initiated after the photo capture request is initiated to the middle layer, waiting for the middle layer to send the preview request initiated after the photo capture request is initiated to the hardware abstraction layer for processing, so as to obtain the updated first queue.

[0180] It should be explained that when the photo capture request is paused, it does not prevent subsequent preview requests from being sent from the intermediate layer to the hardware abstraction layer. The hardware abstraction layer can continue to execute preview requests initiated after the photo capture request is initiated. This reduces the latency of the hardware abstraction layer in processing subsequent preview requests, thus not affecting the screen's display frame rate.

[0181] Understandably, the updated first queue can be obtained by updating the image data acquired by the preview task successfully executed at the second moment. Specifically, the image data acquired at the second moment can update and replace the unselected image data in the first queue.

[0182] Here, the second time point is after the first time point, and the second time point can be any time point after the first time point.

[0183] Since the second moment is any moment after the first moment, the preview task successfully executed at the second moment can be a preview task that was not successfully executed at the first moment, or it can be a preview task indicated by a preview request initiated after the photo-taking request is initiated. This disclosure does not limit this.

[0184] In some embodiments, when a portion of the image data corresponding to the photo-taking request is selected at the first moment, the second moment can be the moment following the first moment, so as to ensure the continuity between the image data stored in the updated first queue and the image data selected at the first moment.

[0185] It should be noted that once the update of the first queue is detected as complete, the photo-taking request can continue to be executed, and the image data corresponding to the photo-taking request can be selected from the updated first queue. Therefore, the image data corresponding to the photo-taking request can be obtained without sending the photo-taking request to the hardware abstraction layer, which helps to improve the generation speed of the target image.

[0186] In some embodiments, the Android system introduces a buffer management interface (camerahal3.buffer management APLs) for the hardware abstraction layer. When this function is enabled, the hardware abstraction layer controls the frequency of requests sent by the camera application. Therefore, each manufacturer needs to manage the time when the hardware abstraction layer executes requests to ensure the display frame rate.

[0187] Since the camera application and the middleware layer belong to different processes, cross-process communication can be achieved through the binder mechanism. Therefore, the request loop system has a first preset thread (binder request thread). Here, the binder request thread is used to send preview requests (or photo requests) using the binder interface and add the preview requests (or photo requests) to the middleware layer's request queue.

[0188] The middleware layer and the hardware abstraction layer belong to different processes. Therefore, the request loop system has a second preset thread (M request thread). Here, the M request thread is used to send the preview requests (or photo requests) stored in the request queue to the hardware abstraction layer.

[0189] In this embodiment of the present disclosure, at a first moment, in response to the intermediate layer of the operating system of the electronic device receiving a photo-taking request initiated by the camera application, image data corresponding to the photo-taking request is selected from the first queue of the intermediate layer; in response to the first preview image data not meeting the preset conditions corresponding to the photo-taking request at the first moment, the execution of the photo-taking request is suspended; based on the second preview image data returned by the hardware abstraction layer at a second moment, the first queue is updated; in response to the completion of the update of the first queue, the photo-taking request is re-executed, and image data meeting the preset conditions corresponding to the photo-taking request is selected from the updated first queue; in response to the successful acquisition of the image data corresponding to the photo-taking request, a target image is generated based on the image data corresponding to the photo-taking request.

[0190] On the one hand, if the first preview image data does not meet the preset conditions corresponding to the photo-taking request, the photo-taking request can be paused, and the second preview image data returned at the second moment can be used to update the first queue, so that image data corresponding to the preset conditions of the photo-taking request can be selected again from the updated first queue. In this way, the effect of delaying the selection of image data can be achieved, which helps to improve the efficiency of generating the target image.

[0191] On the other hand, the photo-taking request can be paused at the intermediate layer, without having to send the photo-taking request to the hardware abstraction layer for processing. In this way, even in multi-frame photo-taking scenarios, the photo-taking request will not prevent subsequent preview requests from being sent from the intermediate layer to the hardware abstraction layer, reducing the latency of the hardware abstraction layer in processing subsequent preview requests, reducing the stuttering phenomenon in the preview interface, and thus ensuring the display frame rate.

[0192] In some embodiments, generating a captured image based on image data in an intermediate layer includes:

[0193] In response to the successful acquisition of image data, the intermediate layer transmits the image data from the first queue to the image processor for image processing to generate the captured image;

[0194] The image processor connects to the middle layer via a data transmission interface.

[0195] Figure 5 This is a schematic diagram of a frame during the photographing stage according to an exemplary embodiment, such as... Figure 5 As shown, after receiving a photo-taking request, the intermediate layer 50 retrieves frames from the first queue 51 based on at least one photo-taking task indicated by the photo-taking request. Here, the data of the at least one photo-taking task indicated by the photo-taking request corresponds to the frame retrieval requirement of the photo-taking request. For example, when the frame retrieval requirement of the photo-taking request is 5 frames, the photo-taking request indicates 5 photo-taking tasks. Based on the 5 photo-taking tasks, 5 frames of preview image data corresponding to frames f25 to f29 can be selected as image data from the 9 frames of preview image data corresponding to frames f21 to f29 cached in the first queue 51. At this time, the second queue 53 can also cache 5 cached tasks corresponding to frames f30 to f34 for retrieving 5 frames of preview image data.

[0196] It should be noted that the five frames of image data are determined from the preview image data in the first queue. These five frames of image data can be locked using a mutex lock or process lock to reduce the possibility of other imaging processes selecting any image data from the five frames, which could cause inter-process conflicts and improve the reliability of generating the captured image. Then, the five frames of image data are transmitted to the image processor 52 for image processing to generate the captured image.

[0197] In this embodiment, the image processor includes image processing-related software algorithms and hardware algorithms, and is used to coordinate the flow of image data between algorithms. The software algorithms address different image processing needs, such as distortion correction, purple fringing removal, and multi-frame fusion algorithms. The operation of the software algorithms depends on the system's central processing unit (CPU) and graphics processing unit (GPU). The hardware algorithms are based on an image signal processor (ISP), organizing the ISP chip components to complete the corresponding image processing requirements.

[0198] It should be noted that the image processor connects to the middle layer via a data transmission interface. Image data can be transferred from the first queue to the image processor for processing through this interface. This reduces CPU involvement in memory copying, conserves memory to the maximum extent, reduces CPU usage, and thus improves the efficiency of generating captured images.

[0199] To better understand the above one or more embodiments, examples of embodiments of this disclosure are as follows:

[0200] Figure 6a This is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 .like Figure 6a As shown:

[0201] In step 601, in response to the middleware layer of the operating system of the electronic device receiving a preview request initiated by the camera application, the preview request is transmitted to the hardware abstraction layer of the operating system.

[0202] In step 602, in response to the middle layer receiving a preview request, the cache task corresponding to the preview request is added to the second queue in the middle layer, and the cache task is transmitted to the hardware abstraction layer;

[0203] In step 603, in response to the hardware abstraction layer transmitting the preview image data corresponding to the cached task to the second queue, the preview image data in the second queue that meets the preset conditions is cached to the first queue;

[0204] In step 604, in response to the middle layer receiving a photo-taking request, the time difference between the time when the photo-taking request is initiated and the time when the preview image data is started to be collected is determined, and the preview image data corresponding to the time difference that is less than a preset duration threshold is determined as the image data corresponding to the photo-taking request.

[0205] In step 605, in response to successfully acquiring image data, the intermediate layer transmits the image data from the first queue to the image processor through the data transmission interface for image processing to generate a captured image;

[0206] In step 606, the captured image is returned to the camera application based on the intermediate layer.

[0207] In this way, the CPU's involvement in memory copying is reduced from the time the camera application initiates a photo-taking request to the time the photo is returned to the camera application, thus shortening the time for returning the photo. Furthermore, since the first queue is located in the intermediate layer, the photo-taking request does not need to be sent down to the hardware abstraction layer, which shortens the time for interaction between the intermediate layer and the hardware abstraction layer, further shortening the time for returning the photo.

[0208] Figure 6b This is a comparison chart of various time periods related to an image processing method according to an exemplary embodiment, such as... Figure 6b As shown: Taking a photo-taking scenario based on a multi-frame noise reduction (MFNR) algorithm that synthesizes 8 frames as an example, the first item is the various durations of image processing based on the native operating system, and the second item is the various durations of image processing in the embodiments of this disclosure.

[0209] It should be noted that when the camera performs MFNR shooting, it continuously acquires 8 frames of image data. These images have different exposure times, ISO settings, etc., which can effectively capture image information under different exposures. MFNR analyzes the noise distribution, brightness and color information in each frame of image data, automatically identifies and excludes noisy image data, and synthesizes the remaining image data to generate a high-quality composite image.

[0210] The first interval duration is the minimum interval time corresponding to the continuous acquisition of two frames of image data; the first frame acquisition duration is the time for selecting one frame of image data from the first queue and transmitting it to the image processor; the total frame acquisition duration is the time for selecting all eight frames of image data from the first queue and transmitting them to the image processor; and the second interval duration is the time from when the camera application sends a photo request to when the generated photo image is refreshed and displayed on the camera application's screen.

[0211] The average of five test results was taken. The difference between the average of the second and first items in the first interval was 7.13 milliseconds (ms), meaning that the present invention reduces the interval between acquiring two consecutive frames of image data by 65.11% compared to related technologies. The difference between the average of the second and first items in the first frame acquisition time was 29.65 milliseconds, meaning that the present invention reduces the acquisition time of the first frame of image data by 82.61% compared to related technologies. The difference between the average of the second and first items in the total frame acquisition time was 51.33 milliseconds, meaning that the present invention reduces the acquisition time of 8 frames of image data by 46.18% compared to related technologies. The difference between the average of the second and first items in the second interval was 40.41 milliseconds, meaning that the present invention reduces the time between the camera application issuing a photo request and the generation and display of the photo image on the camera application's screen by 35.88% compared to related technologies.

[0212] Therefore, it can be seen that the image processing method of this disclosure embodiment has significant improvements in various performance indicators compared with the image processing method of the native operating system.

[0213] Figure 7 This is a structural diagram of an image processing apparatus according to an exemplary embodiment. For example... Figure 7 As shown, the image processing apparatus 700 includes:

[0214] The transmission module 701 is configured to transmit the preview request to the hardware abstraction layer of the operating system of the electronic device in response to the middle layer of the operating system receiving a preview request initiated by the camera application; wherein the middle layer is used to encapsulate the hardware abstraction layer, and the camera application interacts with the hardware abstraction layer based on the middle layer;

[0215] The caching module 702 is configured to cache the preview image data in a first queue in the intermediate layer in response to the intermediate layer receiving preview image data returned by the hardware abstraction layer based on the preview request; wherein the preview image data is acquired by the image sensor driven by the hardware abstraction layer based on the preview request.

[0216] The generation module 703 is configured to, in response to the intermediate layer receiving a photo-taking request, select image data from the first queue and generate a photo image based on the image data in the intermediate layer.

[0217] In some embodiments, the image processing apparatus 700 further includes:

[0218] The task addition module is configured to, in response to the middle layer receiving the preview request, add the cache task corresponding to the preview request to the second queue in the middle layer, and transmit the cache task to the hardware abstraction layer;

[0219] The cache module 702 further includes:

[0220] The first caching module is configured to cache the preview image data in the first queue in response to the hardware abstraction layer transmitting the preview image data corresponding to the caching task to the intermediate layer; or...

[0221] The preview image data is transmitted to the second queue, and the preview image data in the second queue that meets the preset conditions is cached in the first queue.

[0222] In some embodiments, the first cache module is further configured to:

[0223] In response to transmitting the preview image data to the second queue, a first data volume and a second data volume corresponding to the preview image data are determined; wherein, the first data volume is determined by the intermediate layer, and the second data volume is determined by the hardware abstraction layer;

[0224] When the difference between the first data volume and the second data volume is less than a preset data volume threshold, the preview image data is transmitted to the first queue.

[0225] In some embodiments, the image processing apparatus 700 further includes:

[0226] The first determining module is configured to determine the time difference between the time of initiating the photo-taking request and the time of starting to collect the preview image data when the actual number of frames of the preview image data in the first queue is greater than or equal to the required number of frames indicated by the photo-taking request.

[0227] The second determining module is configured to determine the preview image data corresponding to the time difference that is less than a preset duration threshold as the image data corresponding to the photo taking request.

[0228] In some embodiments, the image processing apparatus 700 further includes:

[0229] The first update module is configured to update the first queue based on the preview image data transmitted from the second queue to the first queue when the actual number of frames of the preview image data in the first queue is less than the required number of frames indicated by the photo capture request, until the actual number of frames of the preview image data in the updated first queue reaches the required number of frames.

[0230] In some embodiments, the image processing apparatus 700 further includes:

[0231] The selection module is configured to, at a first moment, in response to the intermediate layer receiving the photo-taking request, select the image data corresponding to the photo-taking request from the first queue of the intermediate layer;

[0232] The pause module is configured to pause the execution of the photo-taking request in response to the first preview image data not meeting the preset conditions corresponding to the photo-taking request at the first moment; wherein, the first preview image data is the preview image data returned based on the hardware abstraction layer before the first moment;

[0233] The second update module is configured to update the first queue based on the second preview image data returned by the hardware abstraction layer at a second time; wherein the second time is after the first time.

[0234] The execution module is configured to re-execute the photo-taking request in response to completing the update of the first queue, and select image data from the updated first queue that meets the preset conditions corresponding to the photo-taking request based on the photo-taking request.

[0235] In some embodiments, the generation module 703 is further configured to:

[0236] In response to successfully acquiring the image data, the intermediate layer transmits the image data from the first queue to the image processor for image processing to generate the captured image;

[0237] The image processor is connected to the intermediate layer via a data transmission interface.

[0238] Figure 8This is a structural block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0239] Reference Figure 8 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0240] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0241] Memory 804 is configured to store various types of data to support operation on electronic device 800. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, and videos. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0242] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0243] Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When electronic device 800 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0244] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0245] Input / output interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0246] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or one of its components, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0247] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0248] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0249] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including executable instructions or a computer program, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0250] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the image processing methods described above in the embodiments of this disclosure. For example, the method includes:

[0251] In response to the middleware layer of the operating system of the electronic device receiving a preview request initiated by the camera application, the preview request is transmitted to the hardware abstraction layer of the operating system; wherein, the middleware layer is used to encapsulate the hardware abstraction layer, and the camera application interacts with the hardware abstraction layer based on the middleware layer;

[0252] In response to the intermediate layer receiving preview image data returned by the hardware abstraction layer based on the preview request, the preview image data is cached in the first queue in the intermediate layer; wherein, the preview image data is acquired by the image sensor driven by the hardware abstraction layer based on the preview request;

[0253] In response to the intermediate layer receiving a photo-taking request, the intermediate layer generates a photo image based on the image data selected from the first queue.

[0254] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image processing methods described above in this disclosure.

[0255] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0256] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: In response to the middleware layer of the operating system of the electronic device receiving a preview request initiated by the camera application, the preview request is transmitted to the hardware abstraction layer of the operating system; wherein, the middleware layer is used to encapsulate the hardware abstraction layer, and the camera application interacts with the hardware abstraction layer based on the middleware layer; In response to the intermediate layer receiving preview image data returned by the hardware abstraction layer based on the preview request, the preview image data is cached in the first queue in the intermediate layer; wherein, the preview image data is acquired by the image sensor driven by the hardware abstraction layer based on the preview request; In response to the intermediate layer receiving a photo-taking request, the intermediate layer generates a photo image based on the image data selected from the first queue.

2. The method according to claim 1, characterized in that, The method further includes: In response to the middle layer receiving the preview request, the cache task corresponding to the preview request is added to the second queue in the middle layer, and the cache task is transmitted to the hardware abstraction layer; The step of caching the preview image data to the first queue in the intermediate layer includes: In response to the hardware abstraction layer transmitting the preview image data corresponding to the cached task to the intermediate layer, the preview image data is cached in the first queue; or, The preview image data is transmitted to the second queue, and the preview image data in the second queue that meets the preset conditions is cached in the first queue.

3. The method according to claim 2, characterized in that, The step of caching preview image data that meets preset conditions in the second queue to the first queue includes: In response to transmitting the preview image data to the second queue, a first data volume and a second data volume corresponding to the preview image data are determined; wherein, the first data volume is determined by the intermediate layer, and the second data volume is determined by the hardware abstraction layer; When the difference between the first data volume and the second data volume is less than a preset data volume threshold, the preview image data is transmitted to the first queue.

4. The method according to claim 1, characterized in that, The method further includes: If the actual number of frames of the preview image data in the first queue is greater than or equal to the required number of frames indicated by the photo-taking request, the time difference between the time when the photo-taking request is initiated and the time when the preview image data is started to be collected is determined. The preview image data corresponding to the time difference that is less than a preset time threshold is determined as the image data corresponding to the photo-taking request.

5. The method according to claim 4, characterized in that, The method further includes: If the actual number of frames of the preview image data in the first queue is less than the required number of frames indicated by the photo-taking request, the first queue is updated based on the preview image data transmitted from the second queue to the first queue until the actual number of frames of the preview image data in the updated first queue reaches the required number of frames.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: At the first moment, in response to the middle layer receiving the photo-taking request, the image data corresponding to the photo-taking request is selected from the first queue; In response to the fact that the first preview image data does not meet the preset conditions corresponding to the photo-taking request at the first moment, the execution of the photo-taking request is suspended; wherein, the first preview image data is the preview image data returned based on the hardware abstraction layer before the first moment; The first queue is updated based on the second preview image data returned by the hardware abstraction layer at the second time point; wherein, the second time point is after the first time point. In response to the completion of the update for the first queue, the photo-taking request is re-executed, and image data that meets the preset conditions corresponding to the photo-taking request is selected from the updated first queue.

7. The method according to any one of claims 1 to 5, characterized in that, The step of generating a photographed image based on the image data selected from the first queue in the intermediate layer includes: In response to successfully acquiring the image data, the intermediate layer transmits the image data from the first queue to the image processor for image processing to generate the captured image; The image processor is connected to the intermediate layer via a data transmission interface.

8. An image processing apparatus, characterized in that, include: A transmission module is configured to, in response to a preview request initiated by a camera application received by the middleware layer of the operating system of an electronic device, transmit the preview request to the hardware abstraction layer of the operating system; wherein, the middleware layer is used to encapsulate the hardware abstraction layer, and the camera application interacts with the hardware abstraction layer based on the middleware layer; A caching module is configured to cache the preview image data in a first queue in the intermediate layer in response to the intermediate layer receiving preview image data returned by the hardware abstraction layer based on the preview request; wherein the preview image data is acquired by the image sensor driven by the hardware abstraction layer based on the preview request. The generation module is configured to generate a photographed image based on image data selected from the first queue in response to the intermediate layer receiving a photographing request.

9. An electronic device, characterized in that, The electronic device includes at least: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image processing method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the image processing method according to any one of claims 1 to 7.

Images