Camera request processing method, and related apparatus

Abstract

A camera request processing method, and a related apparatus. In the method, an electronic device may divide a buffer, which is used for storing camera requests, into two parts, wherein one part stores common camera requests, and the other part stores special camera requests. When a camera in the electronic device is started, a plurality of camera requests are allocated to the buffer for storing common camera requests. Upon receiving a zoom request, the electronic device may directly allocate the zoom request to the buffer for storing special camera requests, and issue the zoom request to an HAL. Since an FWK immediately issues the zoom request to the HAL, after receiving the zoom request issued by the FWK, the HAL may make the zoom request take effect on a generated image frame as soon as possible, thereby shortening the time that the electronic device takes to responds to a zoom operation performed by a user, that is, the touch sensitivity for the user performing the zoom operation can be improved, thereby improving the user experience.

Description

Camera request processing method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 29, 2023, with application number 202310952242.X and application name “A Camera Request Processing Method and Related Devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a camera request processing method and related devices. Background Art

[0003] Currently, electronic devices such as mobile phones and tablets all have camera applications. Users can use the camera application on their mobile phones to take photos and record videos. As the photo and video recording functions of camera applications on mobile phones and other electronic devices become increasingly powerful, more and more shooting modes are available on mobile phones and other electronic devices. Users can change the camera's shooting mode in the camera shooting interface, for example, by changing the camera's current focal length or adding filters. After the user changes the camera's shooting mode, the electronic device's shooting interface will display a preview image frame in the new shooting mode in response to the user's operation.

[0004] However, when a user changes the camera shooting mode in the camera shooting interface, the electronic device will experience a delay in responding to the user's operation. For example, when a user operates the camera to zoom, the electronic device will experience a delay before the zoomed image frame is displayed in the camera shooting interface, thus poor user experience.

[0005] Therefore, how to reduce the response delay of the electronic device to the user's operation of changing the shooting mode is an urgent problem to be solved.

[0006] Summary of the Invention

[0007] The present application provides a camera request processing method and related devices. Through the camera request processing method provided by the present application, the electronic device can reduce the response time to the user's operation of changing the shooting mode, thereby improving the user experience.

[0008] In a first aspect, the present application provides a camera request method, which can be applied to an electronic device equipped with a camera module. The method may include: detecting a user's operation of changing camera parameters, and issuing a first camera request; determining that the camera parameters in the first camera request are different from the camera parameters in the second camera request, and allocating the first camera request to a first cache buffer in a first cache area; a storage area in the electronic device for storing camera requests is divided into a first cache area and a second cache area, the first cache area is used to store the first camera request, the camera parameters including one or more of a zoom ratio, a focus parameter, an exposure parameter, and a filter parameter, the second camera request being a camera request obtained by the electronic device before obtaining the first camera request; obtaining a first image frame according to the camera parameters in the first camera request; and displaying the first image frame.

[0009] When the method provided in the first aspect is implemented, multiple camera requests for obtaining preview images issued when the camera in the electronic device is turned on are only stored in the second buffer area, and the first buffer area is only allocated to camera requests in which camera parameters have changed. When the user changes the zoom ratio, or performs focus, exposure, or adds filters in the camera shooting interface, the electronic device can directly allocate the camera request generated based on the user operation to the special camera request buffer area. The application framework layer does not need to wait for a buffer to be cleared before allocating it to the buffer, and the application framework layer does not need to wait for the HAL to be able to process the next request before allocating a buffer to the camera request. In this way, the response time of the electronic device to the user's operation of changing the zoom ratio, or performing focus, or exposure, or adding filters can be reduced, which can improve the response efficiency of user operations.

[0010] In one possible implementation, the first buffer area includes q buffers and the second buffer area includes p buffers. The method may further include: determining values ​​of q and p based on a camera mode of a camera application in the electronic device. In this way, the electronic device can determine different values ​​of p and q based on different modes.

[0011] In one possible implementation, the values ​​of q and p are determined based on the camera mode of the camera application in the electronic device, including: when the camera mode of the camera application is the photo mode, the value of q is determined to be a first value, and the value of p is determined to be a second value; when the camera mode of the camera application is the video mode, the value of q is determined to be a third value, and the value of p is determined to be a fourth value, the sum of the third value and the fourth value is the fifth value, the sum of the first value and the second value is the sixth value, and the fifth value and the sixth value are equal. In this way, the electronic device can determine different values ​​of p and q based on different modes. In this way, it can ensure that the camera can smoothly send camera requests in different camera modes.

[0012] In one possible implementation, upon detecting a user operation to change camera parameters, before issuing a first camera request, the method may further include: the electronic device detecting an operation to launch a camera application and launching the camera application; the camera application in the electronic device issuing multiple camera requests, the multiple camera requests including a second camera request; if there is an empty buffer in the second buffer area, allocating the second camera request to the second buffer, where the second buffer is an empty buffer in the second buffer area; if there is no empty buffer in the second buffer area, the electronic device waits for the second buffer in the second buffer area to be cleared before allocating the second camera request to the second buffer area. In this way, the second camera request will not be issued to the first buffer area. When the electronic device issues the first camera request, the first buffer area is empty, and the electronic device can quickly allocate a buffer for the first camera request without having to wait for the buffer area to be cleared before allocating a buffer.

[0013] In one possible implementation, after determining that the camera parameters in the first camera request are different from the camera parameters in the second camera request, the method may further include: if there is an empty buffer in the second buffer area, allocating the first camera request to a second buffer in the second buffer area, where the second buffer is an empty buffer. In this way, more empty buffers can be reserved for subsequent camera requests with different camera parameters.

[0014] In one possible implementation, the user's operation of changing the first camera parameter is an operation of changing the zoom ratio. The method may further include: the camera application displays a first preview interface, wherein the first preview interface displays the second image frame, and the zoom ratio corresponding to the second image frame is the first zoom ratio; and the camera detects the user's operation of changing the zoom ratio and displays a second preview interface, wherein the second preview interface displays the first image frame, and the zoom ratio corresponding to the first image frame is the second zoom ratio. In this way, the user can view the preview interfaces before and after the zoom ratio change on the electronic device.

[0015] In one possible implementation, an electronic device includes a request module and a cache module. Determining that camera parameters in a first camera request differ from camera parameters in a second camera request and allocating the first camera request to a first cache buffer in a first cache area may include: the request module determining that the camera parameters in the first camera request differ from camera parameters in the second camera request; the request module sending the first camera request to the cache module; and the cache module allocating the first camera request to the first cache buffer in the first cache area. In this way, the cache module can quickly allocate the first camera request to the first buffer without waiting.

[0016] In one possible implementation, the electronic device may further include a camera processing module. After the cache module allocates the first camera request to a first buffer in the first buffer area, the method may further include: the cache module sending the first camera request to the camera processing module. In this way, the cache module can quickly send the first camera request to the camera processing module.

[0017] Acquiring the first image frame according to the camera parameters in the first camera request may include: the camera processing module sending the camera parameters in the first camera request to the camera module; and the camera module acquiring the first image frame according to the camera parameters in the first camera request. In this way, the electronic device can acquire the first image frame corresponding to the first camera request.

[0018] In a possible implementation, the electronic device further includes a cache module and a camera processing module. After allocating the second camera request to the second buffer, the method may further include: the cache module sending the second camera request to the camera processing module.

[0019] In one possible implementation, the electronic device may further include a camera application, a display module, and after the cache module sends the second camera request to the camera processing module, the method may further include: the camera processing module sending the camera parameters in the second camera request to the camera module and obtaining a second image frame obtained by the camera module according to the camera parameters in the second camera request; the camera processing module sending the second image frame to the display module; the display module displaying the second image frame; and the camera application displaying the second image frame. In this way, the electronic device can obtain the second image frame through various modules and display the second image frame.

[0020] In a second aspect, an electronic device is provided, which may include one or more cameras, a display, one or more processors and one or more memories; wherein the one or more cameras, the display, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method involved in any possible implementation method of the first aspect.

[0021] According to a third aspect, an electronic device is provided. The electronic device may include one or more functional modules, and the one or more functional modules are used for the method involved in any possible implementation of the first aspect.

[0022] In a fourth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method involved in any possible implementation of the first aspect.

[0023] In a fifth aspect, a computer-readable storage medium is provided, comprising instructions, which, when executed on an electronic device, enable the electronic device to execute the method involved in any possible implementation of the first aspect.

[0024] In a sixth aspect, a computer program product is provided. When the program product is run on an electronic device, the electronic device executes the method involved in any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of a user interface provided in an embodiment of the present application;

[0026] Figures 2A-2C are a set of user interface schematic diagrams provided in an embodiment of the present application;

[0027] FIG3 is a schematic diagram of the software and hardware architecture of an electronic device provided in an embodiment of the present application;

[0028] FIG4 is a schematic diagram of the time required for a zoom request to be sent to the HAL layer and the time required for an electronic device to successfully respond to a zoom operation, provided by an embodiment of the present application;

[0029] FIG5 is a schematic diagram of the software and hardware architecture of an electronic device provided in an embodiment of the present application;

[0030] FIG6 is a schematic diagram of software module interaction of a camera request processing method provided in an embodiment of the present application;

[0031] Figures 7 to 9 are schematic diagrams of a set of user interfaces provided in an embodiment of the present application;

[0032] FIG10 is a schematic diagram of a camera request processing method provided in an embodiment of the present application;

[0033] FIG11 is a schematic diagram showing the time it takes for another electronic device to send a zoom request to the HAL layer and the time it takes for the electronic device to successfully respond to a zoom operation, provided by an embodiment of the present application;

[0034] FIG12 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0036] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0037] Currently, electronic devices such as mobile phones have an increasing number of shooting modes. Users can change the camera's shooting mode within the camera interface, such as changing the current focal length or adding filters, exposure, focus, and other camera parameters. In some scenarios, when users change camera parameters on their phones, the phone may experience a pause before displaying the preview image for the new shooting mode. This results in a poor user experience.

[0038] Taking the case where a user changes the focal length of a camera in a mobile phone, FIG1 and FIG2A-FIG2C exemplarily show schematic diagrams of interfaces related to the case where a user changes the focal length of a camera in a mobile phone.

[0039] Exemplarily, the electronic device 100 may display a desktop 101, in which a page with application icons is displayed, and the page includes multiple application icons (for example, a settings application icon, an application market application icon, a gallery application icon, a browser application icon, etc.). A page indicator 104 is also displayed below the multiple application icons to indicate the positional relationship between the currently displayed page and other pages. A tray area 102 is displayed below the page indicator 104. Among them, the tray area 102 includes multiple tray icons, for example, a camera application icon 103, an address book application icon, a phone application icon, and a message application icon. The tray area 102 remains displayed when the page is switched. In some embodiments, the above-mentioned page may also include multiple application icons and a page indicator 104. The page indicator 104 may not be part of the page and may exist independently. The above-mentioned tray icon is also optional, and the embodiments of the present application are not limited to this.

[0040] The electronic device 100 may receive an input operation (eg, a single click) from a user on the camera application icon 103 . In response to the input operation, the electronic device 100 may display the shooting interface 200 as shown in FIG. 2A .

[0041] As shown in Figure 2A, the shooting interface 200 may include an echo control 203A, a shooting control 203B, a camera conversion control 203C, a preview box 201, a zoom ratio control 202, and controls for one or more shooting modes (for example, a control 204A for the large aperture shooting mode, a control 204B for the night scene shooting mode, a control 204C for the portrait shooting mode, a control 204D for the shooting mode, a control 204E for the video recording mode, a control 204F for the multi-lens video recording mode, and a control 204G for more modes).

[0042] As shown in FIG2A , the control 204D for the photo mode is selected, and the electronic device 100 is in photo mode. The preview box 201 displays a preview image 205 captured by the electronic device 100 through the camera in photo mode. The echo control 203A can be used to trigger the display of the captured image or video. The capture control 203B is used to trigger the saving of the image captured by the camera. The camera conversion control 203C can be used to switch the camera used by the electronic device 100 to capture images (for example, the front camera is switched to the rear camera, or the rear camera is switched to the front camera). The zoom ratio control 202 can be used to set the zoom factor for the electronic device 100 to capture photos or videos. The zoom ratio control 202 can display the currently used zoom factor (for example, 0.6x), a commonly used zoom factor 1 (for example, 1x) that is larger than the currently used zoom factor, and a commonly used zoom factor 2 (for example, 2x) that is larger than the currently used zoom factor.

[0043] The controls for the shooting mode can be used to trigger the image processing process corresponding to the shooting mode. For example, the control 204A for the large aperture shooting mode can be used to trigger the camera to shoot images using large aperture parameters. The control 204B for the night scene shooting mode can be used to trigger increasing the brightness and color richness in the captured image. The control 204C for the portrait shooting mode can be used to trigger the electronic device 100 to beautify the portrait in the captured image. The control 204D for the shooting mode can be used to trigger the electronic device 100 to shoot images using default parameters and use the default image processing process to process the images captured by the camera. The control 204E for the video recording mode can be used to trigger the electronic device 100 to record a video through a single camera. The control 204F for the multi-lens recording mode can be used to trigger the electronic device 100 to record a video simultaneously through multiple cameras. The more mode control 204G can be used to trigger the electronic device 100 to display controls for more shooting modes.

[0044] The electronic device 100 may receive a user input operation for the zoom magnification control 202 (e.g., clicking the commonly used zoom magnification 1 (e.g., 1x)). In response to the input operation, the zoom magnification of the electronic device 100 may be switched from the currently used zoom magnification to the commonly used zoom magnification 1.

[0045] If the camera application of the electronic device 100 issues a large number of camera requests before the electronic device 100 receives the user's input operation for the zoom magnification control 202, there will be a delay in the electronic device 100 processing the zoom magnification change request. This will cause the preview image of the electronic device 100 to freeze for a while before displaying the preview image after the zoom magnification change.

[0046] For example, as shown in FIG2B , the zoom magnification control 202 in the user interface 210 shown in FIG2B displays the currently used zoom magnification of 1x. A freeze occurs in the preview frame 201, and the preview image is not displayed. After the freeze occurs for a period of time, the electronic device 100 may display a user interface 220 showing that the zoom magnification can be switched from the currently used zoom magnification to the commonly used zoom magnification of 1.

[0047] As shown in FIG. 2C , the user interface 220 of the electronic device 100 may display a preview image 221 obtained after the electronic device 100 switches from the currently used zoom magnification (eg, 0.6x) to the commonly used zoom magnification 1 (eg, 1x).

[0048] As shown in FIG. 2A and FIG. 2B , when a user operates a camera to zoom, the electronic device will have a delay before displaying the zoomed image frame on the camera shooting interface, thus resulting in a poor user experience.

[0049] FIG3 shows a schematic diagram of the software and hardware architecture of a camera service on an electronic device provided in an embodiment of the present application.

[0050] As shown in Figure 3, the layered architecture divides the system into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the system is divided into five layers: application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer, from top to bottom.

[0051] The application layer can include a series of application packages.

[0052] The application package may include a camera application, etc.

[0053] The application framework layer (Frameworks, FWK) provides the application programming interface (API) and programming framework for the application packages in the application layer. The application framework layer includes some predefined functions.

[0054] In some embodiments, the application framework layer may include a camera interface, which may include an image acquisition interface (eg, a capture() interface) and a continuous image acquisition interface (eg, a setRepeatingRequest() interface).

[0055] In some embodiments, the application framework layer may also include a request queue (RequestQueue) module, which may include camera requests such as capture request 1 (Capture request1) and capture request 2 (Capture request2) issued by the camera application through the capture() interface, and continuous photo taking or preview requests (Repeating Request) issued through the setRepeatingRequest() interface.

[0056] In some embodiments, the application framework layer may also include a request queue processing (RequestThread) module. The request queue processing module may include a wait request (WaitForRequest) module, a send request batch (SendRequestsBatch) module, and a prepare hardware abstraction layer request (PrepareHardware AbstractionLayerRequest, which may be referred to as PrepareHALRequest) module. Among them, the WaitForRequest module can be used to keep the camera request (e.g., Capture request 1) to be executed in the request queue module waiting when there are no empty buffers in the N buffers allocated to the camera request of the camera application in the buffer area of ​​the electronic device. The PrepareHALRequest module can be used to construct the Capture request of the HAL layer and the output buffer outputBuffers. When there are empty buffers in the N buffers allocated to the camera request of the camera application, the PrepareHALRequest module can obtain a buffer from the Allocate Buffer interface for the camera request issued by the camera application. The send request batch (Send) module can be used to send the Capture request of the HAL layer constructed by the PrepareHALRequest module to the HAL layer.

[0057] For example, N can be 8, meaning that the number of buffers allocated by the electronic device to the camera application for camera requests can be 8. N can also be other values, such as 7 or 9, and the present application does not limit the value of N. The following description uses the example of N being 8 as an example.

[0058] In some embodiments, the application framework layer may further include an Allocate Buffer interface for allocating a buffer for a camera request issued by a camera application. For example, the Allocate Buffer interface may specify the address of a buffer for storing the camera request.

[0059] In some embodiments, the application framework layer may also include a display service module that can be used to obtain image frames uploaded by the HAL layer and display the obtained image frames. When the image frame is successfully displayed, the buffer storing the successfully displayed image frame is cleared. The display service module can notify the Allocate Buffer interface of the buffer address storing the successfully displayed image frame.

[0060] In some embodiments, the application framework layer may also be referred to as the application framework layer.

[0061] The hardware abstraction HAL layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system.

[0062] The hardware abstraction layer (HAL) may include a camera hardware abstraction layer (HAL). This layer receives camera requests from the application framework layer and transmits the camera parameters and buffer address in the request to the camera module via the driver layer. It also calls a result-returning interface (e.g., ProcessCaptureResult) to send image frames captured by the camera module according to the camera parameters to the display service in the application framework layer.

[0063] Optionally, the camera hardware abstraction layer may include a camera request waiting module and a camera hardware abstraction processing module. The camera request waiting module and the camera hardware abstraction processing module are not shown in FIG3 . For details about the camera request waiting module and the camera hardware abstraction processing module, please refer to FIG5 below.

[0064] The camera request waiting module can be used to receive camera requests sent by the application framework layer. When the camera hardware abstraction processing module completes processing a camera request, the camera request waiting module can send a new camera request to the camera hardware abstraction module.

[0065] The camera hardware abstraction processing module can send the camera parameters and the buffer address of the new camera request to the camera module through the driver layer. The camera request processing module can also call the return result interface (for example, ProcessCaptureResult) to send the image frame obtained by the camera module according to the camera parameters to the display service in the application framework layer.

[0066] The driver layer is the layer between hardware and software. It includes drivers for various hardware components. These include camera drivers, image processor drivers, and display drivers. The camera driver drives the image sensors (e.g., image sensor 1, image sensor 2, etc.) of one or more cameras in the camera module to capture images and drives the image signal processor to pre-process the images. The image processor driver drives the graphics processor to process images. The display driver drives the display.

[0067] The hardware layer may include a camera module, an image signal processor, a display, and the like. The camera module may include one or more camera image sensors (e.g., image sensor 1, image sensor 2, etc.). Optionally, the camera module may also include a time of flight (TOF) sensor, a multispectral sensor, and the like. The image signal processor may be used to process image frames captured by the camera module. The display may be used to display image frames sent by the display service module.

[0068] In an embodiment of the present application, the request module may be a queue request module, the cache module may be an allocation cache interface module, the camera processing module may be a camera hardware abstraction processing module, and the display module may be a display service module.

[0069] In combination with the above system structure, the following describes in detail the processing mechanism of the electronic device 100 for a camera request generated based on a user's operation for changing the focus, taking zoom as an example:

[0070] ①. The electronic device 100 receives an operation from the user to start the camera application and issues multiple camera requests for obtaining preview images.

[0071] When the electronic device 100 receives an operation from the user to start the camera application (for example, the user clicks the camera application icon 103 shown in FIG1 ), the electronic device 100 starts the camera application. The camera application may issue multiple camera requests for obtaining preview images. The first camera request for obtaining preview images issued by the camera application may fill up the N buffers reserved for camera requests in the cache area.

[0072] It is understandable that after the camera application is turned on, it can periodically send requests to the buffer and obtain corresponding image frame data according to the requests.

[0073] ②. The application framework layer of the electronic device 100 places N requests for multiple preview images into N buffers in sequence according to the order of issuance time.

[0074] The application framework layer of the electronic device 100 may place N requests among the multiple preview image requests into N buffers in sequence according to the order of issuance time.

[0075] The application framework layer can sequentially send the camera requests in N buffers to the HAL layer. The HAL layer processes the camera requests in the N buffers in turn, notifying the camera module of the camera parameters and buffer addresses in the camera requests. The camera module can then obtain an image frame based on the camera parameters and store it in the buffer corresponding to the buffer address. The HAL obtains the image frame and uploads it to the application framework layer, which can then display it. After successful display, the image frame can be displayed in the preview box of the camera application's shooting interface. After successful display, the buffer containing the image frame and the corresponding camera request in the N buffers can be cleared. The application framework layer can then assign one of the remaining camera requests from the multiple camera requests to the cleared buffer. The application framework layer will sequentially assign the remaining camera requests from the multiple camera requests to the cleared buffers in the N buffers.

[0076] It is understandable that the process of the HAL layer processing the camera request can refer to the description in steps 3, 4, and 5 below, which will not be repeated here.

[0077] ③. The electronic device 100 receives the user's operation on the camera application and issues a camera request.

[0078] Electronic device 100 may receive user operations on the camera application, such as changing the focal length magnification. Based on this user operation, the camera application in electronic device 100 may issue a camera request. The request queue module in the application framework layer of electronic device 100 may obtain this camera request through the setRepeatingRequest() or capture() interfaces in the camera interface. When there is an empty buffer in the N buffers reserved for camera requests in the cache area of ​​electronic device 100, electronic device 100 may issue this camera request from the request queue module to the prepare HAL request module in the request queue processing module. After obtaining the buffer address allocated for the camera request using the Allocate Buffer interface, the prepare HAL request module may store the camera request in this buffer. The prepare HAL request module may construct this camera request into a Capture request at the HAL layer and issue it to the camera request waiting module in the HAL layer via the batch request issuing module. When the previous camera request processed by the camera hardware abstraction module has been completed, the camera request waiting module may send this camera request to the camera hardware abstraction processing module. Then, the camera hardware abstract processing module may send the camera parameters in the new camera request and the buffer address of the camera request to the camera module through the driver layer.

[0079] ④. The electronic device 100 turns on the camera and obtains the image frame captured by the camera based on the camera request.

[0080] The camera driver in electronic device 100 can receive the camera parameters and the buffer address of the camera request from the camera hardware abstraction processing module. The camera driver can then send the camera parameters and the buffer address of the camera request to the camera module. The camera module can acquire image frames according to the camera parameters and store the acquired image frames in the buffer corresponding to the buffer address carried in the camera request.

[0081] In some examples, image frames captured by the camera module can be transmitted to an image signal processor, which can pre-process the image frames and upload them to the HAL layer via a camera driver or an image processor driver. The HAL layer can then upload the image frames to a display service in the application framework layer.

[0082] In some examples, the HAL layer can upload the image frame to the camera application through the application framework layer, and the camera application can process the image frame before sending it to the display service.

[0083] ⑤. The electronic device 100 displays the image frame.

[0084] The display service can send the image frame for display, that is, notify the display through the display driver to display the image frame. That is, the display of the electronic device 100 can display the image frame. After the display successfully sends the image frame for display, it will notify the display service that the image frame is successfully displayed. It is understandable that after the image frame is sent for display, the buffer in the cache of the electronic device that stores the image frame and the camera request for obtaining the image frame will be cleared. The display service can send the address of the cleared buffer to the Allocate Buffer interface. The Allocate Buffer interface can subsequently assign the address of the cleared buffer to the next camera request.

[0085] For example, as shown in Figure 3, when the queue request module issues photo request 1, buffer 8 is idle. The HAL request preparation module can then obtain a buffer for photo request 1 from the Allocate Buffer interface. The Allocate Buffer interface allocates photo request 1 to buffer 8. After allocating the buffer, the HAL request preparation module can send photo request 1 to the batch request issuance module. The batch request issuance module can then send photo request 1 to the HAL layer. Photo request 1 can then wait in the HAL layer's request queue for processing.

[0086] When the user changes the current zoom factor in the camera application's interface, the camera application can issue a photo request 2. Since there are no empty buffers in the buffer, the request queue module can send this photo request 2 to the waiting request module to wait for a free buffer. After the batch request issuing module sends photo request 1 from the HAL request preparation module to the HAL layer, the batch request issuing module can notify the waiting request module. The waiting request module can then send this photo request 2 to the HAL request preparation module.

[0087] The HAL layer can first process the request in buffer 1 and return the image frame corresponding to the request in buffer 1 to the display service module. The display service module can display the image frame. When display is successful, the display service module can return the address of buffer 1 to the Allocate Buffer interface. The Allocate Buffer interface can then allocate buffer 1 to photo request 2. After allocating the buffer, the HAL request preparation module can send photo request 2 to the batch request module. The batch request module can then send photo request 2 to the HAL layer. Photo request 2 can wait in the HAL layer's request queue for HAL layer processing. The HAL layer must process requests in buffer 2 through buffer 8 in the order in which the requests were sent before processing photo request 2.

[0088] According to the above-mentioned camera request processing mechanism, if there is a free buffer among the N buffers allocated by the electronic device 100 for the camera request of the camera application, the application framework layer of the electronic device 100 can continue to send the camera request issued by the camera application to the HAL layer. If there is no free buffer among the N buffers allocated by the electronic device 100 for the camera request of the camera application, the HAL layer needs to successfully process a camera request and return the processing request result (for example, the image frame obtained according to the camera parameters of the camera request) to the display service of the application framework layer. The display service will clear a buffer only after it successfully sends it for display.

[0089] When a user opens the camera application, the camera application can send multiple camera requests for obtaining preview images. Since the number of buffers used to store camera requests is limited (for example, 8), the camera requests sent by the camera application will fill the buffer. When the user clicks the zoom control to switch the current zoom ratio (for example, 0.6x) to another zoom ratio (for example, 1x), in response to this operation, the camera application of the electronic device 100 can send a camera request, which can carry the user's newly selected zoom ratio (for example, 1x). The electronic device 100 needs to wait for the HAL to process the camera request in a buffer and clear a buffer before the electronic device 100 can allocate a buffer for the camera request carrying the user's newly selected zoom ratio and send it to the HAL layer. In this way, when the HAL layer processes a request slowly, resulting in a slow output of a certain frame, it will not only affect the slow display of the current frame and cause lag, but also cause subsequent camera requests to be unable to be sent to the HAL layer in time, making the lag problem more and more serious. In this way, the user experience is poor.

[0090] Figure 4 exemplarily shows the time it takes for the electronic device 100 to receive an operation from the user to change the zoom ratio, the time it takes for the electronic device 100 to send the request corresponding to the operation to the HAL layer, and the time it takes for the electronic device 100 to respond to the user operation and successfully display the image frame corresponding to the user operation.

[0091] As shown in Figure (a) in Figure 4 , after the camera application of the electronic device 100 is turned on, in the initial camera request (for example, request 1 shown in Figure (a) in Figure 4 ), the zoom ratio can be 0.6x. Before the user changes the zoom ratio of the camera, the camera module of the electronic device 100 acquires image frames at a zoom ratio of 0.6x. After the user changes the zoom ratio of the camera to 1.0x, the sensor (for example, a touch sensor) in the screen (also referred to as a display or display screen) in the electronic device 100 can acquire the user's touch operation. After acquiring the user operation, the camera application of the electronic device 100 issues a camera request based on the user operation (such as request n (zoom: 1.0 (indicating a zoom ratio of 1.0x)) shown in Figure 4 ). For example, if the number of buffers in the cache area for storing camera requests in the electronic device 100 is 8, when all 8 buffers store camera requests, the duration for the camera request carrying a new zoom factor sent by the camera application in the electronic device 100 to be sent to the HAL layer through the application framework layer is z1. The calculation formula for z1 can be as follows:

[0092] z1=x+y11+v (Formula 1)

[0093] In Formula 1 above, x is the duration it takes for the screen sensor of electronic device 100 to detect a user action, for the camera application to convert the action into a corresponding camera request, and then send it to the FWK. y11 is the duration it takes for the HAL to process the request in buffer 1, obtain the image frame 401 corresponding to the request, return the image frame 401 to the FWK, and for the FWK to successfully display the image frame 401. v is the duration it takes for the FWK to allocate a buffer for the zoom request and send the zoom request to the HAL.

[0094] As shown in FIG4( b ), the duration for the electronic device 100 to receive the user's zoom operation and successfully display the corresponding image frame is T1. The calculation formula of T1 can be as follows:

[0095] T1=x+y11+v+y12+y13+y14+y15+y16+y17+y18+y19 (Formula 2)

[0096] In the above formula 2, y12 is the duration of time it takes for the HAL to process the request in buffer2, obtain the image frame 402 corresponding to the request, return the image frame 402 to FWK, and successfully display the image frame 402. y13 is the duration of time it takes for the HAL to process the request in buffer3, obtain the image frame 403 corresponding to the request, return the image frame 403 to FWK, and successfully display the image frame 403. y14 is the duration of time it takes for the HAL to process the request in buffer4, obtain the image frame 404 corresponding to the request, return the image frame 404 to FWK, and successfully display the image frame 404. y15 is the duration of time it takes for the HAL to process the request in buffer5, obtain the image frame 405 corresponding to the request, return the image frame 405 to FWK, and successfully display the image frame 405. y16 is the HAL processing the request in buffer6, obtaining the image frame 406 corresponding to the request, returning the image frame 406 to FWK, and the duration during which FWK successfully displays the image frame 406. y17 is the HAL processing the request in buffer7, obtaining the image frame 407 corresponding to the request, returning the image frame 407 to FWK, and the duration during which FWK successfully displays the image frame 407. y18 is the HAL processing the request in buffer8, obtaining the image frame 408 corresponding to the request, returning the image frame 408 to FWK, and the duration during which FWK successfully displays the image frame 408. y19 is the HAL processing the zoom request, obtaining the image frame 409 corresponding to the zoom request, returning the image frame 409 to FWK, and the duration during which FWK successfully displays the image frame 409.

[0097] As can be seen from Figures (a) and (b) in Figure 4, when the HAL layer processes a request relatively slowly, resulting in a certain frame (for example, the image frame 401 corresponding to the request in buffer1) being output relatively slowly, it will not only affect the slow display of the current frame, causing a freeze. It will also make y11 become relatively large, so that not only will the time z1 for the electronic device 100 to receive the zoom operation and send the zoom request corresponding to the zoom operation to the HAL be lengthened, but the total time T1 for the electronic device 100 to respond to the user's zoom operation will also be lengthened. In this way, the camera request generated by the camera application of the electronic device 100 based on the user's zoom operation cannot be sent to the HAL layer in time, making the freeze problem more and more serious. In this way, the user experience is poor.

[0098] To reduce the chirality delay of camera-related parameters (e.g., zoom, focus, exposure, and filter) and improve the user experience, an embodiment of the present application provides a camera request processing method. The method may include: the application framework layer in the electronic device 100 sets P buffers out of N buffers used to store camera requests issued by the camera application as buffers for storing normal camera requests, and sets q buffers out of N buffers as buffers for storing special camera requests. The camera application of the electronic device 100 is started, and the application framework layer in the electronic device 100 sequentially stores the multiple camera requests issued by the camera application in the p buffers. The HAL layer in the electronic device 100 may sequentially send the multiple camera requests stored in the p buffers to the camera module to obtain image frames, and then return the image frames to the application framework layer for display. Upon receiving a user operation on the camera application's shooting interface, the camera application of the electronic device 100 sends a first camera request to the application framework layer. When the application framework layer determines that a first parameter in the first camera request has changed, the application framework layer stores the first camera request in buffer 21 out of the q buffers. The HAL layer can then send the first camera request in buffer21 to the camera module to obtain the corresponding image frame and return it to the application framework layer for display. When the application framework layer determines that the first parameter in the first camera request has not changed, if there is no empty buffer in the p buffers, the application framework layer waits for the HAL layer to process the camera request in buffer11 among the p buffers, then stores the first camera request in buffer11 and sends the first request to the HAL layer. After the HAL layer processes the camera requests before the first camera request, the HAL layer can send the first camera request to the camera module to obtain the image frame and return the image frame to the application framework layer for display. The first parameter may include focal length, focus, exposure, filter, etc.

[0099] In an embodiment of the present application, a camera request in which the parameters such as focal length, focus, exposure, and filter have not changed compared to the parameters such as focal length, focus, exposure, and filter in the previous camera request can be called a normal camera request; a camera request in which the parameters such as focal length, focus, exposure, and filter have changed can be called a special camera request.

[0100] FIG5 exemplarily shows the software and hardware architecture of a camera service of an electronic device 100 provided in an embodiment of the present application.

[0101] As shown in Figure 5, the system of electronic device 100 can be divided into five layers: from top to bottom, the application layer, the application framework layer, the hardware abstraction layer, the driver layer, and the hardware layer. The driver layer and the hardware layer are not shown in Figure 5. For details on the other modules included in the driver layer and the hardware layer and their connections with other layers, see Figure 3. The buffer allocation interface in the application framework layer can divide the buffer used to store camera requests into a normal camera request storage area and a special camera request storage area.

[0102] For detailed descriptions of the application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer, please refer to the description in FIG3 , which will not be repeated here.

[0103] Referring to FIG5 , based on the system structure of the electronic device, a camera request processing method provided by an embodiment of the present application may include:

[0104] 1. The electronic device 100 receives an operation from the user to start the camera application, and issues multiple camera requests R1 for obtaining preview images.

[0105] When the electronic device 100 receives the user's operation to start the camera application (for example, the user clicks the camera application icon 103 shown in Figure 1), the electronic device 100 starts the camera application. The camera application can send multiple camera requests for obtaining preview images (for example, camera request R11, camera request R12, ..., camera request R1m). When the user does not operate the focal length, focus, exposure, and filter parameter setting controls, the focal length, focus, exposure, filter and other parameters carried in the camera request sent by the camera application will not change. Because the application framework layer sets p buffers out of the N buffers used to store the camera requests sent by the camera application as buffers for storing ordinary camera requests, and sets q buffers out of the N buffers as buffers for storing special camera requests. Therefore, the multiple camera requests sent by the camera application for the first time will not fill up the N buffers.

[0106] For example, as shown in FIG5 , the electronic device 100 may have eight buffers for storing camera requests: buffer 1, buffer 2, buffer 3, buffer 4, buffer 5, buffer 6, buffer 7, and buffer 8. The application framework layer of the electronic device 100 may configure buffer 1, buffer 2, buffer 3, and buffer 4 to store normal camera requests. The application framework layer may also configure buffer 5, buffer 6, buffer 7, and buffer 8 to store special camera requests.

[0107] 2. The application framework layer of the electronic device 100 places P camera requests among the multiple camera requests for obtaining preview images into P buffers in sequence according to the order of issuance time.

[0108] The application framework layer of the electronic device 100 may place P requests among the multiple camera requests into P buffers in sequence according to the order of issuance time.

[0109] The application framework layer can sequentially send the camera requests in the P buffers to the HAL layer. The HAL layer processes the camera requests in the P buffers in turn, notifying the camera module of the camera parameters and buffer addresses in the camera requests. The camera module can then obtain an image frame based on the camera parameters and store the image frame in the buffer corresponding to the buffer address. The HAL obtains the image frame and uploads it to the application framework layer, which can then display it. After successful display, the image frame can be displayed in the preview box of the camera application's shooting interface. After successful display, the buffers in the P buffers containing the image frame and the corresponding camera request can be cleared. The application framework layer can assign one of the remaining camera requests in the multiple camera requests R1 to the cleared buffer. The application framework layer will sequentially assign the remaining camera requests in the multiple camera requests R1 to the cleared buffers in the P buffers.

[0110] For example, as shown in Figure 5, the application framework layer can first allocate four of the multiple camera requests for obtaining preview images to buffer1, buffer2, buffer3, and buffer4 in sequence. In some examples, when the camera application is turned on, the camera application can continuously issue camera requests for obtaining preview images, and can issue 30 camera requests for obtaining preview images per second. When the camera request in buffer1 is processed, the application framework layer can issue another camera request to buffer1. When the camera request in buffer2 is processed, the application framework layer can issue another camera request to buffer2. Buffer1, buffer2, buffer3, and buffer4 can store multiple camera requests for obtaining preview images issued by the camera application in turn.

[0111] 3. The electronic device 100 receives the user's operation on the camera application and sends a camera request R2.

[0112] Electronic device 100 may receive a user operation on the camera application, such as changing the focal length magnification. Based on this user operation, the camera application in electronic device 100 may issue a camera request R2. The request queue module in the application framework layer of electronic device 100 may obtain this camera request R2 through the setRepeatingRequest() interface or the capture() interface in the camera interface.

[0113] 4. If the application framework layer of the electronic device 100 determines that the first parameter in the camera request R2 has changed, the camera request is allocated to buffer(p+1) among the Q buffers; if the application framework layer of the electronic device 100 determines that the first parameter in the camera request R2 has not changed, the camera request R2 is allocated to buffer1 after buffer1 among the P buffers is cleared.

[0114] The application framework layer of the electronic device 100 can determine that the first parameter (e.g., zoom magnification) in the camera request R2 has changed. For example, the request queue processing module in the application framework layer can compare the camera request R2 with the previous camera request. If the zoom magnification in the previous camera request is 0.6x and the zoom magnification in the camera request R2 is 1.0x, then the request queue processing module can confirm that the first parameter in the camera request R2 has changed. The allocation buffer area interface in the application framework layer can allocate the camera request R2 to any buffer in the special camera request buffer area, such as buffer5. Since the multiple camera requests issued when the camera is turned on are not allocated to buffer5, buffer6, buffer7, and buffer8, buffer5, buffer6, buffer7, and buffer8 are all empty buffers, that is, no camera requests have been stored, so the camera request R2 can be directly allocated to buffer5 without waiting for the buffer to be cleared before being allocated to the buffer.

[0115] If the application framework layer of electronic device 100 determines that the first parameter in camera request R2 has not changed, it waits for buffer1 of the P buffers to clear before allocating camera request R2 to buffer1. For example, the request queue processing module in the application framework layer may compare camera request R2 with the previous camera request. If the zoom magnification, focus, exposure, filter, and other parameters in the previous camera request are the same as those in camera request R2, the request queue processing module may confirm that the first parameter in camera request R2 has not changed. The buffer allocation interface in the application framework layer may allocate camera request R2 to a buffer in the common camera request buffer. Because multiple camera requests issued by the camera application will fill up the buffers in the common camera request buffer (i.e., buffer1, buffer2, buffer3, and buffer4 all store camera requests), it is necessary to wait for one of buffer1, buffer2, buffer3, and buffer4 to clear before allocating camera request R2 to the cleared buffer.

[0116] 5. The electronic device 100 turns on the camera and obtains the image frame captured by the camera based on the camera request.

[0117] The camera driver in electronic device 100 can receive the camera parameters and the buffer address of the camera request from the camera hardware abstraction processing module. The camera driver can then send the camera parameters and the buffer address of the camera request to the camera module. The camera module can acquire image frames according to the camera parameters and store the acquired image frames in the buffer corresponding to the buffer address carried in the camera request.

[0118] In some examples, image frames captured by the camera module can be transmitted to an image signal processor, which can pre-process the image frames and upload them to the HAL layer via a camera driver or an image processor driver. The HAL layer can then upload the image frames to a display service in the application framework layer.

[0119] In some examples, the HAL layer can upload the image frame to the camera application through the application framework layer, and the camera application can process the image frame before sending it to the display service.

[0120] 6. The electronic device 100 displays the image frame.

[0121] The display service can send the image frame for display, that is, notify the display through the display driver to display the image frame. That is, the display of the electronic device 100 can display the image frame. After the display successfully sends the image frame for display, it will notify the display service that the image frame is successfully displayed. It is understandable that after the image frame is sent for display, the buffer in the cache of the electronic device that stores the image frame and the camera request for obtaining the image frame will be cleared. The display service can send the address of the cleared buffer to the Allocate Buffer interface. The Allocate Buffer interface can subsequently assign the address of the cleared buffer to the next camera request.

[0122] For example, as shown in Figure 5, when the queue request module issues photo request 1, buffer 4 in the standard camera request buffer is idle. The HAL request preparation module can then obtain a buffer for photo request 1 from the Allocate Buffer interface. The Allocate Buffer interface allocates photo request 1 to buffer 4. After allocating the buffer, the HAL request preparation module can send photo request 1 to the batch request issuance module. The batch request issuance module can then send photo request 1 to the HAL layer. Photo request 1 can then wait in the HAL layer's request queue for processing.

[0123] When the user changes the current zoom ratio in the camera application interface, the camera application can send photo request 2. Since the camera requests when the camera is turned on are only allocated to the normal camera request buffer, the buffer in the special camera request buffer is not full. The queue request module determines that the zoom ratio in photo request 2 is different from the zoom ratio in photo request 1. The Allocate Buffer interface can allocate buffer 5 in the special camera request buffer to photo request 2, and then send it to the HAL layer. Photo request 2 can wait in the HAL layer's request queue for HAL layer processing. The HAL layer needs to process the requests in buffer 1 to buffer 4 in the order in which the requests are sent, and then process photo request 2.

[0124] When the user does not change the shooting mode in the camera's shooting interface, the camera parameters of each frame of the camera preview image are the same, and the user will not feel any lag between the previous image frame and the next image frame. However, if the user changes the zoom ratio, or performs focus, exposure, or adds a filter in the camera's shooting interface, the current shooting mode of the camera application will be changed, and the camera parameters of the preview image in the camera's shooting interface will also change accordingly. That is, one or more of the camera parameters such as zoom ratio, focus, exposure, filter, etc. of the preview image before and after the user operation will be different. When the user operates, the electronic device 100 takes too long to respond, and the user will not be able to see the preview image in the new shooting mode for a long time after the operation. In this way, the user will feel the camera lag, which will affect the user experience.

[0125] In a camera request processing method provided by an embodiment of the present application, multiple camera requests for obtaining preview images issued when the camera is turned on are stored in only a portion of the buffer (e.g., buffers 1, 2, 3, and 4 of the normal camera request buffer shown in FIG5 ). Another portion of the buffer (e.g., buffers 5, 6, 7, and 8 of the special camera request buffer shown in FIG5 ) is allocated only for camera requests whose first parameter has changed. When a user changes the zoom factor, performs focus, exposure, or adds a filter in the camera's shooting interface, the electronic device 100 can directly allocate the camera request generated based on the user operation to the special camera request buffer. The application framework layer does not need to wait for a buffer to be cleared before allocating it to the buffer, nor does the application framework layer need to wait for the HAL to process the next request before allocating a buffer to the camera request. In this way, the electronic device 100 can reduce the response time to user operations such as changing the zoom factor, focusing, exposure, or adding a filter, thereby improving the efficiency of responding to user operations.

[0126] In the embodiment of the present application, the special camera request buffer can be referred to as the first buffer, and buffer5, buffer6, buffer7, and buffer8 can all be referred to as the first buffer. The normal camera request buffer can be referred to as the second buffer. Buffer1, buffer2, buffer3, and buffer4 can also be referred to as the second buffer.

[0127] In conjunction with the software framework shown in FIG5 , a camera request processing method provided in an embodiment of the present application is described in detail by taking the example of a user changing the current zoom ratio in the shooting interface of the camera application after the camera application is started.

[0128] FIG6 shows a schematic diagram of software module interaction of a camera request processing method in an embodiment of the present application.

[0129] As shown in Figure 6, the software framework of the electronic device 100 may include a camera application, an application framework layer (framework, FWK), and a hardware abstraction layer (HAL). The application framework layer may include a camera interface module 610, a request queue module 620, a request queue processing module 630, an allocation buffer interface module 640, and a display service module 650. The hardware abstraction layer may include a camera hardware abstraction layer (Camera HAL) 660.

[0130] The interaction process between the software modules of the electronic device 100 may include the following steps:

[0131] S601. The camera application detects an operation to start the camera and starts the camera application.

[0132] The camera application can detect the operation of starting the camera, for example, the user can click the camera application icon 103 as shown in Figure 1. In response to the user operation, the camera application is started and the shooting interface of the camera application is displayed, such as the shooting interface 200 shown in Figure 2A.

[0133] S602 . The camera application notifies the allocation buffer interface module 640 of the camera mode (photographing mode or video recording mode).

[0134] Optionally, the camera application may inform the buffer allocation interface module 640 of the camera mode (photographing mode or video recording mode).

[0135] As shown in FIG2A , after the camera application is started, a shooting interface 200 is displayed. As shown in FIG2A , a control 204D for the normal shooting mode is selected, and the camera application is in shooting mode. When the camera application is in shooting mode, the camera application can notify the allocation buffer area interface module 640 that the camera mode of the camera application is shooting mode.

[0136] As shown in FIG7 , the electronic device 100 may receive an input operation from a user selecting a control 204E for recording mode (eg, clicking the control 204E). In response to the input operation, the electronic device 100 may switch from a photographing mode to a recording mode.

[0137] As shown in Figure 8, the electronic device 100 can display a shooting interface 800. The control 204E of the video recording mode in the shooting interface 800 is selected, and the camera application of the electronic device 100 is in the video recording mode. After the electronic device 100 switches to the normal video recording mode, the electronic device 100 can display the preview image 802 acquired by the camera in real time in the normal video recording mode in the preview box 801, and replace the above-mentioned shooting control 203B with the recording start control 203D. After switching to the normal video recording mode, the electronic device 100 can display one or more function controls (for example, a flash control 803A, a setting control 803B, etc.) on the shooting interface 800. The flash control 803A can be used to trigger the electronic device 100 to turn on or off the flash. The setting control 803B can be used to set the shooting parameters of the electronic device 100 (for example, image size, image storage format, etc.).

[0138] When the camera application switches from the photographing mode to the video recording mode, the camera application may inform the allocation buffer interface module 640 that the camera mode of the camera application is the video recording mode.

[0139] For example, the camera application can indicate the camera mode of the camera application by sending an instruction to the allocation buffer area interface module 640. Specifically, the camera application can send instruction 1 to the allocation buffer area interface module 640 to indicate that the camera mode of the camera application is the photo taking mode. The camera application can send instruction 2 to the allocation buffer area interface module 640 to indicate that the camera mode of the camera application is the video recording mode. For example, the content of instruction 1 can be "1" and the content of instruction 2 can be "0". The specific content of instruction 1 and instruction 2 is not limited in this embodiment of the application.

[0140] S603. The buffer allocation interface module 640 sets p buffers among the N buffers for storing camera requests to store common camera requests and q buffers to store special camera requests based on the camera mode.

[0141] Based on the camera mode, the buffer allocation interface module 640 can allocate p of the N buffers used to store camera requests to store normal camera requests and q to store special camera requests. The values ​​of p and q can be different in different camera modes. That is, the values ​​of p and q in photo mode can be different from the values ​​of p and q in video mode.

[0142] For example, taking the value of N as 8, in the photo mode, the value of p may be 4, and the value of q may be 4. In the video mode, the value of p may be 5, and the value of q may be 3.

[0143] It is understandable that in photo mode, the camera application will continuously send requests to obtain the preview data stream (i.e., obtain the preview image frame), and only when the user clicks the shooting control will it be necessary to obtain the image frame at the moment the user presses the shooting control (also known as the shutter). In video recording mode, when the user clicks the recording start control, the camera application will continuously send two types of requests, one for obtaining the preview data stream and the other for obtaining the video data stream (i.e., video frame). In this way, in video recording mode, in order to ensure that the camera application can run smoothly, it is necessary to reserve more buffer for ordinary camera requests. Therefore, the value of p in video recording mode is greater than the value of p in photo mode. The value of q in video recording mode is less than the value of q in photo mode.

[0144] Optionally, in one possible implementation, the camera application may not inform the buffer allocation interface module 640 of the camera mode of the camera application. The buffer allocation interface module 640 may determine the values ​​of p and q based on the camera request type received by the request queue module 620. When the request queue module 620 receives two types of camera requests, one type for obtaining a preview data stream and the other type for obtaining a video data stream, the buffer allocation interface module 640 may determine the value of p to be p1 and the value of q to be q1. When the request queue module 620 receives only one type of camera request for obtaining a preview data stream, the buffer allocation interface module 640 may determine the value of p to be p2 and the value of q to be q2.

[0145] For example, when N is equal to 8, p1 is equal to 5, q1 is equal to 3, p2 is equal to 4, and q2 is equal to 4. The embodiment of the present application does not limit the values ​​of N, p1, q1, p2, and q2.

[0146] S604 . The camera application sends multiple camera requests (camera request R11 , camera request R12 , . . . , camera request R1m ).

[0147] The camera application may send multiple camera requests (camera request R11, camera request R12, ..., camera request R1m) to the camera interface module 610. For example, the camera request R11 may be the photo request 1 shown in FIG. 3 or FIG. 5 .

[0148] S605 . The camera interface module 610 sends the multiple camera requests (camera request R11 , camera request R12 , . . . , camera request R1m ) issued by the camera application to the request queue module 620 .

[0149] The camera interface module 610 may include a capture() interface and a setRepeatingRequest() interface. When the multiple camera requests sent by the camera application are requests for continuously obtaining preview data streams, the camera interface module 610 may obtain the multiple camera requests sent by the camera application through the setRepeatingRequest() interface.

[0150] The camera interface module 610 may send multiple camera requests (camera request R11 , camera request R12 , . . . , camera request R1m) issued by the camera application to the request queue module 620 .

[0151] S606a . The request queue module 620 sends multiple camera requests (camera request R11 , camera request R12 , . . . , camera request R1m ) to the request queue processing module 630 .

[0152] The request queue module 620 may send multiple camera requests (camera request R11 , camera request R12 , . . . , camera request R1m) to the request queue processing module 630 one by one according to the request issuance order.

[0153] Optionally, before sending the multiple camera requests to the request queue processing module 630, the request queue module 620 may also determine whether the multiple camera requests are standard camera requests or special camera requests. If the request queue module 620 determines that the multiple camera requests are standard camera requests, the request queue module 620 may send the multiple camera requests (camera request R11, camera request R12, ..., camera request R1m) one by one to the request queue processing module 630 in the order in which the requests were sent.

[0154] Optionally, the request queue module 620 may not perform the judgment and directly assume that the camera request sent by the camera application when it is turned on is a normal camera request.

[0155] S606b. The request queue processing module 630 allocates buffers to the buffer allocation interface module 640 for the camera request R11, the camera request R12, ..., and the camera request R1p.

[0156] S606c. The camera requests R1(p+1)..., and camera requests R1m in the request queue processing module 630 wait for buffer allocation.

[0157] The request queue processing module 630 can retrieve buffers for multiple camera requests one by one in the order in which the requests were issued. Because the number of buffers available for storing camera requests is limited, the request queue processing module 630 requests buffer allocation from the buffer allocation interface module 640 for camera requests R11, R12, ..., and R1p. Camera requests R1(p+1) ... and R1m in the request queue processing module 630 await buffer allocation.

[0158] The request queue processing module 630 may include a waiting request module and a preparing HAL request module. The preparing HAL request module may request buffer allocation for camera requests R11, R12, ..., and R1p from the buffer allocation interface module 640. Camera requests R1(p+1) ... and R1m may wait for buffer allocation in the waiting request module.

[0159] S607 . Allocate buffer areas. The interface module 640 allocates the camera request R11 , the camera request R12 , . . . , the camera request R1p to p buffers.

[0160] The buffer allocation interface module 640 can allocate camera requests R11, R12, ..., R1p to p buffers one by one. For example, as shown in FIG5 , p is 4, meaning that a typical camera request buffer has four buffers (buffer1, buffer2, buffer3, and buffer4). The buffer allocation interface module 640 can allocate camera request R11 to buffer1, camera request R12 to buffer2, camera request R13 to buffer3, and camera request R14 to buffer4.

[0161] Optionally, the buffer allocation interface module 640 can allocate buffers for camera request 11, camera request R12, ..., camera request R1p, and return the address of the buffer for storing each camera request to the request queue processing module 630. The request queue processing module 630 can store camera request 11, camera request R12, ..., camera request R1p in the buffer corresponding to each camera request according to the buffer address.

[0162] S608 . The buffer allocation interface module 640 sends the camera request R11 , the camera request R12 , . . . , the camera request R1p to the camera hardware abstraction layer 660 in sequence.

[0163] The buffer allocation interface module 640 sends the camera requests in p buffers to the camera hardware abstraction layer 660 in sequence.

[0164] In one possible implementation, the buffer allocation interface module 640 may sequentially send p buffer addresses storing camera requests R11, R12, ..., and R1p to the camera hardware abstraction layer 660. The camera hardware abstraction layer 660 may retrieve the camera requests one by one from the corresponding buffers according to the buffer addresses.

[0165] Optionally, in another possible implementation, the request queue processing module 630 also includes a batch request issuing module. The buffer allocation interface module 640 can allocate buffers for camera requests R11, R12, ..., and R1p, and return the address of the buffer used to store each camera request to the request queue processing module 630. The batch request issuing module can then send the buffer address of each camera request (R11, R12, ..., and R1p) to the camera hardware abstraction layer 660. The camera hardware abstraction layer 660 can retrieve the camera requests from the corresponding buffers one by one according to the buffer address.

[0166] For example, as shown in FIG5 , the common camera request buffer has four buffers (buffer1, buffer2, buffer3, and buffer4). The buffer allocation interface module 640 can allocate camera request R11 to buffer1, camera request R12 to buffer2, camera request R13 to buffer3, and camera request R14 to buffer4. The batch request issuing module can send the addresses of buffer1, buffer2, buffer3, and buffer4 to the camera hardware abstraction layer 660. The camera hardware abstraction layer 660 can retrieve camera request R11 from buffer1 according to the address of buffer1. After the camera hardware abstraction layer 660 processes camera request R11, it can retrieve camera request R12 from buffer2 according to the address of buffer2. The camera hardware abstraction layer 660 can sequentially retrieve the camera requests stored in the buffers according to the buffer addresses.

[0167] S609 . The camera hardware abstraction layer 660 obtains the image frame 1 corresponding to the camera request R11 stored in buffer 1 among the p buffers through the camera.

[0168] The camera hardware abstraction layer 660 can obtain image frame 1 corresponding to camera request R11 stored in buffer 1 of the N1 buffers. Specifically, as shown in Figure 5, the camera hardware abstraction layer 660 may include a camera request waiting module and a camera hardware abstraction processing module. Pending camera requests may be queued in the camera request waiting module. The camera hardware abstraction processing module may send the camera parameters in the camera request R11 being processed to the camera module via the camera driver. The camera module may obtain image frame 1 according to the camera parameters.

[0169] The camera hardware abstraction layer 660 may send the address of buffer 1 and camera parameters to the camera module, and the image frame 1 acquired by the camera module may be stored in the buffer 1. For example, the buffer 1 may be the buffer 1 shown in FIG.

[0170] In a possible implementation, the camera module may send the acquired image frame 1 to the camera hardware abstraction layer 660 .

[0171] In another possible implementation, the camera module may notify the camera hardware abstraction layer 660 that the image frame 1 has been successfully acquired. After receiving the notification, the camera hardware abstraction layer 660 may acquire the image frame 1 from buffer 1.

[0172] S610 . The camera hardware abstraction layer 660 returns image frame 1 to the display service module 650 .

[0173] The camera hardware abstraction layer 660 may return the image frame 1 to the display service module 650 .

[0174] In a possible implementation, the camera hardware abstraction layer 660 may directly send the acquired image frame 1 to the display service module 650 .

[0175] Optionally, in another possible implementation, the camera hardware abstraction layer 660 may send the address of the buffer storing the image frame 1 to the display service module 650 .

[0176] S611. The display service module 650 sends the image frame 1 for display.

[0177] The display service module 650 can send the image frame 1 to the display through the display driver for display. Optionally, the display service module 650 can send the address of buffer 11 corresponding to the image frame 1 to be selected to the display through the display driver, and the display can obtain the image frame from buffer 1 and display it.

[0178] Exemplarily, the image frame 1 may be the preview image 205 shown in FIG. 7 , or the preview image 802 shown in FIG. 8 .

[0179] In the embodiment of the present application, the buffer corresponding to the image frame may refer to the buffer storing the image frame.

[0180] S612 . The display service module 650 returns the address of buffer11 to the allocation buffer interface module 640 .

[0181] When image frame 1 is successfully displayed, the display may notify the display service module 650 via the display driver. The electronic device 100 may clear buffer 1. The display service module 650 may then return the address of buffer 1 to the buffer allocation interface module 640.

[0182] S613a. The buffer allocation interface module 640 allocates buffer1 to the camera request R1(p+1).

[0183] The buffer allocation interface module 640 may allocate buffer 1 to the camera request R1(p+1). The electronic device 100 may process multiple camera requests according to the above steps S608 to S612.

[0184] S613b. The camera requests R1(p+2)..., camera requests R1m in the request queue processing module 630 wait for buffer allocation.

[0185] After the camera request R1(p+1) is allocated to the buffer, the remaining camera requests, namely the camera request R1(p+2)..., camera request R1m, can continue to wait for the allocation of buffers.

[0186] S614. The camera application detects an operation to change the current zoom ratio of the camera.

[0187] The camera application may detect an operation of changing the current zoom magnification of the camera, for example, the operation of the user clicking the zoom magnification 1× in the zoom magnification control 202 shown in FIG. 2A .

[0188] S615 . The camera application sends a camera request R2 to the camera interface module 610 .

[0189] The camera application may send a camera request R2 to the camera interface module 610 .

[0190] S616 . The camera interface module 610 sends the camera request R2 to the request queue module 620 .

[0191] The request queue module 620 may obtain the camera request R2 through the setRepeatingRequest() interface or the capture() interface in the camera interface module 610 .

[0192] S617. The request queue module 620 determines that the zoom factor in the camera request R2 has changed.

[0193] The request queue module 620 may determine that the zoom magnification in the camera request R2 has changed. For example, the request queue processing module 620 may compare the camera request R2 with the previous camera request. If the zoom magnification in the previous camera request is 0.6x and the zoom magnification in the camera request R2 is 1.0x, the request queue processing module may determine that the zoom magnification in the camera request R2 has changed.

[0194] S618. The request queue module 620 requests the buffer allocation interface module 640 to obtain a buffer for the camera R2.

[0195] The request queue module 620 may directly request R2 to obtain a buffer for the camera from the buffer allocation interface module 640 .

[0196] S619. The buffer allocation interface module 640 allocates the camera request R2 to buffer (P+1) among the q buffers.

[0197] The buffer allocation interface module 640 may allocate the camera request R2 to buffer (P+1) among the q buffers. For example, buffer (P+1) may be buffer 5 shown in FIG5 . Buffer (P+1) may also be referred to as a first buffer.

[0198] Optionally, when there is an empty buffer among the p buffers, the buffer allocation interface module 640 may also allocate the camera request R2 to an empty buffer among the p buffers.

[0199] S620 . The buffer allocation interface module 640 sends the camera request R2 to the camera hardware abstraction layer 660 in sequence.

[0200] The buffer allocation interface module 640 may send a camera request R2 to the camera hardware abstraction layer 660. Step S620 may refer to the description of step S608 and will not be repeated here.

[0201] S621. The camera hardware abstraction layer 660 obtains the image frame 2 corresponding to the camera request R2 through the camera.

[0202] The camera hardware abstraction layer 660 can obtain the image frame 2 corresponding to the camera request R2 through the camera. As shown in Figure 5, the camera hardware abstraction layer 660 may include a camera request waiting module and a camera hardware abstraction processing module. The camera request to be processed can be queued in the camera request waiting module first. That is, the camera hardware abstraction layer first processes the camera requests in the p buffers previously issued, and then processes the camera request R2. The camera hardware abstraction processing module can send the camera parameters in the camera request R2 being processed to the camera module through the camera driver. The camera module can obtain image frame 2 according to the camera parameters.

[0203] Step S621 may refer to the description in step S609 and will not be repeated here.

[0204] S622 . The camera hardware abstraction layer 660 returns image frame 2 to the display service module 650 .

[0205] The camera hardware abstraction layer 660 may return the image frame 2 to the display service module 650 .

[0206] In a possible implementation, the camera hardware abstraction layer 660 may directly send the acquired image frame 2 to the display service module 650 .

[0207] Optionally, in another possible implementation, the camera hardware abstraction layer 660 may send the address of the buffer storing the image frame 2 to the display service module 650 .

[0208] S623. The display service module 650 sends the image frame 2 for display.

[0209] The display service module 650 can send the image frame 2 to the display through the display driver for display. Optionally, the display service module 650 can send the address of the buffer 22 corresponding to the image frame 2 to be selected to the display through the display driver, and the display can obtain the image frame from the buffer 22 and display it.

[0210] Exemplarily, the image frame 1 may be the preview image 221 shown in FIG. 2C , or the preview image 901 shown in FIG. 9 .

[0211] In the embodiment of the present application, the camera request R2 and the photo request 2 may be referred to as the first camera request, and the camera request R11 and the photo request 1 may be referred to as the second camera request.

[0212] It is understandable that the above steps S614 to S623 are also applicable to the user performing focus operations, exposure, adding filters and other operations in the camera application. The request queue module 620 can distinguish zoom parameters, focus parameters, exposure parameters, and filter parameters from the identifier carried in the camera request. For example, when the camera request carries the identifier android.control.zoomRatio, the request queue module 620 can determine that the identifier android.control.zoomRatio is used to indicate the zoom parameter. When the value of zoomRatio in the previous camera request of camera request R2 is 0.6x and the value of zoomRatio in camera request R2 is 1.0x, the request queue module 620 can determine that the zoom ratio in camera request R2 has changed.

[0213] When the camera request carries the identifier android.control.aeMode, the request queue module 620 may determine that the identifier android.control.aeMode is used to indicate exposure parameters. When the camera request carries the identifier android.control.afRegions, the request queue module 620 may determine that the identifier android.control.afRegions is used to indicate focus parameters. When the camera request carries the identifier android.control.effectMode, the request queue module 620 may determine that the identifier android.control.effectMode is used to indicate filter parameters.

[0214] In implementing a camera request processing method provided by an embodiment of the present application, multiple camera requests for obtaining preview images issued when the camera is turned on are stored in only a portion of the cache (for example, buffer 1, buffer 2, buffer 3, and buffer 4 of the ordinary camera request cache area shown in Figures 5 and 10 (a) and (b)). Another portion of the cache (for example, buffer 5, buffer 6, buffer 7, and buffer 8 of the special camera request cache area shown in Figures 5 and 10 (a) and (b)) is only allocated to camera requests whose first parameters (for example, zoom ratio, focus parameters, exposure parameters, filter parameters, etc.) have changed. In this way, the application framework layer can control the number of requests when the camera first issues a request according to the camera mode. For example, in the photo mode, the application framework layer can allocate only 4 buffers to camera request 1, camera request 2, camera request 3, and camera request 4 among the multiple camera requests for obtaining preview images. In video recording mode, the application framework layer can allocate only 5 buffers to camera request 1, camera request 2, camera request 3, camera request 4, and camera request 5 in multiple camera requests for obtaining preview images. The first request sent by the camera application will not fill up the bufferQueue. When the user changes the zoom ratio, or performs focus, exposure, or adds filters in the camera shooting interface, the electronic device 100 can directly allocate the camera request generated based on the user operation to the special camera request buffer area. The application framework layer does not need to wait for a buffer to be cleared before allocating it to the buffer, nor does the application framework layer need to wait for the HAL to process the next request before allocating a buffer to the camera request. For example, as shown in Figure 10 (b), when the camera APK sends a zoom request, since the bufferQueue is not fully occupied by the first stream (i.e., the first request sent), the camera FWK can immediately obtain an empty buffer and send it to the HAL. Since FWK immediately sends a zoom request to HAL, HAL can apply the zoom request sent by FWK to the generated image frame as soon as possible after receiving it, thereby reducing the time it takes for the electronic device 100 to respond to the user's zoom operation, that is, improving the tracking performance of the user's zoom operation and enhancing the user experience.

[0215] It will be understood that the user interface schematic diagrams provided in the embodiments of the present application are for reference only. The user interfaces shown in the above Figures 1, 2A-2C, and 7-9 may include more or fewer interface elements, and the positions of the interface elements in the user interface may not be limited to the positions shown in the above figures.

[0216] Figure 11 exemplarily shows a schematic diagram of the time it takes for the electronic device 100 to receive an operation from the user to change the zoom ratio, the time it takes for the electronic device 100 to send the request corresponding to the operation to the HAL layer, and the time it takes for the electronic device 100 to respond to the user operation and successfully display the image frame corresponding to the user operation, when the electronic device 100 processes the zoom request according to the camera request processing method shown in Figures 5 and 6 above.

[0217] As shown in Figure 11 (a), after the camera application of the electronic device 100 is turned on, the zoom magnification in the initial camera request (for example, request 1 shown in Figure 11 (a)) issued can be 0.6x. Before the user changes the zoom magnification of the camera, the camera module of the electronic device 100 obtains image frames at a zoom magnification of 0.6x. After the user changes the zoom magnification of the camera to 1.0x, the sensor (for example, a touch sensor) in the screen (also referred to as a display or display screen) in the electronic device 100 can obtain the user's touch operation. After obtaining the user operation, the camera application of the electronic device 100 issues a camera request based on the user operation (such as request n (zoom: 1.0 (indicating that the zoom magnification is 1.0x)) shown in Figure 11). Taking the example of 8 buffers in the cache area for storing camera requests in the electronic device 100, when buffer1-buffer4 are used to store common camera requests and buffer5-buffer8 are used to store special camera requests, the time it takes for a camera request carrying a new zoom factor sent by the camera application in the electronic device 100 to be sent to the HAL layer through the application framework layer is z2. The calculation formula for z2 can be as follows:

[0218] z2=x+v (Formula 3)

[0219] In Formula 3 above, x is the duration it takes for the screen sensor of electronic device 100 to detect a user action, and then for the camera application to convert the action into a corresponding camera request and send it to the FWK. v is the duration it takes for the FWK to allocate a buffer for the zoom request and send it to the HAL.

[0220] As shown in FIG. 11( b ), the duration for the electronic device 100 to receive the user's zoom operation and successfully display the corresponding image frame is T2. The calculation formula for T2 can be as follows:

[0221] T2=x+v+y21+y22+y23+y24+y25 (Formula 4)

[0222] In Formula 4 above, y21 is the duration of time it takes for the HAL to process a request in buffer 1, obtain the corresponding image frame 1101, return image frame 1101 to FWK, and successfully display image frame 1101. y22 is the duration of time it takes for the HAL to process a request in buffer 2, obtain the corresponding image frame 1102, return image frame 1102 to FWK, and successfully display image frame 1102. y23 is the duration of time it takes for the HAL to process a request in buffer 3, obtain image frame 1103, return image frame 1103 to FWK, and successfully display image frame 1103. y24 is the duration of time it takes for the HAL to process a request in buffer 4, obtain image frame 1104, return image frame 1104 to FWK, and successfully display image frame 1104. y25 is the time duration for HAL to process the zoom request, obtain the image frame 1105 corresponding to the zoom request, and return the image frame 1105 to FWK, and for FWK to successfully display the image frame 1105.

[0223] Compared with z1 and T1 shown in FIG4 , z2 in FIG11 is smaller than z1, and T2 is smaller than T1. That is, when the electronic device 100 processes the zoom request according to the camera request processing method shown in FIG5 and FIG6 , the time from when the electronic device 100 receives the user's operation to change the zoom ratio to when the electronic device 100 sends the request corresponding to the operation to the HAL layer, and the time required for the electronic device 100 to respond to the user operation and successfully send the image frame corresponding to the user operation are reduced. When a camera request processing method provided in an embodiment of the present application is implemented, after the camera APK sends a zoom request, since the bufferQueue is not fully occupied by the first stream (i.e., the first request sent), the camera FWK can immediately obtain an empty buffer and send it to the HAL. Since FWK immediately sends a zoom request to HAL, HAL can apply the zoom request sent by FWK to the generated image frame as soon as possible after receiving it, thereby reducing the time it takes for the electronic device 100 to respond to the user's zoom operation, that is, improving the tracking performance of the user's zoom operation and enhancing the user experience.

[0224] The following introduces an exemplary electronic device 100 provided in an embodiment of the present application.

[0225] FIG12 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0226] The following embodiments are described in detail using electronic device 100 as an example. It should be understood that electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in the figure may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0227] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0228] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0229] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0230] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0231] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0232] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0233] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device 100.

[0234] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0235] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0236] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0237] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0238] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0239] The SIM interface can be used to communicate with the SIM card interface 195 to implement the function of transmitting data to the SIM card or reading data in the SIM card.

[0240] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0241] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0242] The charging management module 140 is configured to receive charging input from a charger, which may be a wireless charger or a wired charger.

[0243] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0244] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0245] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0246] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0247] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0248] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0249] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0250] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0251] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0252] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0253] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0254] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0255] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0256] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0257] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0258] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0259] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0260] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0261] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0262] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0263] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0264] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0265] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0266] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0267] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0268] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0269] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0270] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0271] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0272] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0273] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0274] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0275] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to implement a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 reduces the performance of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0276] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0277] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0278] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0279] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0280] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications.

[0281] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0282] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0283] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0284] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A camera request processing method, characterized in that: The method is applied to an electronic device having a camera module, and the method comprises: Detecting the user's operation of changing the camera parameters, sending a first camera request; Determining that the camera parameters in the first camera request are different from the camera parameters in the second camera request, allocating the first camera request to a first cache buffer in a first cache area, wherein a storage area in the electronic device for storing camera requests is divided into the first cache area and a second cache area, the first cache area is used to store the first camera request, the camera parameters include one or more of a zoom ratio, a focus parameter, an exposure parameter, and a filter parameter, and the second camera request is a camera request obtained by the electronic device before obtaining the first camera request; Acquire a first image frame according to the camera parameters in the first camera request; The first image frame is displayed.

2. The method according to claim 1, characterized in that The first buffer area includes q buffers, the second buffer area includes p buffers, and the method further includes: The values ​​of q and p are determined based on a camera mode of a camera application in the electronic device.

3. The method according to claim 2, characterized in that The determining the values ​​of q and p based on the camera mode of the camera application in the electronic device includes: When the camera mode of the camera application is a photo mode, determining that the value of q is a first value and the value of p is a second value; When the camera mode of the camera application is the video recording mode, the value of q is determined to be a third value, the value of p is determined to be a fourth value, the sum of the third value and the fourth value is a fifth value, the sum of the first value and the second value is a sixth value, and the fifth value is equal to the sixth value.

4. The method according to any one of claims 1 to 3, characterized in that: Before the detection of the user changing the camera parameters and sending the first camera request, the method further includes: The electronic device detects an operation of starting the camera application and starts the camera application; The camera application in the electronic device sends a plurality of camera requests, wherein the plurality of camera requests include the second camera request; When there is an empty buffer in the second buffer area, allocating the second camera request to a second buffer, where the second buffer is an empty buffer in the second buffer area; When there is no empty buffer in the second buffer area, the electronic device waits for the second buffer in the second buffer area to be cleared and then allocates the second camera request to the second buffer.

5. The method according to claim 1, characterized in that After determining that the camera parameters in the first camera request are different from the camera parameters in the second camera request, the method further includes: When there is an empty buffer in the second buffer area, the first camera request is allocated to a second buffer in the second buffer area, and the second buffer is an empty buffer.

6. The method according to claim 5, characterized in that The operation of the user changing the first camera parameter is an operation of the user changing the zoom ratio, and the method further includes: The camera application displays a first preview interface, the first preview interface displays the second image frame, and the zoom ratio corresponding to the second image frame is a first zoom ratio; The camera detects a user's operation of changing the zoom ratio and displays a second preview interface, in which the first image frame is displayed. The zoom ratio corresponding to the first image frame is a second zoom ratio.

7. The method according to any one of claims 1 to 6, characterized in that: The electronic device comprises a request module, a cache module, The determining that the camera parameters in the first camera request are different from the camera parameters in the second camera request, and allocating the first camera request to a first cache buffer in a first cache area, includes: The request module determines that camera parameters in the first camera request are different from camera parameters in the second camera request; The request module sends the first camera request to the cache module; The cache module allocates the first camera request to a first cache buffer in a first cache area.

8. The method according to claim 7, characterized in that The electronic device further includes a camera processing module. After the cache module allocates the first camera request to a first cache buffer in the first cache area, the method further includes: The cache module sends the first camera request to the camera processing module; Acquiring a first image frame according to the camera parameters in the first camera request includes: The camera processing module sends the camera parameters in the first camera request to the camera module; The camera module obtains a first image frame according to the camera parameters in the first camera request.

9. The method according to claim 4, characterized in that The electronic device further includes a cache module and a camera processing module. After allocating the second camera request to the second buffer, the method further includes: The cache module sends the second camera request to the camera processing module.

10. The method according to claim 9, characterized in that The electronic device further includes a camera application and a display module. After the cache module sends the second camera request to the camera processing module, the method further includes: The camera processing module sends the camera parameters in the second camera request to the camera module, and obtains a second image frame obtained by the camera module according to the camera parameters in the second camera request; The camera processing module sends the second image frame to the display module; The display module displays the second image frame; The camera application displays the second image frame.

11. An electronic device, characterized in that: It comprises a camera, one or more processors and one or more memories; wherein the camera, the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method as described in any one of claims 1-10 is executed.

12. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is configured to call computer instructions so as to execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 10 is executed.

Images