Image capturing methods, devices and electronic equipment

Abstract

This application discloses an image capturing method, apparatus, and electronic device, belonging to the field of image capturing technology. The image capturing method is applied to an electronic device whose operating system includes an application layer, a framework layer, and a hardware abstraction layer. The application layer includes a camera application, and the hardware abstraction layer includes an information transmission module, an image sensor, and a processing module. The image capturing method includes: the image sensor sending a captured first face image to a face detection module, which is located in the application layer or framework layer; the face detection module performing face detection on the first face image to obtain face detection information, and sending the face detection information to the processing module through the information transmission module; the processing module generating a face image capturing command based on the face detection information and sending it to the image sensor; and the image sensor capturing a second face image based on the face image capturing command and sending the second face image to the camera application for display.

Description

Technical Field

[0001] This application belongs to the field of imaging technology, specifically relating to an image capturing method, apparatus, and electronic device. Background Technology

[0002] With the development of computer technology, people have increasingly higher requirements for portrait photography, especially for capturing facial images. Capturing facial images that meet user needs often relies on facial coordinate information, which can be obtained through face detection algorithms. Therefore, the accuracy and real-time performance of face detection results directly affect the processing effect of facial images.

[0003] To meet users' real-time requirements, the current approach is to couple face detection algorithms to the hardware abstraction layer of electronic devices. However, since the hardware abstraction layer is updated with version or manufacturer changes, the face detection algorithm needs to be recoupled to the hardware abstraction layer, resulting in high maintenance costs for the hardware abstraction layer. Summary of the Invention

[0004] The purpose of this application is to provide an image capturing method, apparatus, and electronic device that can solve the problem that related technologies cannot balance real-time performance and maintenance costs when capturing facial images.

[0005] In a first aspect, embodiments of this application provide an image capturing method applied to an electronic device. The electronic device includes an operating system, which includes an application layer, a framework layer, and a hardware abstraction layer. The application layer is provided with a camera application, and the hardware abstraction layer is provided with an information transmission module, an image sensor, and a processing module.

[0006] The method includes:

[0007] The image sensor sends the captured first face image to the face detection module, which is set at the application layer or framework layer.

[0008] The face detection module performs face detection on the first face image, obtains face detection information, and sends the face detection information to the processing module through the information transmission module;

[0009] The processing module generates a face image capture command based on the face detection information and sends the face image capture command to the image sensor;

[0010] The image sensor captures a second face image according to the face image capture command and sends the second face image to the camera application for display.

[0011] Secondly, embodiments of this application provide an image capturing device for use in electronic devices. The device includes: an image sensor, a face detection module, an information transmission module, and a processing module.

[0012] An image sensor, located in the hardware abstraction layer of an electronic device, is used to capture a first face image and send the first face image to a face detection module;

[0013] The face detection module is set in the application layer or framework layer of the electronic device. It is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the information transmission module.

[0014] The information transmission module, located in the hardware abstraction layer, is used to send face detection information to the processing module, which is also located in the hardware abstraction layer.

[0015] The processing module is used to generate a face image capture command based on the face detection information and send the face image capture command to the image sensor;

[0016] The image sensor is also used to capture a second face image according to a face image capture instruction, and send the second face image to the camera application of the electronic device for display. The camera application is set at the application layer.

[0017] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect.

[0018] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0019] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the steps of the method as described in the first aspect.

[0020] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method as described in the first aspect.

[0021] In this embodiment, the electronic device includes an operating system, which comprises an application layer, a framework layer, and a hardware abstraction layer. The application layer includes a camera application, and the hardware abstraction layer includes an information transmission module, an image sensor, and a processing module. The face detection module is located in the application layer or framework layer other than the hardware abstraction layer, thus decoupling the face detection module from the hardware abstraction layer and reducing the maintenance cost of the hardware abstraction layer. By setting an information transmission module in the hardware abstraction layer, the face detection information obtained by the face detection module can be directly transmitted to the processing module in real time through the information transmission module within the hardware abstraction layer, without having to execute according to the request queue maintained by the traditional hardware abstraction layer, thereby reducing the feedback time of face detection information and ensuring real-time performance. Thus, the solution provided by this embodiment can simultaneously address the maintenance cost of the hardware abstraction layer and the feedback time of face detection information. Attached Figure Description

[0022] Figure 1 A schematic diagram of a request queue provided for related technologies;

[0023] Figure 2a A schematic diagram of the software architecture layer of an electronic device provided in an embodiment of this application;

[0024] Figure 2b A schematic diagram illustrating the decoupling of a face detection algorithm from a hardware abstraction layer, as provided in an embodiment of this application;

[0025] Figure 3 A flowchart illustrating an image capturing method provided in this application embodiment;

[0026] Figure 4 A schematic diagram illustrating the processing procedure of an information transmission module provided in an embodiment of this application;

[0027] Figure 5 A schematic diagram of the software architecture layer of another electronic device provided in an embodiment of this application;

[0028] Figure 6 A flowchart illustrating another image capturing method provided in this application embodiment;

[0029] Figure 7 This is a schematic diagram of the structure of an image capturing device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0033] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0034] When capturing facial images, users have increasingly higher requirements for the quality of the images. For example, users want the exposure and focus of the captured facial images to be adjusted in real time according to the shooting scene. Capturing facial images that meet user needs often relies on facial coordinate information, which can be obtained through face detection algorithms.

[0035] To meet users' real-time requirements, the current approach primarily involves coupling face detection algorithms to the hardware abstraction layer (HAL) of electronic devices. However, due to version or manufacturer updates, the HAL needs continuous upgrades. This necessitates recoupling the face detection algorithm to the HAL, increasing its maintenance costs.

[0036] Some solutions decouple the face detection algorithm from the hardware abstraction layer, for example, coupling the face detection algorithm to a layer other than the hardware abstraction layer. Since the frame rate of camera output is relatively fast, typically 30fps, 15fps, or 60fps, to ensure smooth image display, the hardware abstraction layer needs to maintain a request queue to ensure that images from multiple cameras are processed in parallel in a pipeline, so that the camera application displays the image at the same frame rate as the camera output, resulting in smooth display.

[0037] Typically, the request queue depth is around 7. Figure 1 Taking request queue 100 with a depth of 7 as an example, meaning request queue 100 contains 7 requests, and requestN represents the Nth frame image output by the camera. According to... Figure 1As shown in request queue 100, the face detection information of the first frame can only be fed back to the subsequent processing modules and presented in the eighth frame image after all 7 frames in request queue 100 have been processed. That is, only after requests 1-7 have been processed can the face detection information of request 1 be fed back to the next module, allowing the camera to capture the image corresponding to request 8 based on the face detection information of request 1. In other words, the face detection information of request 1 is applied to request 8, the face detection information of request 2 is applied to request 9, and so on. Each face detection information requires a 7-frame delay before it can be used, thus affecting the real-time performance of camera exposure and focus.

[0038] The image output from request queue 100 can be processed sequentially through RAW processing module 101, RAW2YUV 102, YUV processing module 103 and YUV2JPG 104 to finally obtain a JPG format image, which is then output to the user.

[0039] To address this issue, this application provides an image capturing method, apparatus, and electronic device, resolving the problem that related technologies cannot balance real-time performance and maintenance costs when capturing facial images.

[0040] The image capture method provided in this application can be applied to electronic devices such as mobile phones, tablets, and laptops, and can be applied to scenarios that require real-time exposure and focusing of captured face images. Of course, it can also be applied to scenarios that require a balance between real-time performance and maintenance costs, such as human eye detection, object detection, and pet detection.

[0041] Electronic devices may include operating systems, such as Figure 2a As shown, the operating system 201 may include, but is not limited to, Android, Linux, and Windows systems. Taking Android as an example, the operating system 201 may include an application layer 2010, a framework layer 2011, and a hardware abstraction layer 2012. In practical applications, the operating system 201 may also include other layers, such as a kernel layer and a hardware layer. Layers communicate with each other through software interfaces.

[0042] The application layer 2010 can include a series of application packages. Application packages may include, for example, applications such as camera, gallery, calendar, and calling.

[0043] The Framework Layer 2011 provides application programming interfaces and programming frameworks for applications in the Application Layer 2010.

[0044] Hardware Abstraction Layer 2012 runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0045] In this embodiment of the application, the hardware abstraction layer 2012 may be provided with an information transmission module 20121, an image sensor 20122, and a processing module 20123.

[0046] The information transmission module 20121 is used to distribute face detection information to the processing module 20123 in real time within the hardware abstraction layer 2012, bypassing the traditional request queue and achieving high real-time performance. The face detection information is obtained by the face detection module set in the application layer 2010 or the framework layer 2011.

[0047] Image sensor 20122 is used to capture images. Processing module 20123, also known as the 3A module, is used to determine exposure time and gain. 3A refers to Auto Focus (AF), Auto Exposure (AE), and Auto White Balance (AWE).

[0048] This embodiment of the application decouples the face detection module from the hardware abstraction layer 2012 and adds an information transmission module 20121 to the hardware abstraction layer 2012, which can simultaneously take into account the maintenance cost of the hardware abstraction layer and the real-time exposure and focus of the face.

[0049] Figure 2b An exemplary diagram illustrating the decoupling of a face detection algorithm from a hardware abstraction layer is provided. Figure 3 Taking the face detection algorithm coupled to the framework layer 2011 as an example, the framework layer 2011 can be configured with a face detection module. The face detection module 20111 can call the face detection algorithm to perform face detection on the face image. Taking an electronic device using the Android system as an example, the face detection module 20111 and the hardware abstraction layer 2012 can communicate through the Android standard interface.

[0050] For example, the framework layer 2011 can send a configuration request to the hardware abstraction layer 2012 according to the user's configuration requirements, in order to request the hardware abstraction layer 2012 to configure different image streams. For example, it can configure a preview stream corresponding to a preview image and a shooting stream for a captured image. The screen size, display position and other parameters corresponding to the preview stream and the shooting stream can be different.

[0051] For example, the framework layer 2011 can send a stream acquisition request to the hardware abstraction layer 2012 based on the user's information stream requirements. Upon receiving the stream acquisition request, the hardware abstraction layer 2012 can obtain the parameters of the stream corresponding to the request and send them to the framework layer 2011. The framework layer 2011 then sends these parameters to the camera application, which displays the corresponding image stream based on the stream parameters. Once the camera application receives the stream parameters, it can display the corresponding image stream according to those parameters.

[0052] By decoupling the face detection algorithm from the hardware abstraction layer, the repeated maintenance costs of re-porting and integrating the face detection algorithm are avoided, which are caused by changes in the hardware abstraction layer as it changes with versions and manufacturers.

[0053] The image capturing method, apparatus, and electronic device provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0054] Figure 3 A flowchart of an image capturing method provided in an embodiment of this application is shown below. Figure 3 As shown, the image capturing method may include the following steps:

[0055] S310, the image sensor sends the captured first face image to the face detection module.

[0056] S320 The face detection module performs face detection on the first face image, obtains face detection information, and sends the face detection information to the processing module through the information transmission module.

[0057] S330: The processing module generates a face image capture command based on the face detection information and sends the face image capture command to the image sensor.

[0058] S340: The image sensor captures a second face image according to the face image capture command and sends the second face image to the camera application for display.

[0059] The above steps are explained in detail below:

[0060] In S310, the first face image is the original image captured by the image sensor. In this embodiment, the image sensor can send the captured first face image to the face detection module, which then performs face detection on the first face image according to the face detection algorithm.

[0061] The face detection module can be set in the application layer or the framework layer. This decouples the face detection module, i.e. the face detection algorithm, from the hardware abstraction layer. When the hardware abstraction layer changes due to version or manufacturer changes, the face detection algorithm does not need to be recoupled to the hardware abstraction layer, thus reducing the maintenance cost and upgrade difficulty of the hardware abstraction layer.

[0062] Taking the Android system as an example, in some embodiments, the face detection module is set in the framework layer and adopts the Android standard interface.

[0063] In S320, upon receiving the first face image, the face detection module can invoke a face detection algorithm to perform face detection on the first face image, obtaining face detection information, which may include, for example, the position and size of the face. This application embodiment does not limit the specific face detection algorithm.

[0064] After obtaining face detection information, the face detection module can send the information to the information transmission module within the hardware abstraction layer (HAL) through a corresponding interface. The information transmission module then forwards the information to the processing module within the HAL, i.e., the 3A module. In other words, after sending the face detection information to the HAL, the face detection module in this embodiment can distribute the information to the 3A module in real time via the information transmission module, bypassing the camera application's request queue and improving real-time performance.

[0065] It should be noted that the information transmission module added in this application embodiment will not break the native camera process of Android, that is, the images captured by the image sensor will still be sent into the request queue for execution.

[0066] like Figure 4 As shown, without changing the native camera process, the image output by request1 can be directly sent to the information transmission module 20121 after passing through the face detection module 20111, and then sent to request2 by the information transmission module 20121 through the processing module 20123. That is, the face detection information of the face image output by request1 can be directly applied to request2, thus reducing latency. Figure 4 Taking the face detection module 20111 set in the frame layer 2011 as an example.

[0067] exist Figure 4 In the example, the face detection information output by the face detection module 20111 can also be applied to the RAW processing module 101, RAW2YUV 102, YUV processing module 103, and YUV2JPG to perform face swapping, beautification, and other processing on the face image based on the face detection information.

[0068] In S330, the face image capture command is used to instruct the image sensor to capture a face image that meets the user's requirements. After receiving the face detection information, the 3A module can generate the face image capture command based on the face detection information. In some embodiments, the face image capture command may include the exposure duration and / or gain of the image sensor. The exposure duration and gain will vary depending on the scene in which the face image is located.

[0069] For example, the 3A module can look up the exposure table based on face detection information and scene to obtain the exposure duration corresponding to that face detection information and scene; the gain processing is similar. The exposure table is used to store the relationship between multiple face detection information and scene and exposure duration.

[0070] After the 3A module generates a face image capture command, it can directly send the face image capture command to the image sensor in the hardware abstraction layer, and the image sensor will capture the image according to the face image capture command.

[0071] In S340, after receiving a face image capture command, the image sensor can capture a second face image according to the face image capture command. That is, the image sensor can capture an image according to the new exposure time and gain to obtain a second face image with an exposure time and gain that meets the user's requirements.

[0072] The above method significantly reduces the number of frames between the second face image and the first face image, thus improving real-time performance. This allows the face detection information from the initially captured face image to be promptly fed back into the next frame, further enhancing real-time performance.

[0073] In some embodiments, after the image sensor captures a second face image according to the face image capture instruction, the second face image can be sent to the camera application in the application layer for display.

[0074] In some embodiments, the image sensor can also send the second face image to the face detection module, that is, perform the processing of the first face image, and feed back the face detection information of the second face image to the next frame image, thereby achieving the effect of updating the captured image in real time according to the face detection information and meeting the user's real-time requirements.

[0075] In some embodiments, such as Figure 5As shown, the hardware abstraction layer 2012 also includes an image signal processing (ISP) module 20124. The image signal processing module 20124 communicates with the image sensor 20122, the face detection module 20111, and the information transmission module 20121. The information transmission module 20121 also communicates with the processing module 20123, and the processing module 20123 also communicates with the image sensor 20122.

[0076] The image signal processing module 20124 is mainly used to perform noise reduction and other processing on the raw image output by the image sensor 20122 in order to improve the accuracy of subsequent face detection results and thus improve the face image shooting effect.

[0077] according to Figure 5 The architecture shown Figure 6 A flowchart illustrating another image capturing method provided in this application embodiment. Figure 6 and Figure 3 The difference is that, Figure 3 S310 in the text can be further refined into Figure 6 S610-S620 in the middle, Figure 3 The S320 in the text can be further refined into Figure 6 The S630 in the middle.

[0078] S610: The image sensor sends the captured first face image to the image signal processing module.

[0079] In some embodiments, the image sensor may also send the captured first face image to the ISP of the hardware abstraction layer, whereby the ISP preprocesses the first face image to improve its quality. The specific processing steps are not limited in this application embodiment.

[0080] S620: The image signal processing module performs image processing on the first face image and sends the processed first face image to the face detection module.

[0081] In some embodiments, the processed first face image can be a YUV format image. That is, in this application embodiment, the YUV format face image can be directly output by the ISP and sent directly to the face detection module for face detection, thereby improving the real-time performance while improving image quality.

[0082] S630: The face detection module performs face detection on the processed first face image to obtain face detection information. Then, the face detection module can send the face detection information to the processing module via the information transmission module.

[0083] Since the image quality of the processed first face image is improved, the accuracy of face detection information can be improved by using the face detection module to perform face detection on the processed first face image.

[0084] In some embodiments, the image capturing method may further include the following steps:

[0085] The information transmission module sends face detection information to the image signal processing module. That is, the information transmission module can send face detection information not only to the processing module in real time, but also to the image signal processing module in real time.

[0086] For example, the information transmission module can first send the face detection information to the processing module, and then send the face detection information to the image signal processing module. Alternatively, it can first send the face detection information to the image signal processing module, and then send the face detection information to the processing module. Or it can send the face detection information to both the image signal processing module and the processing module simultaneously.

[0087] Based on this, in some embodiments, the above-described S620 may include the following steps:

[0088] The image signal processing module determines the face region of the first face image based on the face detection information sent by the information transmission module;

[0089] The image signal processing module performs image processing on the face region to obtain the first processed face image.

[0090] For example, after receiving the face detection information sent by the information transmission module, the image signal processing module can determine the face region of the first face image based on the face detection information. The face region is the region that contains all faces. The face region can be represented by a face bounding box, which can be a rectangle or other shapes.

[0091] In other words, the image signal processing module can first determine the face region from the first face image based on the face detection information, providing a basis for subsequent image processing.

[0092] After obtaining the face region, the image signal processing module can perform noise reduction and other processing on the face region to obtain the processed first face image, which is also the third face image.

[0093] This embodiment of the application sends face detection information to the image signal processing module in real time, enabling the image signal processing module to perform targeted processing on the first face image based on the face detection information, thereby reducing the computational load of the image signal processing module and improving the image processing efficiency.

[0094] In some embodiments, prior to S310, the image capturing method may further include the following steps:

[0095] The camera application sends a shooting request to the hardware abstraction layer through the framework layer;

[0096] The hardware abstraction layer sends the capture request to the image sensor;

[0097] The image sensor captures the first face image based on the shooting request.

[0098] In this embodiment, when the camera application receives user input for shooting, it can send a shooting request to the hardware abstraction layer through the framework layer. Shooting input may include, but is not limited to, touch input from the user via a finger or stylus, voice commands, specific gestures, or other feasible inputs. The specific input can be determined based on actual usage needs, and this embodiment does not impose limitations. Specific gestures in this embodiment can be any one of the following: single-click gesture, swipe gesture, drag gesture, pressure-recognition gesture, long-press gesture, area-change gesture, double-press gesture, or double-tap gesture. Click input in this embodiment can be single-click input, double-tap input, or any number of clicks, and can also be long-press input or short-press input. For example, the shooting input mentioned above can be: user click input to the camera application.

[0099] After receiving a shooting request, the hardware abstraction layer can notify the image sensor to take a picture, thereby obtaining the first face image.

[0100] This application embodiment, while balancing maintenance costs and real-time performance, satisfies the user's shooting needs through the interaction between layers.

[0101] In some embodiments, the above-mentioned "sending the second face image to the camera application for display" may include the following steps:

[0102] The image sensor sends the second face image to the frame layer;

[0103] The framework layer performs image processing on the second face image to obtain the third face image, and then sends the third face image to the camera application for display.

[0104] In some embodiments, the electronic device can further process the second face image to obtain an image that meets the user's needs. For example, the second face image can be blurred, beautified, or have fun face-swapping effects, and the processed second face image, i.e., the third face image, can be displayed to the user through a camera application, thus satisfying the user's personalized shooting needs.

[0105] The process of image capture is described below.

[0106] Image sensor 20122 can send the captured face image to ISP 20124. ISP 20124 can preprocess the face image sent by image sensor 20122 and send the processed face image to RAW processing module 101 for further processing. Then, it is sent to RAW2YUV 102, YUV processing module 103 and YUV2JPG for processing in sequence, and finally the JPG format image is displayed to the user.

[0107] The ISP20124 can also send the processed face image to the face detection module 20111, which performs face detection on the processed face image to obtain face detection information. The face detection module 20111 can then send the face detection information to the RAW processing module 101 or the YUV processing module 103. The RAW processing module 101 or the YUV processing module 103 can then further process the processed face image sent by the ISP20124 based on the face detection information. For example, it can perform face swapping, beautification, or other processing on the processed face image based on the face detection information.

[0108] The face detection module 20111 can also send face detection information to the information transmission module 20121. After receiving the face detection information, the information transmission module 20121 can send the face detection information to the processing module 20123. The processing module 20123 generates a face image capturing command based on the face detection information and sends the face image capturing command to the image sensor 20122. In this way, the image sensor 20122 can capture a face image that meets the user's needs according to the face image capturing command, reducing latency.

[0109] This application embodiment reduces the maintenance cost of the hardware abstraction layer by decoupling the face detection module from the hardware abstraction layer. At the same time, an information transmission module is added to the hardware abstraction layer, which enables the face detection information obtained by the face detection module to be distributed to the 3A module in real time within the hardware abstraction layer through the information transmission module. This bypasses the original request queue, reduces latency, and improves real-time performance.

[0110] It should be noted that the image capturing method provided in this application embodiment can be executed by an image capturing device or a processing module within that image capturing device for executing the image capturing method. This application embodiment uses an image capturing device executing the image capturing method as an example to illustrate the image capturing device provided in this application embodiment.

[0111] Figure 7 This is a schematic diagram of the structure of an image capturing device provided in an embodiment of this application.

[0112] like Figure 7 As shown, the image capturing device 700 may include: an image sensor 701, a face detection module 702, an information transmission module 703, and a processing module 704;

[0113] Image sensor 701, located in the hardware abstraction layer of the electronic device, is used to capture a first face image and send the first face image to the face detection module;

[0114] The face detection module 702 is set in the application layer or framework layer of the electronic device. It is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the information transmission module 703.

[0115] The information transmission module 703, located in the hardware abstraction layer, is used to send face detection information to the processing module 704, which is also located in the hardware abstraction layer.

[0116] Processing module 704 is used to generate a face image capturing command based on face detection information and send the face image capturing command to image sensor 701;

[0117] Image sensor 701 is also used to capture a second face image according to a face image capture instruction, and send the second face image to a camera application of an electronic device for display, the camera application being set at the application layer.

[0118] In this embodiment, the electronic device includes an operating system, which comprises an application layer, a framework layer, and a hardware abstraction layer. The application layer includes a camera application, and the hardware abstraction layer includes an information transmission module, an image sensor, and a processing module. The face detection module is located in the application layer or framework layer other than the hardware abstraction layer, thus decoupling the face detection module from the hardware abstraction layer and reducing the maintenance cost of the hardware abstraction layer. By setting an information transmission module in the hardware abstraction layer, the face detection information obtained by the face detection module can be directly transmitted to the processing module in real time through the information transmission module within the hardware abstraction layer, without having to execute according to the request queue maintained by the traditional hardware abstraction layer, thereby reducing the feedback time of face detection information and ensuring real-time performance. Thus, the solution provided by this embodiment can simultaneously address the maintenance cost of the hardware abstraction layer and the feedback time of face detection information.

[0119] In some possible implementations of the embodiments of this application, the image capturing device 700 may further include:

[0120] Image signal processing module, which is located in the hardware abstraction layer;

[0121] Image sensor 701 is also used to send the first face image to the image signal processing module;

[0122] The image signal processing module is used to perform image processing on the first face image and send the processed first face image to the face detection module 702;

[0123] The face detection module 702 is specifically used to: perform face detection on the processed first face image to obtain face detection information.

[0124] In some possible implementations of the embodiments of this application, the information transmission module 703 is further configured to send face detection information to the image signal processing module;

[0125] The image signal processing module is specifically used for:

[0126] Based on the face detection information sent by the information transmission module, the face region of the first face image is determined;

[0127] Image processing is performed on the face region to obtain the first processed face image.

[0128] In some possible implementations of the embodiments of this application, the image sensor 701 is specifically used for:

[0129] Receive shooting requests sent by the hardware abstraction layer. The shooting requests are sent by the camera application to the hardware abstraction layer through the framework layer.

[0130] Capture the first facial image as requested.

[0131] In some possible implementations of the embodiments of this application, the image sensor 701 is specifically used for:

[0132] The second face image is sent to the frame layer so that the frame layer can process the second face image to obtain the third face image, and then send the third face image to the camera application for display.

[0133] The image capturing device in this application embodiment can be a device or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0134] The electronic device in this application embodiment can be a terminal with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0135] The image capturing device provided in this application embodiment can achieve... Figures 1 to 6 The various processes in the image capture method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0136] like Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores programs or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described image capturing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0137] It should be noted that the electronic devices in the embodiments of this application include the mobile terminals and non-mobile terminals mentioned above.

[0138] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0139] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0140] The processor 910 includes an application layer, a framework layer, and a hardware abstraction layer. The application layer includes a camera application, and the hardware abstraction layer includes an information transmission module, an image sensor, and a processing module. In some embodiments, the framework layer includes a face detection module.

[0141] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The structure of the electronic device 900 shown does not constitute a limitation on the electronic device 900. The electronic device 900 may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0142] The image sensor is used to send the captured first face image to the face detection module, which is located in the application layer or the framework layer.

[0143] The face detection module is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the processing module through the information transmission module;

[0144] The processing module is used to generate a face image capture command based on the face detection information and send the face image capture command to the image sensor;

[0145] The image sensor is also used to capture a second face image according to the face image capture instruction, and send the second face image to the camera application for display.

[0146] This application's embodiments decouple the face detection module from the hardware abstraction layer, reducing the maintenance cost of the hardware abstraction layer. Furthermore, by setting up an information transmission module within the hardware abstraction layer, the face detection information obtained by the face detection module can be directly transmitted to the processing module in real time through this module, eliminating the need for the request queue maintained by the traditional hardware abstraction layer. This reduces the feedback time of face detection information and ensures real-time performance. Thus, the solution provided by this application's embodiments can simultaneously address both the maintenance cost of the hardware abstraction layer and the feedback time of face detection information.

[0147] In some possible implementations of the embodiments of this application, the hardware abstraction layer is further provided with an image signal processing module;

[0148] Specifically, the image sensor is used to send the captured first face image to the image signal processing module;

[0149] The image signal processing module is used to process the first face image and send the processed first face image to the face detection module;

[0150] The face detection module is specifically used to: perform face detection on the processed first face image to obtain face detection information.

[0151] In some possible implementations of the embodiments of this application, the information transmission module is further used to send face detection information to the image signal processing module;

[0152] The image signal processing module is specifically used to: determine the face region of the first face image based on the face detection information sent by the information transmission module; and perform image processing on the face region to obtain the processed first face image.

[0153] In some possible implementations of the embodiments of this application, the image sensor is specifically used to: receive a shooting request sent by the camera application through the frame layer before sending the captured first face image to the face detection module; the shooting request is sent by the camera application through the frame layer to the hardware abstraction layer; and capture the first face image according to the shooting request.

[0154] In some possible implementations of the embodiments of this application, the image sensor is specifically used to: send a second face image to the frame layer, so that the frame layer performs image processing on the second face image to obtain a third face image, and send the third face image to the camera application for display.

[0155] This application embodiment reduces the maintenance cost of the hardware abstraction layer by decoupling the face detection module from the hardware abstraction layer. At the same time, an information transmission module is added to the hardware abstraction layer, which enables the face detection information obtained by the face detection module to be distributed to the 3A module in real time within the hardware abstraction layer through the information transmission module. This bypasses the original request queue, reduces latency, and improves real-time performance.

[0156] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0157] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0158] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0159] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image capturing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0160] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0161] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described image capturing method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0162] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0163] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image capturing method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0164] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image capturing method, applied to electronic devices, characterized in that, The electronic device includes an operating system, which includes an application layer, a framework layer, and a hardware abstraction layer. The application layer includes a camera application, and the hardware abstraction layer includes an information transmission module, an image sensor, and a processing module. The method includes: The image sensor sends the captured first face image to the face detection module, which is located in the application layer or the framework layer. The face detection module performs face detection on the first face image to obtain face detection information, and sends the face detection information to the processing module through the information transmission module; The processing module generates a face image capture command based on the face detection information and sends the face image capture command to the image sensor; The image sensor captures a second face image according to the face image capture command, and sends the second face image to the camera application for display.

2. The method according to claim 1, characterized in that, The hardware abstraction layer also includes an image signal processing module; The image sensor sends the captured first face image to the face detection module, including: The image sensor sends the captured first face image to the image signal processing module; The image signal processing module performs image processing on the first face image and sends the processed first face image to the face detection module; The face detection module performs face detection on the first face image to obtain face detection information, including: The face detection module performs face detection on the processed first face image to obtain face detection information.

3. The method according to claim 2, characterized in that, The method further includes: The information transmission module sends the face detection information to the image signal processing module; The image signal processing module performs image processing on the first face image, including: The image signal processing module determines the face region of the first face image based on the face detection information sent by the information transmission module; The image signal processing module performs image processing on the face region to obtain the processed first face image.

4. The method according to any one of claims 1-3, characterized in that, Before the image sensor sends the captured first face image to the face detection module, the method further includes: The camera application sends a shooting request to the hardware abstraction layer through the framework layer; The hardware abstraction layer sends the shooting request to the image sensor; The image sensor captures the first face image according to the shooting request.

5. The method according to any one of claims 1-3, characterized in that, Sending the second face image to the camera application for display includes: The image sensor sends the second face image to the frame layer; The framework layer performs image processing on the second face image to obtain a third face image, and sends the third face image to the camera application for display.

6. An image capturing device, applied to electronic equipment, characterized in that, The device includes: an image sensor, a face detection module, an information transmission module, and a processing module; The image sensor is disposed in the hardware abstraction layer of the electronic device and is used to capture a first face image and send the first face image to the face detection module; The face detection module is located in the application layer or framework layer of the electronic device and is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the information transmission module. The information transmission module is located in the hardware abstraction layer and is used to send the face detection information to the processing module, which is located in the hardware abstraction layer. The processing module is used to generate a face image capturing instruction based on the face detection information, and send the face image capturing instruction to the image sensor; The image sensor is also used to capture a second face image according to the face image capture instruction, and send the second face image to the camera application of the electronic device for display, wherein the camera application is set in the application layer.

7. The apparatus according to claim 6, characterized in that, The device further includes: an image signal processing module, which is disposed in the hardware abstraction layer; The image sensor is also used to send the first face image to the image signal processing module; The image signal processing module is used to perform image processing on the first face image and send the processed first face image to the face detection module; The face detection module is specifically used to: perform face detection on the processed first face image to obtain face detection information.

8. The apparatus according to claim 7, characterized in that, The information transmission module is also used to send the face detection information to the image signal processing module; The image signal processing module is specifically used for: The face region of the first face image is determined based on the face detection information sent by the information transmission module. The face region is processed to obtain the processed first face image.

9. The apparatus according to any one of claims 6-8, characterized in that, The image sensor is specifically used for: Receive a shooting request sent by the hardware abstraction layer, wherein the shooting request is sent by the camera application to the hardware abstraction layer through the framework layer; According to the shooting request, the first face image is captured.

10. The apparatus according to any one of claims 6-8, characterized in that, The image sensor is specifically used for: The second face image is sent to the frame layer so that the frame layer can perform image processing on the second face image to obtain a third face image, and then send the third face image to the camera application for display.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 5.

Images