Image shooting method and device and electronic equipment

Abstract

The invention discloses an image shooting method and device and electronic equipment, and belongs to the technical field of shooting. The image shooting method is applied to the electronic equipment, an operating system of the electronic equipment comprises an application layer, a framework layer and a hardware abstraction layer, the application layer is provided with a camera application, and the hardware abstraction layer is provided with an information transmission module, an image sensor and a processing module. The image shooting method comprises the steps that an image sensor sends a shot first face image to a face detection module, and the face detection module is arranged on an application layer or a frame layer; the face detection module performs face detection on the first face image to obtain face detection information, and sends the face detection information to the processing module through the information transmission module; the processing module generates a face image shooting instruction according to the face detection information and sends the face image shooting instruction to the image sensor; and the image sensor shoots a second face image according to the face image shooting instruction, and sends the second face image to the camera application for display.

Description

Technical Field

[0001] The present application belongs to the field of photographing technology, and specifically relates to an image photographing method, device and electronic equipment. Background Art

[0002] With the development of computer technology, people have higher and higher requirements for portrait photography, especially for taking face images. Taking face images that meet user needs often depends on face coordinate information, which can be obtained through face detection algorithms. Therefore, the accuracy and real-time performance of face detection results will directly affect the processing effect of face images.

[0003] In order to meet the real-time needs of users, the current main approach is to couple the face detection algorithm to the hardware abstraction layer of the electronic device. Since the hardware abstraction layer will be updated with changes in versions or manufacturers, the face detection algorithm needs to be recoupled to the hardware abstraction layer, resulting in high maintenance costs for the hardware abstraction layer. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image capturing method, device and electronic device, which can solve the problem that the related technology cannot take into account both real-time performance and maintenance cost when capturing facial images.

[0005] In a first aspect, an embodiment of the present application provides an image capturing method, which is applied to an electronic device, wherein the electronic device includes an operating system, the operating system includes an application layer, a framework layer, and a hardware abstraction layer, the application layer is provided with a camera application, and the hardware abstraction layer is provided with an information transmission module, an image sensor, and a processing module;

[0006] The method includes:

[0007] The image sensor sends the captured first face image to a face detection module, and the face detection module is arranged at an application layer or a framework layer;

[0008] The face detection module performs face detection on the first face image to obtain face detection information, and sends the face detection information to the processing module through the information transmission module;

[0009] The processing module generates a face image shooting instruction according to the face detection information, and sends the face image shooting instruction to the image sensor;

[0010] The image sensor captures a second face image according to the face image capturing instruction, and sends the second face image to the camera application for display.

[0011] In a second aspect, an embodiment of the present application provides an image capturing device, which is applied to an electronic device, and the device includes: an image sensor, a face detection module, an information transmission module, and a processing module;

[0012] An image sensor, arranged in a hardware abstraction layer of the electronic device, for capturing a first face image and sending the first face image to a face detection module;

[0013] A face detection module, arranged in the application layer or framework layer of the electronic device, is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the information transmission module;

[0014] An information transmission module, arranged in the hardware abstraction layer, for sending face detection information to a processing module, the processing module being arranged in the hardware abstraction layer;

[0015] A processing module, used to generate a face image shooting instruction according to the face detection information, and send the face image shooting instruction to the image sensor;

[0016] The image sensor is also used to capture a second face image according to the face image capturing instruction, and send the second face image to a camera application of the electronic device for display, where the camera application is set at the application layer.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0021] In an embodiment of the present application, the electronic device includes an operating system, and the operating system includes an application layer, a framework layer, and a hardware abstraction layer. The application layer is provided with a camera application, and the hardware abstraction layer is provided with an information transmission module, an image sensor, and a processing module. The face detection module is provided in the application layer or the framework layer other than the hardware abstraction layer, thereby realizing the decoupling of the face detection module from the hardware abstraction layer and reducing the maintenance cost of the hardware abstraction layer. By providing the information transmission module in the hardware abstraction layer, the face detection information obtained by the face detection module can be directly transmitted to the processing module in real time through the information transmission module in the hardware abstraction layer, and there is no need to execute according to the request queue maintained by the traditional hardware abstraction layer, thereby reducing the feedback time of the face detection information and ensuring real-time performance. In this way, the solution provided by the embodiment of the present application can take into account both the maintenance cost of the hardware abstraction layer and the feedback time of the face detection information. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of a request queue provided for related technologies;

[0023] Figure 2a A schematic diagram of a software architecture layer of an electronic device provided in an embodiment of the present application;

[0024] Figure 2b A schematic diagram of decoupling a face detection algorithm from a hardware abstraction layer provided in an embodiment of the present application;

[0025] Figure 3 A flowchart of an image capturing method provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a processing process of an information transmission module provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a software architecture layer of another electronic device provided in an embodiment of the present application;

[0028] Figure 6 A flowchart of another image capturing method provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the structure of an image capturing device provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0033] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0034] When shooting face images, users have higher and higher requirements for face images. For example, users hope that the exposure and focus of the face images they shoot can be adjusted in real time according to the shooting scene, etc. Shooting face images that meet user requirements often depends on face coordinate information, which can be obtained through face detection algorithms.

[0035] In order to meet the real-time needs of users, the current approach is to couple the face detection algorithm to the hardware abstraction layer of the electronic device. However, due to version or manufacturer updates, the hardware abstraction layer needs to be constantly upgraded, which means that the face detection algorithm needs to be recoupled to the hardware abstraction layer, increasing the maintenance cost of the hardware abstraction layer.

[0036] There are also some solutions that decouple the face detection algorithm from the hardware abstraction layer, for example, coupling the face detection algorithm to other layers except the hardware abstraction layer. Since the frame rate of the camera is relatively fast, usually 30fps, 15fps or 60fps, in order to ensure the smoothness of image display, the hardware abstraction layer needs to maintain a set of request queues to ensure that the images output by multiple cameras are processed in parallel in a pipeline, so that the camera application display screen can be displayed smoothly at the same frame rate as the camera frame.

[0037] Normally, the queue depth of the request queue is around 7. Figure 1 For example, the depth of request queue 100 is 7, that is, request queue 100 contains 7 requests, and requestN represents the Nth frame image output by the camera. Figure 1In the request queue 100 shown, only when all 7 frames of images in the request queue 100 are processed, can the face detection information of the first frame of image be fed back to the subsequent processing module, and then presented on the eighth frame of image. That is, only when all requests 1-7 are processed, can the face detection information of request 1 be fed back to the next module, so that the camera can shoot the image corresponding to request 8 according to the face detection information of request 1. That is, the face detection information of request 1 acts on request 8, the face detection information of request 2 acts on request 9, and so on. Each face detection information needs to be delayed by 7 frames before it can be used, which affects the real-time exposure and focus of the camera.

[0038] The image output from the request queue 100 may be processed in sequence by the RAW processing module 101 , the RAW2YUV 102 , the YUV processing module 103 and the YUV2JPG 104 , and finally obtain an image in the JPG format and output it to the user.

[0039] To this end, the embodiments of the present application provide an image capturing method, device, and electronic device to solve the problem that related technologies cannot take into account both real-time performance and maintenance costs when capturing facial images.

[0040] The image capture method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablets, and laptop computers, and can be applied to scenarios that require real-time exposure and focusing of captured facial images. Of course, it can also be applied to scenarios such as human eye detection, object detection, and pet detection that require both real-time performance and maintenance costs.

[0041] An electronic device may include an operating system, such as Figure 2a As shown, the operating system 201 may include but is not limited to the Android system, the Linux system, and the Windows system. Taking the Android system as an example, the operating system 201 may include an application layer 2010, a framework layer 2011, and a hardware abstraction layer 2012. In actual application, the operating system 201 may also include other layers, such as a kernel layer, a hardware layer, etc. The layers communicate with each other through software interfaces.

[0042] The application layer 2010 may include a series of application packages, such as camera, gallery, calendar, call, etc.

[0043] The framework layer 2011 provides an application programming interface and a programming framework for the application programs of the application layer 2010 .

[0044] The hardware abstraction layer 2012 runs in the user space, encapsulates the kernel layer drivers, and provides a calling interface to the upper layer.

[0045] In the embodiment of the present application, the hardware abstraction layer 2012 may be provided with an information transmission module 20121 , an image sensor 20122 and a processing module 20123 .

[0046] The information transmission module 20121 is used to distribute the face detection information to the processing module 20123 in real time within the hardware abstraction layer 2012, bypassing the traditional request queue and having high real-time performance. The face detection information is obtained by the face detection module set in the application layer 2010 or the framework layer 2011.

[0047] The image sensor 20122 is used to capture images. The processing module 20123, also known as the 3A module, is used to determine the exposure time and gain. 3A refers to Auto Focus (AF), Auto Exposure (AE) and Auto White Balance (AWE).

[0048] The embodiment of the present application decouples the face detection module from the hardware abstraction layer 2012 and adds an information transmission module 20121 in the hardware abstraction layer 2012, so as to take into account both the maintenance cost of the hardware abstraction layer and the real-time exposure and focus of the face.

[0049] Figure 2b A schematic diagram of decoupling a face detection algorithm from a hardware abstraction layer is exemplified. Figure 3 Taking the coupling of the face detection algorithm to the framework layer 2011 as an example, the framework layer 2011 can be configured with a face detection module, and the face detection module 20111 can call the face detection algorithm to perform face detection on the face image according to the face detection algorithm. Taking the electronic device using the Android system as an example, illustratively, the face detection module 20111 and the hardware abstraction layer 2012 can communicate through the Android standard interface.

[0050] Exemplarily, the framework layer 2011 can send a configuration request to the hardware abstraction layer 2012 according to the user's configuration requirements to request the hardware abstraction layer 2012 to configure different image streams. For example, the preview stream corresponding to the preview image and the shooting stream of the shot image can be configured. The parameters such as the screen size and display position corresponding to the preview stream and the shooting stream can be different.

[0051] For example, the framework layer 2011 can send a stream acquisition request to the hardware abstraction layer 2012 according to the user's information stream requirements. After receiving the stream acquisition request, the hardware abstraction layer 2012 can obtain the parameters of the stream corresponding to the stream acquisition request and send it to the framework layer 2011, which is then sent to the camera application, and the camera application displays the corresponding image stream according to the corresponding stream parameters. After receiving the stream parameters, the camera application can display the corresponding image stream according to the stream parameters.

[0052] By decoupling the face detection algorithm from the hardware abstraction layer, the repeated maintenance cost of re-porting the face detection algorithm due to changes in the hardware abstraction layer version and manufacturer is avoided.

[0053] The image capturing method, device and electronic device provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0054] Figure 3 A flowchart of an image capturing method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the image capturing method may include the following steps:

[0055] S310: The image sensor sends the captured first face image to a face detection module.

[0056] S320: The face detection module performs face detection on the first face image to obtain face detection information, and sends the face detection information to the processing module through the information transmission module.

[0057] S330: The processing module generates a face image shooting instruction according to the face detection information, and sends the face image shooting instruction to the image sensor.

[0058] S340: The image sensor captures a second facial image according to the facial image capturing instruction, and sends the second facial image to the camera application for display.

[0059] The above steps are described in detail below:

[0060] In S310, the first face image is an original image taken by the image sensor. In the embodiment of the present application, the image sensor can send the taken first face image to the face detection module, and the face detection module performs face detection on the first face image according to the face detection algorithm.

[0061] The face detection module can be set at the application layer or the framework layer. In this way, the face detection module, that is, the decoupling of the face detection algorithm and the hardware abstraction layer is realized. When the hardware abstraction layer changes due to version or manufacturer, the face detection algorithm does not need to be recoupled to the hardware abstraction layer, thereby reducing the maintenance cost and upgrade difficulty of the hardware abstraction layer.

[0062] Taking the Android system as an example, in some embodiments, the face detection module is set at the framework layer and adopts the Android standard interface.

[0063] In S320, the face detection module receives the first face image and calls the face detection algorithm to perform face detection on the first face image to obtain face detection information, which may include, for example, the position and size of the face. The embodiment of the present application does not limit the specific face detection algorithm.

[0064] After the face detection module obtains the face detection information, it can send the face detection information to the information transmission module in the hardware abstraction layer through the corresponding interface, and the information transmission module sends it to the processing module in the hardware abstraction layer, that is, the 3A module. That is, after the face detection module of the embodiment of the present application sends the face detection information to the hardware abstraction layer, it can distribute the face detection information to the 3A module in real time through the information transmission module, bypassing the request queue of the camera application and improving the real-time effect.

[0065] It should be noted that the information transmission module added in the embodiment of the present application will not break the native camera process of Android, that is, the images taken by the image sensor will still be sent to the request queue for execution.

[0066] like Figure 4 As shown, without changing the native camera process, the image output by request1 can be directly sent to the information transmission module 20121 after passing through the face detection module 20111, and sent to request2 by the information transmission module 20121 through the processing module 20123, that is, the face detection information of the face image output by request1 can be directly applied to request2, thereby reducing the delay. Figure 4 Take the face detection module 20111 being set in the framework layer 2011 as an example.

[0067] exist Figure 4 In the example, the face detection information output by the face detection module 20111 can also be applied to the RAW processing module 101, RAW2YUV102, YUV processing module 103 and YUV2JPG to perform face changing, beautification and other processing on the face image according to the face detection information.

[0068] In S330, the face image shooting instruction is used to make the image sensor shoot a face image that meets the user's needs. After receiving the face detection information, the 3A module can generate a face image shooting instruction based on the face detection information. In some embodiments, the face image shooting instruction may include the exposure time and / or gain of the image sensor. The exposure time and gain corresponding to the face image are different depending on the scene.

[0069] Exemplarily, the 3A module can search the exposure table according to the face detection information and the scene to obtain the exposure time corresponding to the face detection information and the scene, and the gain processing process is similar. The exposure table is used to store the relationship between multiple face detection information and scenes and exposure time.

[0070] After the 3A module generates the face image shooting instruction, it can directly send the face image shooting instruction to the image sensor of the hardware abstraction layer, and the image sensor shoots the image according to the face image shooting instruction.

[0071] In S340, after receiving the face image shooting instruction, the image sensor can shoot a second face image according to the face image shooting instruction, that is, the image sensor can shoot the image according to the new exposure time and gain to obtain a second face image whose exposure time and gain meet the user's requirements.

[0072] Through the above scheme, the frame difference between the second face image and the first face image is significantly reduced, thereby improving the real-time effect, so that the face detection information of the face image obtained by the initial shooting can be fed back to the next frame image in time, thereby improving the real-time effect.

[0073] In some embodiments, after the image sensor obtains the second facial image according to the facial image capturing instruction, the second facial image may be sent to a camera application of the application layer for display.

[0074] In some embodiments, the image sensor can also send the second face image to the face detection module, that is, execute the processing process of the first face image, and feed back the face detection information of the second face image to the next frame image, thereby achieving the effect of real-time updating of the captured image according to the face detection information, and meeting the real-time needs of the user.

[0075] In some embodiments, Figure 5As shown, the hardware abstraction layer 2012 is also provided with an image signal processing module (Image Signal Processing, ISP) 20124. The image signal processing module 20124 communicates with the image sensor 20122, the face detection module 20111 and the information transmission module 20121, and the information transmission module 20121 also communicates with the processing module 20123, and the processing module 20123 also communicates with the image sensor 20122.

[0076] The image signal processing module 20124 is mainly used to perform noise reduction and other processing on the original image output by the image sensor 20122 to improve the accuracy of subsequent face detection results, thereby improving the shooting effect of the face image.

[0077] according to Figure 5 The architecture shown, Figure 6 A flowchart of another image capturing method provided in an embodiment of the present application, Figure 6 and Figure 3 The difference is that Figure 3 The S310 in can be refined into Figure 6 S610-S620 in Figure 3 The S320 in can be refined into Figure 6 The S630 in the series.

[0078] S610: The image sensor sends the captured first face image to an image signal processing module.

[0079] In some embodiments, the image sensor may also send the captured first face image to the ISP of the hardware abstraction layer, and the ISP preprocesses the first face image to improve the quality of the first face image. The specific processing process is not limited in the embodiments of the present application.

[0080] S620: The image signal processing module performs image processing on the first face image, and sends the processed first face image to the face detection module.

[0081] In some embodiments, the processed first face image can be an image in YUV format, that is, the embodiment of the present application can directly output the face image in YUV format by the ISP, and send the face image directly to the face detection module for face detection, thereby improving the real-time effect while improving the image quality.

[0082] S630: The face detection module performs face detection on the processed first face image to obtain face detection information. Afterwards, the face detection module can send the face detection information to the processing module through the information transmission module.

[0083] Since the image quality of the processed first face image is improved, when the face detection module is used to perform face detection on the processed first face image, the accuracy of face detection information can be improved.

[0084] In some embodiments, the image capturing method may further include the following steps:

[0085] The information transmission module sends the face detection information to the image signal processing module. That is, the information transmission module can not only send the face detection information to the processing module in real time, but also send it to the image signal processing module in real time.

[0086] Exemplarily, the information transmission module may first send the face detection information to the processing module, and then send the face detection information to the image signal processing module. Alternatively, the face detection information may first be sent to the image signal processing module, and then to the processing module. Alternatively, the face detection information may be sent to the image signal processing module and the processing module at the same time.

[0087] Based on this, in some embodiments, the above S620 may include the following steps:

[0088] The image signal processing module determines the face area of ​​the first face image according to the face detection information sent by the information transmission module;

[0089] The image signal processing module performs image processing on the face area to obtain a processed first face image.

[0090] Exemplarily, after the image signal processing module receives the face detection information sent by the information transmission module, it can determine the face area of ​​the first face image based on the face detection information. The face area is the area containing the entire face. The face area can be represented by a face frame, which can be a rectangular frame or other shapes.

[0091] That is, the image signal processing module may first determine the face region from the first face image according to the face detection information, so as to provide a basis for subsequent image processing.

[0092] After obtaining the face region, the image signal processing module can perform noise reduction and other processing on the face region to obtain a processed first face image, that is, a third face image.

[0093] The embodiment of the present application sends face detection information to the image signal processing module in real time, so that the image signal processing module can perform targeted processing on the first face image according to the face detection information, thereby reducing the computational complexity of the image signal processing module and improving image processing efficiency.

[0094] In some embodiments, before S310, the image capturing method may further include the following steps:

[0095] The camera application sends a shooting request to the hardware abstraction layer through the framework layer;

[0096] The hardware abstraction layer sends the capture request to the image sensor;

[0097] The image sensor captures a first face image according to the capture request.

[0098] In an embodiment of the present application, when the camera application receives the user's shooting input, a shooting request can be sent to the hardware abstraction layer through the framework layer. The shooting input may include but is not limited to the user's touch input to the camera application through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which can be determined according to actual use requirements and is not limited in the embodiment of the present application. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, the above-mentioned shooting input can be: a click input by the user to the camera application.

[0099] After receiving the shooting request, the hardware abstraction layer may notify the image sensor to shoot, thereby obtaining a first face image.

[0100] The embodiment of the present application satisfies the user's shooting needs through interaction between layers while taking into account maintenance costs and real-time performance.

[0101] In some embodiments, the above “sending the second facial image to the camera application for display” may include the following steps:

[0102] The image sensor sends the second face image to the framework layer;

[0103] The framework layer performs image processing on the second facial image to obtain a third facial image, and sends the third facial image to the camera application for display.

[0104] In some embodiments, the electronic device may further process the second face image to obtain an image that meets the user's needs. For example, the second face image may be processed by blurring, beautifying, or performing interesting face swapping, and the processed second face image, i.e., the third face image, may be displayed to the user through a camera application, thereby meeting the user's personalized shooting needs.

[0105] The image capturing process is described below.

[0106] The image sensor 20122 can send the captured facial image to ISP20124. ISP20124 can pre-process the facial image sent by the image sensor 20122 and send the processed facial image to the RAW processing module 101. The RAW processing module 101 continues to process it and then sends it to RAW2YUV102, YUV processing module 103 and YUV2JPG for processing in turn, and finally obtains an image in JPG format and displays it to the user.

[0107] ISP20124 can also send the processed face image to the face detection module 20111, which performs face detection on the processed face image to obtain face detection information. The face detection module 20111 can send the face detection information to the RAW processing module 101 or the YUV processing module 103, which can continue to process the processed face image sent by ISP20124 according to the face detection information, for example, it can perform face replacement, beautification and other processing on the processed face image according to the face detection information.

[0108] The face detection module 20111 can also send the face detection information to the information transmission module 20121. After receiving the face detection information, the information transmission module 20121 can send the face detection information to the processing module 20123. The processing module 20123 generates a face image shooting instruction based on the face detection information, and sends the face image shooting instruction to the image sensor 20122. In this way, the image sensor 20122 can shoot a face image that meets the user's needs according to the face image shooting instruction, thereby reducing delay.

[0109] The embodiment of the present application reduces the maintenance cost of the hardware abstraction layer by decoupling the face detection module from the hardware abstraction layer, and at the same time adds an information transmission module in the hardware abstraction layer, so that the face detection information obtained by the face detection module can be distributed to the 3A module in real time within the hardware abstraction layer through the information transmission module, thereby bypassing the native request queue, reducing latency, and improving real-time effects.

[0110] It should be noted that the image capture method provided in the embodiment of the present application can be executed by an image capture device or a processing module in the image capture device for executing the image capture method. In the embodiment of the present application, the image capture device provided in the embodiment of the present application is described by taking the image capture method executed by the image capture device as an example.

[0111] Figure 7 A schematic diagram of the structure of an image capturing device provided in an embodiment of the present application.

[0112] like Figure 7 As shown, the image capturing device 700 may include: an image sensor 701, a face detection module 702, an information transmission module 703 and a processing module 704;

[0113] An image sensor 701 is provided in a hardware abstraction layer of the electronic device, and is used to capture a first face image and send the first face image to a face detection module;

[0114] The face detection module 702 is provided in the application layer or the framework layer of the electronic device, and is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the information transmission module 703;

[0115] An information transmission module 703, which is arranged in the hardware abstraction layer, is used to send the face detection information to the processing module 704, and the processing module 704 is arranged in the hardware abstraction layer;

[0116] The processing module 704 is used to generate a face image shooting instruction according to the face detection information, and send the face image shooting instruction to the image sensor 701;

[0117] The image sensor 701 is further used to capture a second facial image according to the facial image capturing instruction, and send the second facial image to a camera application of the electronic device for display, where the camera application is arranged at the application layer.

[0118] In an embodiment of the present application, the electronic device includes an operating system, and the operating system includes an application layer, a framework layer, and a hardware abstraction layer. The application layer is provided with a camera application, and the hardware abstraction layer is provided with an information transmission module, an image sensor, and a processing module. The face detection module is provided in the application layer or the framework layer other than the hardware abstraction layer, thereby realizing the decoupling of the face detection module from the hardware abstraction layer and reducing the maintenance cost of the hardware abstraction layer. By providing the information transmission module in the hardware abstraction layer, the face detection information obtained by the face detection module can be directly transmitted to the processing module in real time through the information transmission module in the hardware abstraction layer, and there is no need to execute according to the request queue maintained by the traditional hardware abstraction layer, thereby reducing the feedback time of the face detection information and ensuring real-time performance. In this way, the solution provided by the embodiment of the present application can take into account both the maintenance cost of the hardware abstraction layer and the feedback time of the face detection information.

[0119] In some possible implementations of the embodiments of the present application, the image capturing device 700 may further include:

[0120] An image signal processing module, wherein the image signal processing module is arranged in a hardware abstraction layer;

[0121] The image sensor 701 is further used to send the first face image to the image signal processing module;

[0122] An image signal processing module, used to perform image processing on the first face image and send the processed first face image to the face detection module 702;

[0123] The face detection module 702 is specifically used to perform face detection on the processed first face image to obtain face detection information.

[0124] In some possible implementations of the embodiments of the present application, the information transmission module 703 is further used to send the face detection information to the image signal processing module;

[0125] Image signal processing module, specifically used for:

[0126] Determine a face region of the first face image according to the face detection information sent by the information transmission module;

[0127] Perform image processing on the face region to obtain a processed first face image.

[0128] In some possible implementations of the embodiments of the present application, the image sensor 701 is specifically used for:

[0129] Receive the shooting request sent by the hardware abstraction layer. The shooting request is sent by the camera application to the hardware abstraction layer through the framework layer;

[0130] According to the shooting request, a first face image is shot.

[0131] In some possible implementations of the embodiments of the present application, the image sensor 701 is specifically used for:

[0132] The second facial image is sent to the framework layer, so that the framework layer performs image processing on the second facial image to obtain a third facial image, and the third facial image is sent to the camera application for display.

[0133] The image capture device in the embodiment of the present application can be a device or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a user computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0134] The electronic device in the embodiment of the present application may be a terminal with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0135] The image capture device provided in the embodiment of the present application can achieve Figures 1 to 6 The various processes in the image shooting method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0136] like Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned image capture method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0137] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0138] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0139] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

[0140] The processor 910 includes an application layer, a framework layer and a hardware abstraction layer, wherein the application layer is provided with a camera application, and the hardware abstraction layer is provided with an information transmission module, an image sensor and a processing module. In some embodiments, the framework layer is provided with a face detection module.

[0141] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The structure of the electronic device 900 shown in the figure does not constitute a limitation on the electronic device 900. The electronic device 900 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0142] The image sensor is used to send the captured first face image to the face detection module, and the face detection module is arranged at the application layer or the framework layer;

[0143] The face detection module is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the processing module through the information transmission module;

[0144] The processing module is used to generate a face image shooting instruction according to the face detection information, and send the face image shooting instruction to the image sensor;

[0145] The image sensor is also used to capture a second face image according to the face image capturing instruction, and send the second face image to the camera application for display.

[0146] The embodiment of the present application realizes the decoupling of the face detection module and the hardware abstraction layer, and reduces the maintenance cost of the hardware abstraction layer. And by setting the information transmission module in the hardware abstraction layer, the face detection information obtained by the face detection module can be directly transmitted to the processing module in real time through the information transmission module in the hardware abstraction layer, and there is no need to execute according to the request queue maintained by the traditional hardware abstraction layer, thereby reducing the feedback time of the face detection information and ensuring real-time performance. In this way, the solution provided by the embodiment of the present application can take into account both the maintenance cost of the hardware abstraction layer and the feedback time of the face detection information.

[0147] In some possible implementations of the embodiments of the present application, the hardware abstraction layer is further provided with an image signal processing module;

[0148] The image sensor is specifically used to: send the captured first face image to the image signal processing module;

[0149] The image signal processing module is used to perform image processing on the first face image and send the processed first face image to the face detection module;

[0150] The face detection module is specifically used to perform face detection on the processed first face image to obtain face detection information.

[0151] In some possible implementations of the embodiments of the present application, the information transmission module is further used to send the face detection information to the image signal processing module;

[0152] The image signal processing module is specifically used to: determine the face area of ​​the first face image according to the face detection information sent by the information transmission module; and perform image processing on the face area to obtain a processed first face image.

[0153] In some possible implementations of the embodiments of the present application, the image sensor is specifically used to: receive a shooting request sent by the camera application through the framework layer before sending the captured first face image to the face detection module, the shooting request is sent by the camera application to the hardware abstraction layer through the framework layer; and capture the first face image according to the shooting request.

[0154] In some possible implementations of the embodiments of the present application, the image sensor is specifically used to: send the second facial image to the framework layer, so that the framework layer performs image processing on the second facial image to obtain a third facial image, and send the third facial image to the camera application for display.

[0155] The embodiment of the present application reduces the maintenance cost of the hardware abstraction layer by decoupling the face detection module from the hardware abstraction layer, and at the same time adds an information transmission module in the hardware abstraction layer, so that the face detection information obtained by the face detection module can be distributed to the 3A module in real time within the hardware abstraction layer through the information transmission module, thereby bypassing the native request queue, reducing latency, and improving real-time effects.

[0156] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0157] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0158] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0159] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image capture method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0160] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0161] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image capture method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0162] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0163] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image capture method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0164] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0166] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image capturing method, applied to an electronic device, characterized in that: The electronic device includes an operating system, the operating system includes an application layer, a framework layer and a hardware abstraction layer, the application layer is provided with a camera application, and the hardware abstraction layer is provided with an information transmission module, an image sensor and a processing module; The method comprises: The image sensor sends the captured first face image to a face detection module, where the face detection module is arranged in the application layer or the framework layer; The face detection module performs face detection on the first face image to obtain face detection information, and sends the face detection information to the processing module through the information transmission module; The processing module generates a face image shooting instruction according to the face detection information, and sends the face image shooting instruction to the image sensor; The image sensor captures a second facial image according to the facial image capturing instruction, and sends the second facial image to the camera application for display.

2. The method according to claim 1, characterized in that The hardware abstraction layer is also provided with an image signal processing module; The image sensor sends the captured first face image to the face detection module, including: The image sensor sends the captured first face image to the image signal processing module; The image signal processing module performs image processing on the first face image, and sends the processed first face image to the face detection module; The face detection module performs face detection on the first face image to obtain face detection information, including: The face detection module performs face detection on the processed first face image to obtain face detection information.

3. The method according to claim 2, characterized in that The method further comprises: The information transmission module sends the face detection information to the image signal processing module; The image signal processing module performs image processing on the first face image, including: The image signal processing module determines the face area of ​​the first face image according to the face detection information sent by the information transmission module; The image signal processing module performs image processing on the face area to obtain a processed first face image.

4. The method according to any one of claims 1 to 3, characterized in that: Before the image sensor sends the captured first face image to the face detection module, the method further includes: The camera application sends a shooting request to the hardware abstraction layer through the framework layer; The hardware abstraction layer sends the shooting request to the image sensor; The image sensor captures the first face image according to the capture request.

5. The method according to any one of claims 1 to 3, characterized in that: The sending the second facial image to the camera application for display includes: The image sensor sends the second facial image to the framework layer; The framework layer performs image processing on the second facial image to obtain a third facial image, and sends the third facial image to the camera application for display.

6. An image capturing device, applied to an electronic device, characterized in that: The device comprises: an image sensor, a face detection module, an information transmission module and a processing module; The image sensor is arranged in the hardware abstraction layer of the electronic device, and is used for capturing a first face image and sending the first face image to the face detection module; The face detection module is arranged in the application layer or the framework layer of the electronic device, and is used to perform face detection on the first face image, obtain face detection information, and send the face detection information to the information transmission module; The information transmission module is arranged in the hardware abstraction layer, and is used to send the face detection information to the processing module, and the processing module is arranged in the hardware abstraction layer; The processing module is used to generate a face image shooting instruction according to the face detection information, and send the face image shooting instruction to the image sensor; The image sensor is further used to capture a second facial image according to the facial image capturing instruction, and send the second facial image to a camera application of the electronic device for display, wherein the camera application is arranged in the application layer.

7. The device according to claim 6, characterized in that The device further comprises: an image signal processing module, the image signal processing module being arranged in the hardware abstraction layer; The image sensor is further used to send the first face image to the image signal processing module; The image signal processing module is used to perform image processing on the first face image and send the processed first face image to the face detection module; The face detection module is specifically used to perform face detection on the processed first face image to obtain face detection information.

8. The device according to claim 7, characterized in that The information transmission module is further used to send the face detection information to the image signal processing module; The image signal processing module is specifically used for: determining a face region of the first face image according to the face detection information sent by the information transmission module; Perform image processing on the face region to obtain a processed first face image.

9. The device according to any one of claims 6 to 8, characterized in that: The image sensor is specifically used for: receiving a shooting request sent by the hardware abstraction layer, where the shooting request is sent by the camera application to the hardware abstraction layer through the framework layer; According to the photographing request, the first face image is photographed.

10. The device according to any one of claims 6 to 8, characterized in that: The image sensor is specifically used for: The second facial image is sent to the framework layer, so that the framework layer performs image processing on the second facial image to obtain a third facial image, and the third facial image is sent to the camera application for display.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images