Image processing method, device, chip, electronic device and medium

By separating and cropping streams in image processing, the problem of failing to meet the frame rate caused by long image processing time is solved, efficient image display of video calls is realized, and user experience is improved.

CN117714833BActive Publication Date: 2024-11-26HONOR DEVICE CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202310572669.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-26
Estimated Expiration
2043-05-19

AI Technical Summary

Technical Problem

The prior art has the problem that the image processing time is long in image processing, resulting in the failure of the picture frame number, especially in video call scenarios, which affects the user experience.

Method used

By dividing the data stream into two channels, one for image detection and the other for image cropping, and performing asynchronously between the image acquisition module, the algorithm detection module and the image cropping module, the detection result is synchronously provided to the image cropping module. Implement parallel processing.

Benefits of technology

It shortens the single-frame image processing time, improves the number of picture frames, ensures the real-time video calls and the immediacy of image display, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117714833B_ABST
    Figure CN117714833B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide an image processing method, device, chip, electronic device and medium, the method comprising: storing a first image captured by a camera into a first storage space; obtaining the first image by executing a first operation of obtaining the image from the first storage space, and obtaining the first image by executing a second operation of obtaining the image from the first storage space, the two operations being executed asynchronously; performing image detection on the first image obtained by executing the first operation, generating a detection result of the first image and storing it in a second storage space; after obtaining the first image by executing the second operation, executing a third operation of obtaining the detection result from the second storage space; if the detection result is obtained by executing the third operation, cropping the first image obtained by executing the second operation according to the detection result to obtain an image to be displayed, and after the detection result is obtained by executing the third operation, the second storage space does not include the detection result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of electronic equipment, and in particular to an image processing method, device, chip, electronic equipment and medium. Background Art

[0002] In order to achieve the desired image display effect, the electronic device can process the image captured by the camera and display the processed image. Therefore, it is necessary to provide an image processing method. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, chip, electronic device and medium, which can support electronic devices to achieve the desired image display effect.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising: storing a first image into a first storage space, the first image being any image taken by a camera; obtaining the first image by executing a first operation of acquiring the image from the first storage space, and obtaining the first image by executing a second operation of acquiring the image from the first storage space, wherein the first operation and the second operation are executed asynchronously; performing image detection on the first image obtained by executing the first operation, generating a detection result of the first image, and storing the detection result of the first image into a second storage space; after obtaining the first image by executing the second operation, executing a third operation of acquiring the detection result from the second storage space; if the detection result is acquired by executing the third operation, cropping the first image obtained by executing the second operation according to the acquired detection result to obtain an image to be displayed, wherein after the detection result is acquired by executing the third operation, the second storage space does not include the acquired detection result.

[0005] In an embodiment of the present application, the data stream is divided into two paths, one path is sent for detection, and the other path is sent for cropping, and the result of the algorithm operation is synchronously provided to the module for image cropping. Based on the parallel execution of the two-path processing, not only can the expected image display effect be achieved, but also the single-frame image processing time can be shortened, so that the frame rate of the picture can be achieved.

[0006] In one embodiment, the image processing method further includes: if no detection result is obtained by performing the third operation, using the first image as the image to be displayed. In this way, the immediacy and non-interruption of image display can be supported.

[0007] In one embodiment, the image processing method further includes: if no detection result is obtained by performing the third operation, determining whether a cropping process has been performed; if it is determined that a cropping process has been performed, cropping the first image obtained by performing the second operation according to the detection result used in the last cropping process to obtain an image to be displayed. In this way, consistency of image display effects can be supported.

[0008] In one embodiment, the image processing method further includes: upon receiving an image acquisition request, executing a fourth operation of acquiring an image from a third storage space at a third frequency until a frame of image is acquired, wherein the third storage space is used to store images captured by the camera performing an image capture operation at a first frequency; executing the first operation at a second frequency; executing the second operation at a second frequency; wherein the second frequency is greater than the first frequency, and the third frequency is greater than the first frequency; storing the first image in the first storage space includes: storing the first image acquired by executing the fourth operation in the first storage space. In this way, the real-time display of the image can be achieved, and the number of frames of the picture can be guaranteed.

[0009] In one embodiment, the first storage space is an image buffer queue; the second storage space is a detection result buffer queue; and the third operation is used to obtain the earliest detection result stored in the detection result buffer queue when there is more than one detection result in the detection result buffer queue. In this way, accurate execution of image cropping can be achieved.

[0010] In one embodiment, the image processing method further includes: performing stream distribution processing according to a set frame rate so that the camera performs image capturing operation at a first frequency; wherein the first frequency is obtained according to the set frame rate.

[0011] By performing stream processing according to the set frame rate before image acquisition, the camera can be controlled to capture images at the expected frequency, thereby providing support for achieving the target frame rate.

[0012] In one embodiment, the image processing method further includes: in response to any one of a camera preview event, a camera recording event, and a video call event, executing an operation of acquiring an image captured by the camera; wherein the video call event includes at least one of an event of actively initiating a video call and an event of answering a video call initiated by the other party.

[0013] By asynchronously executing image detection and image cropping in camera preview, camera recording, and video call scenarios, an image display effect in which a person is displayed in the center can be achieved, which provides a better user experience.

[0014] In a second aspect, an embodiment of the present application provides an image processing device, comprising: an image acquisition module, used to store a first image into a first storage space, the first image being any image taken by a camera; an algorithm detection module, used to obtain the first image by executing a first operation of obtaining the image from the first storage space; performing image detection on the first image obtained by executing the first operation, generating a detection result of the first image, and storing the detection result of the first image into a second storage space; an image cropping module, used to obtain the first image by executing a second operation of obtaining the image from the first storage space; after obtaining the first image by executing the second operation, executing a third operation of obtaining the detection result from the second storage space; if the detection result is obtained by executing the third operation, the first image obtained by executing the second operation is cropped according to the obtained detection result to obtain an image to be displayed; wherein the first operation and the second operation are executed asynchronously; wherein, after obtaining the detection result by executing the third operation, the second storage space does not include the obtained detection result.

[0015] In a third aspect, an embodiment of the present application provides an electronic chip, comprising: a processor, which is used to execute computer program instructions stored in a memory, wherein when the computer program instructions are executed by the processor, the electronic chip is triggered to execute any method as in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing computer program instructions, a processor for executing the computer program instructions, and a communication device, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute a method as described in any one of the first aspects.

[0017] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes any method in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes any method in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, DD231246I02 can be obtained based on these drawings without creative work.

[0020] Other attached pictures.

[0021] Figure 1 A schematic diagram of the structure of an electronic device provided by one embodiment of the present application;

[0022] Figure 2 A schematic diagram of a software architecture for single-threaded data stream processing;

[0023] Figure 3 A module interaction timing diagram for image processing provided by one embodiment of the present application;

[0024] Figure 4 Another module interaction timing diagram for image processing is provided for one embodiment of the present application;

[0025] Figure 5 A flowchart of an image processing method provided for one embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0027] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0028] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0029] It should be understood that the term "at least one" used in this article refers to one or more, and "plurality" refers to two or more. The term "and / or" used in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. Among them, A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple.

[0030] It should be understood that, although the terms first, second, etc. may be used to describe the set thresholds in the embodiments of the present application, these set thresholds should not be limited to these terms. These terms are only used to distinguish the set thresholds from each other. For example, without departing from the scope of the embodiments of the present application, the first set threshold may also be referred to as the second set threshold, and similarly, the second set threshold may also be referred to as the first set threshold.

[0031] The image processing method provided in any embodiment of the present application can be applied to Figure 1 In the electronic device 100 shown. Figure 1 A schematic structural diagram of an electronic device 100 is shown.

[0032] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0033] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0034] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0035] In some embodiments, the processor 110 may be a system on chip (SOC), which may include a central processing unit (CPU) and may further include other types of processors. In some embodiments, the processor 110 may be a PWM control chip.

[0036] The processor 110 may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the technical solution program, etc. In addition, the processor 110 may have the function of operating one or more software programs, which may be stored in a storage medium.

[0037] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0038] In some embodiments, the memory of the electronic device 100 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or may be any computer-readable medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0039] In some embodiments, the processor 110 and the memory may be combined into a processing device, or may be independent components, and the processor 110 may be used to execute program codes stored in the memory. In specific implementation, the memory may also be integrated into the processor 110, or may be independent of the processor 110.

[0040] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0041] The USB interface 130 is an interface that complies with USB standard specifications, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.

[0042] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0043] The charging management module 140 is used to receive charging input from a charger. The power management module 141 is used to connect the battery 142 , the charging management module 140 and the processor 110 .

[0044] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0045] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0046] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0047] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0048] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0049] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0050] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0051] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0052] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0053] The ISP is used to process data fed back by the camera 193. The camera 193 is used to capture static images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element.

[0054] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0055] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0056] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0057] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0058] The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor can be set on the display screen 194. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. When a touch operation is applied to the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor.

[0059] The gyroscope sensor can be used to determine the motion posture of the electronic device 100. The air pressure sensor is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor to assist positioning and navigation. The magnetic sensor includes a Hall sensor. The acceleration sensor can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). Distance sensor, used to measure distance. The electronic device 100 can measure the distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use a distance sensor to measure the distance to achieve fast focus. The proximity light sensor may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor is used to sense the brightness of ambient light. The fingerprint sensor is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc. The temperature sensor is used to detect temperature.

[0060] A touch sensor is also called a "touch control device". The touch sensor can be arranged on the display screen 194. The touch sensor and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0061] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button or a touch button. The motor 191 may generate a vibration prompt. The motor 191 may be used for an incoming call vibration prompt or for touch vibration feedback. The indicator 192 may be an indicator light, which may be used to indicate the charging status, the change in power, or may be used to indicate messages, missed calls, notifications, etc.

[0062] The SIM card interface 195 is used to connect a SIM card. The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0063] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.

[0064] To achieve the desired image display effect, the electronic device can process the image captured by the camera and display the processed image. The electronic device can perform algorithm detection on the image captured by the camera, and based on the algorithm detection result, process the image captured by the camera according to set rules to obtain the processed image for image display.

[0065] In a feasible implementation, when the user is within the image acquisition range of the camera, the electronic device can perform face detection on the image captured by the camera according to the face detection algorithm, and obtain the algorithm detection result of the face position (i.e., the position of the image of the user's face in the captured image). Based on the detected face position, the electronic device can add special effects (such as mosaic, etc.) to the face image at the corresponding position to obtain an image with special effects added to the user's face. The electronic device can then display the processed image, thereby having the desired image display effect.

[0066] In another feasible implementation, there is a user within the image acquisition range of the camera, and the electronic device can perform face detection on the image captured by the camera according to the face detection algorithm to obtain the algorithm detection result of the face position. The electronic device can crop the image captured by the camera based on the detected face position so that the image of the user's face is located in a relatively centered position in the cropped image. The electronic device can then display the processed image to achieve the desired image display effect.

[0067] In another feasible implementation, there are multiple users within the image acquisition range of the camera, and the electronic device can perform face detection and distance detection on the image captured by the camera according to the face detection algorithm and the distance detection algorithm, and obtain the algorithm detection result of the face position and face distance (i.e., the distance between the user and the camera). The electronic device can crop the image captured by the camera based on the detected face position and face distance, so that the face image of the main user is located in a relatively centered position in the cropped image. Among them, within the same azimuth range, the distance between the main user and the camera is smaller than the distance between other users and the camera. The electronic device can then display the processed image, thereby having the desired image display effect.

[0068] In a related technology, electronic devices can use a single-threaded image processing method to process data streams, and perform algorithm detection and rule setting on a frame of images captured by a camera in sequence, that is, perform algorithm detection and rule setting in series. Since the algorithm detection process is time-consuming, and the rule setting process is also time-consuming, the image processing time is long, which will result in substandard frame count or screen distortion.

[0069] In a feasible implementation of the above-mentioned related technology, the software architecture for single-threaded data flow processing can be as follows: Figure 2 As shown, the electronic device may include an application layer 201 , an application framework (FWK, Framework) layer 202 , a hardware abstraction (Hal, Hardware abstraction layer) layer 203 and a camera 204 .

[0070] The application layer 201 includes a first application 2011 ; the application framework layer 202 includes a native camera service (cameraserver) module 2021 ; the hardware abstraction layer 203 includes an image acquisition module 2031 , an image decoding module 2032 , an algorithm detection module 2033 , and an image cropping module 2034 .

[0071] The image acquisition module 2031 may be used to acquire compressed raw data streams through the camera 204. For example, in one implementation, the acquired data may be 1080P data, and 1080P may represent a resolution.

[0072] The image decoding module 2032 may be used to perform image decoding on the compressed data stream acquired by the image acquisition module 2031. For example, in one implementation, image data in yuyv format may be generated after decoding.

[0073] The algorithm detection module 2033 can be used to perform algorithm detection on the decoded image to obtain algorithm detection results, such as detection algorithms such as distance detection and face recognition.

[0074] The image cropping module 2034 can be used to perform image cropping based on the algorithm detection results. For example, in one implementation, after image cropping, the image of the user's face can be located in a relatively centered position in the cropped image. After the image cropping is completed, the cropped image is further passed to the upper-level first application 2011 via the camera service module 2021 for display.

[0075] Since face detection in multi-person scenarios takes more than 20ms and image cropping takes about 22ms, Figure 2 As shown, when the data stream is processed by a single thread, the image processing time is long, and the number of frames displayed by the electronic device per second is about ten frames, which is not up to standard. For example, the number of frames does not reach the required 30fps (Frame Per Second).

[0076] To increase the frame rate, please refer to Figure 3 , an embodiment of the present application provides a module interaction timing diagram for image processing. Figure 3 The illustrated embodiment may be applicable to a video call scenario.

[0077] like Figure 3 As shown, the electronic device may include an application layer 301, an application framework layer 302, a hardware abstraction layer 303 and a camera 304. The application layer 301 includes a video call application 3011; the application framework layer 302 includes a camera service module 3021; ​​the hardware abstraction layer 303 includes an image acquisition module 3031, an algorithm detection module 3032, and an image cropping module 3033.

[0078] Please refer to Figure 3 , the image processing flow can include the following:

[0079] Step 3.1, in response to an event of actively initiating a video call or an event of answering a video call initiated by the other party, the video call application 3011 sends streaming information (such as a frame rate) to the camera service module 3021. In one embodiment, the frame rate may be 30 fps.

[0080] In one embodiment, the streaming information may include frame rate and resolution.

[0081] In one embodiment, the video call application 3011 may call a configstream function to send stream configuration information to the camera service module 3021 .

[0082] In one embodiment, the event of actively initiating a video call may be: the user issues an instruction to actively initiate a video call by triggering a video initiation control in the video call application 3011. The other party of the video call waits to answer the video call request initiated by the user.

[0083] In one embodiment, the event of answering a video call initiated by the other party may be: when the other party user sends a video call request, the user triggers the answer video call control in the video call application 3011 to issue an instruction to answer the video call.

[0084] Step 3.2: The camera service module 3021 sends the distribution information to the image acquisition module 3031.

[0085] Step 3.3: the image acquisition module 3031 performs flow distribution processing according to the flow distribution information, so that the camera 304 performs an image capturing operation at a first frequency.

[0086] In one embodiment, the image acquisition module 3031 may include a flow allocation submodule, an image acquisition submodule and an image decoding submodule. The flow allocation submodule is used to perform flow allocation processing according to flow allocation information; the image acquisition submodule is used to acquire compressed images generated by camera shooting; and the image decoding submodule is used to decode compressed images.

[0087] In one embodiment of the present application, the flow distribution information may include a resolution, and the resolution of the image captured by the camera may be set according to the resolution sent by the upper layer application.

[0088] In one embodiment of the present application, the streaming information includes a frame rate, and the frame rate sent by the upper layer application can affect the frequency of capturing images by the camera 304. In a feasible implementation, if the frame rate is 30fps, the first frequency is executed once every 33ms, so that the camera 304 can perform an image capturing operation once every 33ms, that is, the camera 304 can capture and generate a frame of image every 33ms.

[0089] In one embodiment, the camera 304 may store each frame of image generated by shooting into a corresponding buffer queue in real time, and the image acquisition module 3031 may obtain the image generated by shooting by the camera 304 from the buffer queue.

[0090] Through stream distribution processing, the camera can be controlled to capture images at the expected frequency, providing support for achieving the required frame rate.

[0091] Step 3.4, after the image acquisition module 3031 completes the stream distribution processing, the video call application 3011 sends a repeat request to the camera service module 3021.

[0092] In one embodiment, the video call application 3011 may call a RepeatRequest function to send a repeat request to the camera service module 3021 .

[0093] Step 3.5: the camera service module 3021 sends an image acquisition request (Request) to the image acquisition module 3031 according to the repeated request.

[0094] In one embodiment, the image acquisition module 3031 may acquire a frame of image generated by the camera 304 according to the received image acquisition request, and put the acquired frame of image into the image buffer queue, thereby completing the processing of the image acquisition request.

[0095] After the image acquisition module 3031 completes processing of the received image acquisition request, it can report a processing completion notification to the camera service module 3021; ​​after receiving the reported processing completion notification, the camera service module 3021 can issue an image acquisition request again according to the above-mentioned repeated request, so that the image acquisition module 3031 can capture the next frame of image generated by the camera 304 accordingly.

[0096] In this way, the camera service module 3021 can successively send image acquisition requests to the image acquisition module 3031, so that the image acquisition module 3031 can successively acquire each frame of images taken by the camera 304, and then the algorithm detection module 3032 cooperates with the image cropping module 3033 to process each frame of the acquired image, and send the processed frame of images to the upper-level application for display until the video call ends.

[0097] Step 3.6: the image acquisition module 3031 acquires a frame of image taken by the camera 304 according to the image acquisition request.

[0098] In one embodiment, the camera 304 can store each frame of the captured image in the target buffer queue of the corresponding camera, and the image acquisition module 3031 can obtain images from the target buffer queue at a third frequency (for example, once every 10ms) according to the received image acquisition request until a frame of image is obtained. The images in the target buffer queue follow the first-in-first-out principle.

[0099] If there is no image in the target buffer queue when the image acquisition module 3031 acquires the image from the target buffer queue, the image acquisition module 3031 waits and acquires the image from the target buffer queue again at the next moment corresponding to the third frequency until a frame of image is acquired.

[0100] Step 3.7, the image acquisition module 3031 performs image decoding processing on the acquired image, and puts the decoded image into the image buffer queue. The images in the image buffer queue can be arranged in chronological order, so that when an image is acquired from the image buffer queue, the image that enters the queue relatively earliest is acquired (i.e., the images in the image buffer queue follow the first-in-first-out principle).

[0101] In one embodiment, the image acquisition module 3031 can obtain images from the camera 304 according to a native capture request thread (capturerequest).

[0102] In one embodiment, the acquired image may be an image in h264 format, which is a video encoding format.

[0103] In one embodiment of the present application, the decoded image can be copied and sent to the image buffer queue so that the algorithm detection module 3032 and the image cropping module 3033 can obtain the image from the image buffer queue respectively. After both the algorithm detection module 3032 and the image cropping module 3033 obtain the image, the image is removed from the queue.

[0104] After the image acquisition module 3031 completes image acquisition according to the currently received image acquisition request and stores the acquired image in the image buffer queue, it can report a processing completion notification to the camera service module 3021 so that the camera service module 3021 can issue an image acquisition request again. The image acquisition module 3031 then acquires the next frame of image according to the image acquisition request received again.

[0105] In addition, after the image acquisition module 3031 stores the acquired image in the image buffer queue, the algorithm detection module 3032 cooperates with the image cropping module 3033 to obtain the image from the image buffer queue and perform image processing to obtain a frame of image to be displayed. Based on the asynchronous execution of the image acquisition process and the image processing process, the overall processing time of a frame of image can be shortened, providing support for meeting the frame number standard.

[0106] Step 3.8a1, the algorithm detection module 3032 executes the operation of obtaining an image from the image buffer queue at a second frequency, and the second frequency is greater than the first frequency.

[0107] The algorithm detection module 3032 executes each image acquisition operation at a second frequency. If the current operation fails (for example, there is no image to be acquired in the image buffer queue), it waits and performs the next image acquisition operation when the time for the next operation is reached. In this way, each frame of images acquired by the image acquisition module 3031 and placed in the image buffer queue are acquired in sequence.

[0108] Step 3.8b1, the image cropping module 3033 executes the operation of acquiring an image from the image buffer queue at a second frequency.

[0109] The algorithm detection module 3032 and the image cropping module 3033 asynchronously obtain images from the image buffer queue. This embodiment does not limit the execution order of step 3.8a1 and step 3.8b1.

[0110] In one embodiment, the first frequency is executed once every 33ms, and the second frequency is executed once every 10ms. Based on this technical design, during the period when the image acquisition module 3031 acquires two adjacent frames of images, the algorithm detection module 3032 can execute up to three times to obtain images from the image buffer queue, so that the algorithm detection module 3032 can usually obtain images in time. Similarly, this technical design can ensure that the image cropping module 3033 can usually obtain images in time.

[0111] When the algorithm detection module 3032 or the image cropping module 3033 has not yet completed the acquisition of the same frame image in the image buffer queue, the frame image will not be queued temporarily (i.e., it will be in the queue), and after the algorithm detection module 3032 and the image cropping module 3033 have completed the acquisition, the frame image will be queued (i.e., it will not be in the queue).

[0112] Step 3.8a2, after acquiring an image, the algorithm detection module 3032 performs algorithm detection processing on the acquired image to obtain a detection result, and puts the obtained detection result into a detection result buffer queue. The detection results in the detection result buffer queue can be arranged in chronological order, so that when the detection result is obtained from the detection result buffer queue, the detection result obtained is the detection result that enters the queue relatively earliest in the queue.

[0113] For any detection result in the detection result buffer queue, when the image cropping module 3033 has not yet completed the acquisition of the detection result, the detection result will not be queued temporarily (that is, it will be in the queue), and after the image cropping module 3033 has completed the acquisition, the detection result will be queued (that is, it will not be in the queue).

[0114] Step 3.8b2, after the image cropping module 3033 acquires an image, it executes an operation of acquiring a detection result from the detection result buffer queue. If the operation is successful (i.e., a detection result is acquired), the image is cropped according to the detection result, and the cropped image is sent to the camera service module 3021. If the operation fails (i.e., no detection result is acquired), the acquired image is sent to the camera service module 3021 (i.e., no image cropping is performed, and the original image is directly returned for display). Alternatively, if image cropping has been performed, image cropping is performed according to the detection result used in the last image cropping process, and the cropped image is sent to the camera service module 3021.

[0115] In one embodiment of the present application, in order to improve image processing efficiency, the image cropping module 3033 can execute the operation of obtaining a detection result from the detection result buffer queue in real time after acquiring an image.

[0116] In one embodiment, for the first frame image, since there is no previous frame image of the first frame image, image cropping processing has not been performed. Therefore, if the image cropping module 3033 does not obtain a detection result after obtaining the first frame image, the first frame image may not be cropped, and the obtained first frame image may be directly sent to the camera service module 3021 for display by the upper-level application.

[0117] In one embodiment, for the first frame image, if the algorithm detection module 3032 fails to store the detection result of the first frame image into the detection result buffer queue in time due to factors such as slow algorithm detection operation, then after the image cropping module 3033 obtains the first frame image, if it does not obtain a detection result from the detection result buffer queue, the image cropping module 3033 can send the first frame image directly to the upper-layer application for display.

[0118] Step 3.9, the camera service module 3021 sends the received image to the video call application 3011 for image display.

[0119] In one embodiment, after receiving the image, the camera service module 3021 can display the image. In this way, the image processing and display of a frame of image collected by the image acquisition module 3031 are completed. Based on each image acquisition request successively issued by the camera service module 3021, the hardware abstraction layer 303 can successively collect and process the corresponding frame images, and send each processed frame image to the upper layer application for display until the video call ends.

[0120] In one embodiment of the present application, after the image acquisition module 3031 puts the first frame image P1 into the image buffer queue, the algorithm detection module 3032 and the image cropping module 3033 both obtain the first frame image P1 from the image buffer queue at a frequency of once every 10ms. After the algorithm detection module 3032 performs algorithm detection on the first frame image P1, it puts the detection result R1 of the first frame image P1 into the detection result buffer queue. After the image cropping module 3033 obtains the first frame image P1, it performs operation 1 for obtaining a detection result from the detection result buffer queue in real time.

[0121] If before executing operation 1, the algorithm detection module 3032 has placed the detection result R1 of the first frame image P1 into the detection result buffer queue (case 1), then the image cropping module 3033 can obtain the detection result R1 by executing operation 1, and then crop the first frame image P1 according to the detection result R1 to obtain the cropped image P1′ (i.e., R1+P1), and send the cropped image P1′ to the video call application 3011 for display. The video call application 3011 then displays the cropped image P1′ of the first frame image.

[0122] If the algorithm detection module 3032 puts the detection result R1 of the first frame image P1 into the detection result buffer queue after executing operation 1 (case 2), the image cropping module 3033 does not obtain the detection result R1 by executing operation 1, and the image cropping process has not been performed yet, so the obtained first frame image P1 is sent to the video call application 3011 for display. The video call application 3011 displays the first frame image P1 at this time.

[0123] In the above situation 2, the detection result R1 of the first frame image P1 is in the detection result buffer queue and has not been out of the queue. After that, after the image acquisition module 3031 puts the acquired second frame image P2 into the image buffer queue, the algorithm detection module 3032 and the image cropping module 3033 both obtain the second frame image P2 from the image buffer queue at a frequency of once every 10ms. After the algorithm detection module 3032 performs algorithm detection on the second frame image P2, it puts the detection result R2 of the second frame image P2 into the detection result buffer queue. After the image cropping module 3033 obtains the second frame image P2, it executes operation 2 for obtaining a detection result from the detection result buffer queue in real time.

[0124] In the above situation 2, since the detection result R1 of the first frame image P1 is in the detection result buffer queue, the image cropping module 3033 can obtain the detection result R1 by performing operation 2, and crops the second frame image P2 according to the detection result R1 to obtain the cropped image P2″ (i.e. R1+P2), and sends the cropped image P2″ to the video call application 3011 for display. The video call application 3011 then displays the cropped image P2″ of the second frame image P2.

[0125] In the above situation 2, since the detection result R1 is used to crop the second frame image P2, the detection result R2 of the second frame image P2 is in the detection result buffer queue and has not been dequeued. Based on the same implementation principle, for the third frame image P3 captured by the image acquisition module 3031, the third frame image P3 is cropped according to the detection result R2, and the video call application 3011 displays the cropped image P3″ (ie, R2+P3). This cycle is repeated until the video call ends.

[0126] Since the acquisition time of two adjacent frames of images by the camera is not much different, the image content of the two adjacent frames of images is not much different. Therefore, when the current frame image is cropped according to the detection result of the previous frame image, it can usually have a cropping effect of cropping the current frame image according to the detection result of the current frame image, which can ensure the accuracy of image processing, so that the image display effect meets expectations.

[0127] In the above situation 1, the detection result R1 of the first frame image P1 is not in the detection result buffer queue and has been out of the queue. Then, after the image acquisition module 3031 captures the second frame image P2, based on whether the detection result R2 of the second frame image P2 is placed in the detection result buffer queue before the image cropping module 3033 performs the operation of obtaining the detection result, the video call application 3011 displays the second frame image P2, or displays the image P2″ (i.e., R1+P2) obtained by cropping the second frame image P2 according to the detection result (i.e., R1) used in the last image cropping process, or displays the image P2′ (i.e., R2+P2) obtained by cropping the second frame image according to the detection result R2.

[0128] In the above situation 1, if the video call application 3011 displays the second frame image P2 or image P2″, the video call application 3011 may subsequently display image P3″ (ie, R2+P3) obtained by cropping the third frame image P3 according to the detection result R2.

[0129] In the above situation 1, if the video call application 3011 displays image P2′, the detection result R2 is not in the detection result buffer queue and has been out of the queue. Then, after the image acquisition module 3031 acquires the third frame image P3, based on whether the detection result R3 of the third frame image P3 is placed in the detection result buffer queue before the image cropping module 3033 performs the operation of obtaining the detection result, the video call application 3011 displays the third frame image P3, or displays image P3″ (i.e. R2+P3) obtained by cropping the third frame image P3 according to the detection result (i.e. R2) used in the last image cropping process, or displays image P3′ (i.e. R3+P3) obtained by cropping the third frame image P3 according to the detection result R3. This cycle is repeated until the video call ends.

[0130] Figure 3 In the illustrated embodiment, the data stream is divided into two paths, one path is sent to the algorithm detection module for detection, and the other path is sent for cropping, and the result of the algorithm operation is synchronously provided to the module for image cropping. Based on the parallel execution of the two-path processing, not only can the expected image display effect be achieved, but also the single-frame image processing time can be shortened, so that the frame rate of the picture can be achieved. Figure 3 The illustrated embodiment may be applicable to a video call scenario at a high resolution, or may be applicable to a video call scenario at a low resolution.

[0131] In scenarios where image processing time is long and algorithm detection performance is insufficient, Figure 3With the implementation of the illustrated embodiment, the electronic device can have the expected image display effect (for example, the main user is relatively centered in the video display screen), and the video frame rate meets the standard (for example, the detection frame rate of portrait tracking in video calls can reach 30fps), so that the user has a good video call experience.

[0132] In order to increase the frame rate of the picture, another embodiment of the present application provides an image processing flow of an electronic device, which is suitable for camera preview scenarios. The electronic device may include an application layer, an application framework layer, a hardware abstraction layer and a camera; the application layer includes a camera application; the application framework layer includes a camera service module; the hardware abstraction layer includes an image acquisition module, an algorithm detection module, and an image cropping module.

[0133] The camera application can send the distribution information to the camera service module in response to the camera preview event, and after the image acquisition module completes the distribution processing according to the distribution information, send a repeat request to the camera service module, and display the image sent by the image acquisition module through the camera service module. For the implementation of the module interaction between the application framework layer, the hardware abstraction layer and the camera, please refer to Figure 3 The related technical description of the illustrated embodiment will not be repeated here for this embodiment.

[0134] In order to increase the number of picture frames, another embodiment of the present application provides an image processing flow of an electronic device, which is suitable for camera recording scenarios. The electronic device may include an application layer, an application framework layer, a hardware abstraction layer and a camera; the application layer includes a camera application; the application framework layer includes a camera service module; the hardware abstraction layer includes an image acquisition module, an algorithm detection module, and an image cropping module.

[0135] The camera application can send the distribution information to the camera service module in response to the camera recording event, and after the image acquisition module completes the distribution processing according to the distribution information, send a repeat request to the camera service module, and display the image sent by the image acquisition module through the camera service module. For the implementation of the module interaction between the application framework layer, the hardware abstraction layer and the camera, please refer to Figure 3 The related technical description of the illustrated embodiment will not be repeated here for this embodiment.

[0136] To increase the frame rate, please refer to Figure 4 , an embodiment of the present application provides another module interaction timing diagram for image processing. Figure 4 The illustrated embodiment may be applicable to video call scenarios, camera preview scenarios, camera recording scenarios, and the like.

[0137] like Figure 4As shown, the electronic device may include an application package 401 of a first application, a camera manager (for example, recorded as CameraManager) 402, a camera device instance (for example, recorded as CameraDeviceImpl) 403, a camera device client (for example, recorded as CameraDeviceClient) 404, a camera device (for example, recorded as Camera3Device) 405, and a camera device session (for example, recorded as CameraDeviceSession) 406.

[0138] In one embodiment, the first application may be a video call application corresponding to a video call scene. In another embodiment, the first application may be a camera application corresponding to a camera preview scene or a camera recording scene.

[0139] In one embodiment, the application package 401 may be an APK (Android application package) of the first application.

[0140] In one embodiment, Camera 2 API (Application Programming Interface) may include a camera manager 402 and a camera device instance 403 .

[0141] In one embodiment, the CameraService (camera service module) in the application framework layer of the electronic device may include a camera device client 404 and a camera device 405 .

[0142] In one embodiment, the CameraHal (camera hardware abstraction layer) of the electronic device may include a camera device session 406 .

[0143] In one embodiment of the present application, the camera device session 406 may include an image acquisition module, an algorithm detection module, and an image cropping module. For the implementation of the module functions of these three modules, please refer to Figure 3 Description of related technologies in the illustrated embodiments.

[0144] like Figure 4 As shown, the image processing flow may include the following:

[0145] Step 4.1, in response to the first instruction, the application package 401 sends a request for obtaining a camera identification number (Identity document, id) to the camera manager 402, so that the camera manager 402 returns a list of camera identification numbers.

[0146] The camera manager 402 may provide the application package 401 with a camera identification number list according to the received camera identification number acquisition request, so that the application package 401 can acquire the camera identification number accordingly.

[0147] In one embodiment, the relevant implementation code for sending the request for obtaining the camera identification number to the camera manager 402 may include: CameraManager.getCameraIdList().

[0148] In one embodiment of the present application, when the first application is a video call application, the first instruction may be an instruction to start a video call. In one feasible implementation, the user may trigger a video initiation control in the video call application to issue an instruction to start a video call.

[0149] Step 4.2, the application package 401 obtains the camera identification number according to the camera identification number list sent by the camera manager 402. In one embodiment, the relevant implementation code for obtaining the camera ID may include: CameraManager.getCameraCharacteristics(CameraId).

[0150] In a feasible implementation, the acquired camera identification number may be the identification number of a camera used for image acquisition, and the image acquisition module in the camera device session 406 may acquire the image captured by the camera.

[0151] Step 4.3, the application package 401 determines whether the key of the corresponding camera exists according to the obtained camera identification number, and if so, executes step 4.4.

[0152] In a feasible implementation, if it is determined that the key of the corresponding camera does not exist, step 4.4 may not be performed.

[0153] In one embodiment, the relevant implementation code for determining whether a Key exists may include:

[0154] mCharacteristics.getKeys().contains(HIHONOR_FACEAUTOTRACK_SUPPORTED)KEY NAME:com.hihonor.device.capabilities.faceAutoTrackModeSupported Byte type.

[0155] In step 4.4, the application package 401 determines whether the corresponding camera supports portrait tracking according to the obtained camera identification number, and if it does, the camera manager 402 is triggered to open the corresponding camera according to the camera key. The camera in the open state can sequentially capture each frame of the image within its image acquisition range.

[0156] In another embodiment of the present application, step 4.4 may be performed first and then step 4.3, that is, step 4.3 is performed when step 4.4 is judged to be passed, and step 4.5 is performed when step 4.3 is judged to be passed.

[0157] In one embodiment, the relevant implementation code for determining whether the corresponding camera supports portrait tracking may include: mCharacteristics.get(HIHONOR_FACEAUTOTRACK_SUPPORTED) 0 Not supported 1 Supported byte type.

[0158] Since face detection can be performed on the images captured by the camera to achieve user face tracking, and then the image can be cropped according to the face detection result to display the cropped image, if the judgment result is "0 not supported", the camera manager 402 will not be triggered to open the corresponding camera. If the judgment result is "1 supported", the camera manager 402 will be triggered to open the corresponding camera.

[0159] In one embodiment, the relevant implementation code for triggering the camera manager 402 to open the corresponding camera may include: CameraManager.openCamera().

[0160] In step 4.5, the application package 401 sends an instruction for creating a capture request to the camera device instance 403. In one embodiment, the relevant implementation code of step 4.5 may include: CameraDevice.createCaptureRequest().

[0161] In step 4.6, the camera device instance 403 sends an instruction for creating a system setting request to the camera device client 404 according to the instruction for creating a capture request. In one embodiment, the relevant implementation code of step 4.6 may include: createDefaultRequest(templateType).

[0162] In step 4.7, the camera device client 404 transmits the received instruction of creating a system setting request to the camera device 405.

[0163] In step 4.8, the camera device 405 sends the instruction for setting the system setting request to the camera device session 406 according to the instruction for creating the system setting request. In one embodiment, the relevant implementation code of step 4.8 may include: constructDefaultRequestSettings.

[0164] Step 4.9, the camera device session 406 obtains the camera metadata and the current application package name according to the instruction of setting the system setting request, and returns the obtained camera metadata and the current application package name to the camera device 405.

[0165] In one embodiment, the relevant implementation code of step 4.9 may include:

[0166] com.hihonor.capture.metadata.packageName.

[0167] In one embodiment, the camera metadata acquired by the camera device session 406 may be metadata of an opened camera.

[0168] In one embodiment, the current application package name may be the package name of the application program package 401 of the first application.

[0169] In step 4.10, the camera device 405 adds the current application package name to the camera metadata, and sends the camera metadata with the current application package name added to the camera device client 404. By adding the current application package name to the camera metadata, the current scene information is brought in.

[0170] In step 4.11, the camera device client 404 transmits the camera metadata with the current application package name added to it to the application package 401.

[0171] By passing the camera metadata with the current application package name added thereto to the application package 401 , the capture request builder (CaptureRequest.Builder) of the application package 401 can include the package name and the device capability of the camera.

[0172] After the application package 401 obtains the camera metadata with scene information, it can start the stream distribution process (see steps 4.12 to 4.16).

[0173] Step 4.12: the application package 401 generates an instruction for creating a capture session according to the camera metadata to which the current application package name is added, and sends the instruction for creating a capture session to the camera device instance 403.

[0174] In one embodiment, the relevant implementation code of step 4.12 may include:

[0175] CameraDevice.CreateCaptureSession(SessionConfiguration config).

[0176] Step 4.13, the camera device instance 403 sends the interface flow distribution information and the information of configuring the session key (sessionkey) to the camera device client 404 according to the instruction of creating the capture session.

[0177] In one embodiment, the flow allocation information may include frame rate and resolution. The flow allocation module of the Hal layer may configure the flow according to the flow allocation information, and configure the frame rate and resolution for the camera.

[0178] Step 4.14, the camera device client 404 processes the received interface flow distribution information and the information for configuring the session key, and sends the processed session key and the flow distribution information corresponding to the camera hardware abstraction layer (camera hal layer) to the camera device 405.

[0179] Step 4.15, the camera device 405 sends the session key and the stream allocation information corresponding to the camera hardware abstraction layer to the camera device session 406.

[0180] Step 4.16, the camera device session 406 performs stream allocation processing according to the session key and the stream allocation information of the corresponding camera hardware abstraction layer, and sends a stream allocation completion notification to the application package 401 after the stream allocation processing is completed.

[0181] In one embodiment, the flow distribution process may include mode selection and pipeline (transmission) configuration.

[0182] In one embodiment, after the flow allocation process is completed, the message "OK" may be sent to the application package 401 so that the application package 401 is informed that the flow allocation process is completed.

[0183] In one embodiment, the technical implementation of the flow distribution processing can refer to Figure 3 The relevant technical contents in the illustrated embodiment will not be elaborated in detail in this embodiment.

[0184] After the camera device session 406 completes the streaming process, the application package 401 may send a repeat request to trigger the camera device session 406 to perform image processing (such as algorithm detection and cropping processing) on ​​the image captured by the camera through the camera service module.

[0185] In step 4.17, the application package 401 sends an instruction for setting a repeat request to the camera device instance 403. In one embodiment, the relevant implementation code of step 4.17 may include: captureSession.setRepeatingRequest.

[0186] Step 4.18, the camera device instance 403 sends an instruction to submit a request list (for example, recorded as submitRequesList) to the camera device client 404 according to the instruction to set a repeated request.

[0187] Step 4.19, the camera device client 404 sends an instruction for setting a streaming request list (for example, recorded as setStreamingRequestList) to the camera device 405 according to the instruction for submitting the request list.

[0188] Step 4.20: the camera device 405 sends an image acquisition request to the camera device session 406 according to the instruction for setting the streaming request list.

[0189] Step 4.21, the camera device session 406 acquires a frame of image captured by the camera according to the image acquisition request, processes the acquired image to obtain a processed image, and sends the processed image to the application package 401 for display.

[0190] In one embodiment of the present application, the camera device session 406 may include an image acquisition module, an algorithm detection module, and an image cropping module. The image acquisition module acquires images from the turned-on camera, and the acquired images are placed in an image buffer queue, and the algorithm detection module and the image cropping module asynchronously acquire images from the image buffer queue.

[0191] The algorithm detection module performs face detection on the acquired image and stores the generated detection results in the detection result buffer queue. After acquiring the image, the image cropping module attempts to obtain the detection result from the detection result buffer queue. If the detection result is obtained, the image is cropped accordingly, and the cropped image is used as the capture result (for example, recorded as CaptureResult). If the detection result is not obtained, no cropping is performed, and the acquired image is directly used as the capture result, or the image is cropped according to the detection result used in the last cropping, and the cropped image is used as the capture result.

[0192] The image cropping module then returns the captured result to the application package 401 for display. In this way, the camera device session 406 completes the processing of the image acquisition request (for example, recorded as processcapturerequest). For the specific implementation of image processing, please refer to Figure 3The relevant technical contents in the illustrated embodiment will not be elaborated in detail in this embodiment.

[0193] like Figure 5 As shown, an embodiment of the present application provides an image processing method, which may include the following steps 501 to 506. The execution subject of the image processing method may be an electronic device.

[0194] In one embodiment, the electronic device may be a large-screen device, such as a large-screen display device capable of supporting video conferencing.

[0195] Step 501, storing a first image in a first storage space, the first image being any image captured by a camera. After step 501, steps 502 and 504 may be performed respectively.

[0196] In one embodiment of the present application, the image processing method may further include: in response to any one of a camera preview event, a camera recording event, and a video call event, executing an operation of acquiring an image captured by a camera; wherein the video call event includes at least one of an event of actively initiating a video call and an event of answering a video call initiated by the other party. By asynchronously executing image detection and image cropping in the camera preview, camera recording, and video call scenarios, an image display effect in which a person is displayed in the center can be achieved, thereby providing a better user experience.

[0197] The electronic device may be installed with a camera application, a video call application, etc. In one embodiment, the video call application may be a video conferencing application.

[0198] In a feasible implementation, when a user opens a camera application of an electronic device, the electronic device can detect a camera preview event.

[0199] In a feasible implementation, after a user opens a camera application of an electronic device, if the user triggers a video recording control of the camera application, the electronic device can detect a camera recording event.

[0200] In one feasible implementation, when a user triggers a video call control of a video call application, the electronic device can detect an event of actively initiating a video call, and when a user triggers a video answering control of a video call application, the electronic device can detect an event of answering a video call initiated by the other party.

[0201] In one embodiment, the camera may be a built-in device of the electronic device. In another embodiment, the camera may be an external device of the electronic device.

[0202] In one embodiment of the present application, the camera of the electronic device may be a wide-angle camera of a large-screen device. In order to avoid the situation in which the mobile phone (e.g., the other device in the video call scenario) cannot see the person in the picture clearly in a video call or other scenario, the electronic device may crop the image according to the position of the person in the image, so that the cropped image can better highlight the person, thereby improving the user experience.

[0203] Each frame of image captured by the camera can be collected in sequence, and after a frame of image is collected, the frame of image is stored in the first storage space. In one embodiment of the present application, the first storage space is an image buffer queue. Based on the order of image acquisition time, the images in the image buffer queue can comply with the first-in-first-out principle.

[0204] In one embodiment, the electronic device may include a first thread, a second thread and a third thread, the first thread executes step 501, the second thread executes steps 502 to 503, and the third thread executes steps 504 to 506, and the second thread and the third thread execute asynchronously.

[0205] In one embodiment, the Figure 3 In the illustrated embodiment, the image acquisition module 3031 executes a first thread, the algorithm detection module 3032 executes a second thread, and the image cropping module 3033 executes a third thread.

[0206] In one embodiment of the present application, the image processing method also includes: upon receiving an image acquisition request, executing a fourth operation of acquiring an image from a third storage space at a third frequency until a frame of image is acquired, wherein the third storage space is used to store images captured by the camera performing an image capturing operation at a first frequency; executing the first operation at a second frequency; executing the second operation at a second frequency; wherein the second frequency is greater than the first frequency, and the third frequency is greater than the first frequency; storing the first image in the first storage space includes: storing the first image acquired by executing the fourth operation in the first storage space.

[0207] In one embodiment, the first frequency can be acquired once every 33ms, and the second frequency can be acquired once every 10ms. This ensures that within the image acquisition time interval corresponding to the first frequency (i.e., 33ms), the first operation and the second operation can usually acquire the current frame image and complete the determination of the image to be displayed.

[0208] In one embodiment, the third frequency may be equal to the second frequency. In another embodiment, the third frequency may not be equal to the second frequency.

[0209] By limiting the second frequency to be greater than the first frequency and the third frequency to be greater than the first frequency, real-time image display can be achieved and the number of picture frames can be guaranteed.

[0210] Step 502: Obtain a first image by executing a first operation of acquiring an image from a first storage space.

[0211] After the first image is stored in the first storage space, the first operation and the second operation can be asynchronously performed to obtain the first image respectively, the first image obtained by the first operation is used for image detection, and the first image obtained by the second operation is used for image cropping, so that image detection and image cropping are performed asynchronously. Based on the asynchronous execution of image detection and image cropping, the image processing time of one frame of image can be shortened, thereby increasing the number of picture frames.

[0212] Step 503: perform image detection on the first image obtained by executing the first operation, generate a detection result of the first image, and store the detection result of the first image in a second storage space.

[0213] After completing the image detection, the detection result of the first image can be stored in the second storage space in real time. In one embodiment of the present application, the second storage space is a detection result buffer queue. Based on the chronological order of the generation of the detection results, the detection results in the detection result buffer queue can comply with the first-in-first-out principle. After any detection result is out of the queue, the detection result buffer queue will no longer contain the detection result.

[0214] Step 504: Obtain the first image by executing a second operation of acquiring the image from the first storage space, wherein the first operation and the second operation are executed asynchronously.

[0215] Step 505: After obtaining the first image by executing the second operation, executing a third operation of acquiring the detection result from the second storage space.

[0216] In one embodiment, after obtaining the first image for image cropping, an attempt may be made to obtain the detection result from the second storage space in real time. Depending on whether there is a detection result in the second storage space, the detection result may be obtained or not.

[0217] In one embodiment of the present application, the second storage space is a detection result buffer queue; the third operation is used to obtain the detection result that is relatively earliest stored in the detection result buffer queue when there is more than one detection result in the detection result buffer queue. When there is more than one detection result, by obtaining the detection result based on the first-in-first-out rule, it can be ensured that the electronic device always crops the current frame image with the detection result of the previous frame image, so that the image cropping can be accurately executed.

[0218] Step 506: If the detection result is obtained by executing the third operation, the first image obtained by executing the second operation is cropped according to the obtained detection result to obtain an image to be displayed, wherein after the detection result is obtained by executing the third operation, the second storage space does not include the obtained detection result.

[0219] If a detection result is obtained in the second storage space, image cropping can be performed accordingly to obtain an image to be displayed, and then the obtained image to be displayed can be displayed.

[0220] Figure 5 In the illustrated embodiment, the data stream is divided into two paths, one for detection and the other for cropping, and the result of the algorithm operation is synchronously provided to the module for image cropping. Based on the parallel execution of the two-path processing, not only can the expected image display effect be achieved, but also the single-frame image processing time can be shortened, so that the frame rate of the picture can be achieved.

[0221] In one embodiment of the present application, the image processing method further includes: if no detection result is obtained by performing the third operation, image cropping may not be performed, and the first image may be directly used as the image to be displayed. In this way, the immediacy and non-interruption of image display can be supported.

[0222] In another embodiment of the present application, the image processing method further includes: if no detection result is obtained by performing the third operation, determining whether the cropping process has been performed; if it is determined that the cropping process has been performed, the first image obtained by performing the second operation is cropped according to the detection result used in the last cropping process to obtain the image to be displayed. In this way, the consistency of the image display effect can be supported.

[0223] Considering that there is no previous frame image of the first frame image, so that when the first frame image is acquired, the cropping process has not been performed, if no detection result is obtained after the first frame image is acquired, the first frame image can be directly displayed. For non-first frame images, if no detection result is obtained after the frame image is acquired, the frame image can be cropped according to the detection result used in the last cropping process, and the cropped image can be displayed.

[0224] Thus, in one embodiment, the specific implementation method of determining whether the cropping process has been performed may be: determining whether it is the first frame image. If it is the first frame image, it indicates that the cropping process has not been performed; if it is not the first frame image, it indicates that the cropping process has been performed. If the detection result is not obtained by performing the third operation, and the first image is the first frame image, the first image is used as the image to be displayed; if the detection result is not obtained by performing the third operation, and the first image is not the first frame image, the image to be displayed is the image obtained by cropping the first image according to the detection result used in the last cropping process.

[0225] In one embodiment of the present application, the image processing method further includes: performing stream distribution processing according to a set frame rate so that the camera performs image capturing operation at a first frequency; wherein the first frequency is obtained according to the set frame rate.

[0226] In one embodiment, the distribution module of the hal layer can perform distribution processing according to the set frame rate.

[0227] In one embodiment, the distribution module of the HAL layer can be the main process, and the image acquisition module, image decoding module, algorithm detection module, and image cropping module of the HAL layer can be four threads, and the image acquisition module can be the main thread among the four threads. In a feasible implementation method, the distribution module, image acquisition module, and image decoding module in the HAL layer can all be used as Figure 3 Submodules of the image acquisition module 3031 in the illustrated embodiment.

[0228] By performing stream processing according to the set frame rate before image acquisition, the camera can be controlled to capture images at the expected frequency, thereby providing support for achieving the target frame rate.

[0229] An embodiment of the present application also provides an image processing device, including: an image acquisition module, used to store a first image into a first storage space, the first image being any image taken by a camera; an algorithm detection module, used to obtain the first image by executing a first operation of obtaining the image from the first storage space; performing image detection on the first image obtained by executing the first operation, generating a detection result of the first image, and storing the detection result of the first image into a second storage space; an image cropping module, used to obtain the first image by executing a second operation of obtaining the image from the first storage space; after obtaining the first image by executing the second operation, executing a third operation of obtaining the detection result from the second storage space; if the detection result is obtained by executing the third operation, cropping the first image obtained by executing the second operation according to the obtained detection result to obtain an image to be displayed; wherein the first operation and the second operation are executed asynchronously; wherein, after obtaining the detection result by executing the third operation, the second storage space does not include the obtained detection result.

[0230] An embodiment of the present application also provides an electronic chip, which task processing chip is installed in an electronic device (UE), and the electronic chip includes: a processor, which is used to execute computer program instructions stored in a memory, wherein when the computer program instructions are executed by the processor, the electronic chip is triggered to execute the method steps provided by any method embodiment of the present application.

[0231] An embodiment of the present application also proposes a terminal device, which includes a communication module, a memory for storing computer program instructions, and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, the terminal device is triggered to execute the method steps provided by any method embodiment of the present application.

[0232] An embodiment of the present application also proposes a server device, which includes a communication module, a memory for storing computer program instructions, and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, the server device is triggered to execute the method steps provided by any method embodiment of the present application.

[0233] An embodiment of the present application also provides an electronic device, which includes multiple antennas, a memory for storing computer program instructions, a processor for executing computer program instructions, and a communication device (such as a communication module that can implement 5G communication based on the NR protocol), wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps provided by any method embodiment of the present application.

[0234] Specifically, in one embodiment of the present application, one or more computer programs are stored in the above-mentioned memory, and the one or more computer programs include instructions. When the instructions are executed by the above-mentioned device, the above-mentioned device executes the method steps described in the embodiment of the present application.

[0235] Furthermore, the devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0236] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0237] In the several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0238] Specifically, in one embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is run on a computer, the computer executes the method steps provided in the embodiment of the present application.

[0239] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes the method steps provided in the embodiment of the present application.

[0240] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0241] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0242] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0243] The integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program code.

[0244] In the embodiments of the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0245] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0246] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0247] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the same or similar parts between the various embodiments in this application can be referred to each other. For example, the specific working process of the system, device and unit described in the embodiments of this application can refer to the corresponding process in the method embodiment of this application, and will not be repeated here.

[0248] The above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: storing a first image in a first storage space, wherein the first image is any image captured by a camera; the camera performs an image capturing operation at a first frequency; The first image is obtained by executing a first operation of acquiring an image from the first storage space, and the first image is obtained by executing a second operation of acquiring an image from the first storage space, wherein the first operation and the second operation are performed asynchronously; and a frequency of performing the second operation is greater than the first frequency; Performing image detection on the first image obtained by executing the first operation, the image detection including face detection, generating a detection result of the first image, the detection result including a face position, and storing the detection result of the first image in a second storage space; After obtaining the first image by performing the second operation, performing a third operation of acquiring a detection result from the second storage space; If a detection result is obtained by performing the third operation, the first image obtained by performing the second operation is cropped according to the obtained detection result to obtain an image to be displayed, wherein after the detection result is obtained by performing the third operation, the second storage space does not include the obtained detection result; The method further comprises: If no detection result is obtained by performing the third operation, the first image is used as the image to be displayed, or the first image obtained by performing the second operation is cropped according to the detection result used in the last cropping process to obtain the image to be displayed.

2. The method according to claim 1, characterized in that The method further comprises: If no detection result is obtained by performing the third operation, determining whether a cropping process has been performed; In the case where it is determined that the cropping process has been performed, the step of performing cropping on the first image obtained by performing the second operation according to the detection result used for the last cropping process to obtain the image to be displayed is performed.

3. The method according to claim 1, characterized in that The method further comprises: When an image acquisition request is received, a fourth operation of acquiring an image from a third storage space is performed at a third frequency until a frame of image is acquired, wherein the third storage space is used to store images captured by the camera performing the image capturing operation at the first frequency; performing the first operation at a second frequency; performing the second operation at the second frequency; Wherein, the second frequency is greater than the first frequency, and the third frequency is greater than the first frequency; The storing the first image into the first storage space includes: storing the first image acquired by performing the fourth operation into the first storage space.

4. The method according to claim 1, characterized in that: The first storage space is an image buffer queue; The second storage space is a detection result buffer queue; The third operation is used to obtain the earliest detection result stored in the detection result buffer queue when there is more than one detection result in the detection result buffer queue.

5. The method according to claim 1, characterized in that The method further comprises: Performing stream distribution processing according to a set frame rate so that the camera performs an image capturing operation at a first frequency; The first frequency is obtained according to the set frame rate.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to any one of a camera preview event, a camera recording event, and a video call event, performing an operation of acquiring an image captured by the camera; The video call event includes at least one of an event of actively initiating a video call and an event of answering a video call initiated by the other party.

7. An image processing device, characterized in that: include: An image acquisition module, used for storing a first image into a first storage space, where the first image is any image captured by a camera; the camera performs an image capturing operation at a first frequency; an algorithm detection module, configured to obtain the first image by performing a first operation of acquiring the image from the first storage space; Performing image detection on the first image obtained by executing the first operation, the image detection including face detection, generating a detection result of the first image, the detection result including a face position, and storing the detection result of the first image in a second storage space; an image cropping module, configured to obtain the first image by performing a second operation of acquiring the image from the first storage space; After obtaining the first image by performing the second operation, performing a third operation of acquiring a detection result from the second storage space; If a detection result is obtained by performing the third operation, the first image obtained by performing the second operation is cropped according to the obtained detection result to obtain an image to be displayed; the frequency of performing the second operation is greater than the first frequency; wherein the first operation and the second operation are performed asynchronously; After the detection result is obtained by executing the third operation, the second storage space does not include the obtained detection result; The image processing device is also used to: if no detection result is obtained by performing the third operation, use the first image as the image to be displayed, or crop the first image obtained by performing the second operation according to the detection result used for the last cropping process to obtain the image to be displayed.

8. An electronic chip, characterized in that: include: A processor, configured to execute computer program instructions stored in a memory, wherein when the computer program instructions are executed by the processor, the electronic chip is triggered to execute the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions, a processor for executing the computer program instructions, and a communication device, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Video call method, terminal and computer-readable storage medium

    CN107528938A

  • Picture processing method and device, equipment and storage medium

    CN112036319A

  • Video processing method and electronic equipment

    CN113014793A

  • Automatic person frame selection method and storage equipment

    CN113657255A

  • Vision-based positioning shooting method and system

    CN114241570A