Image processing method, terminal equipment, chip system and readable storage medium

Abstract

The invention provides an image processing method, terminal equipment, a chip system and a readable storage medium, the method is applied to the terminal equipment, the terminal equipment comprises a camera, the camera comprises a color image sensor and a depth sensor, a camera application runs on the terminal equipment, and the method comprises the following steps: in response to a first operation, controlling the depth sensor, a depth image is collected according to a first frame rate, a color image sensor is controlled, a color image is collected according to a second frame rate, and the first operation comprises a starting operation on a camera application program; and when the acquisition moment of acquiring the depth image according to the first frame rate is not aligned with the acquisition moment of acquiring the color image according to the second frame rate, adjusting the frame rate of the depth sensor from the first frame rate to a third frame rate, so that the depth sensor acquires the depth image according to the third frame rate. Therefore, it can be ensured that the acquisition moment of the depth image is aligned with the acquisition moment of the color image, the subsequent image processing effect is ensured, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, specifically to an image processing method, a terminal device, a chip system, and a readable storage medium. Background Technology

[0002] Mobile phones, tablets and other terminal devices are usually equipped with cameras, which can capture images.

[0003] Currently, the camera in a terminal device is equipped with both a color image sensor (red-green-blue sensor, RGB) and a depth sensor (direct time-of-flight sensor, DTOF). RGB is used to capture color images, and DTOF is used to capture depth images. After the RGB image is captured and the DTOF image is captured, the terminal device can perform relevant image processing on the color image and depth image at the same time to display the interface.

[0004] However, in the above implementation, there may be a situation where the RGB image captured by RGB and the depth image captured by DTOF are not aligned in time. If the RGB image captured by RGB and the depth image captured by DTOF are not aligned in time, it will affect the subsequent image processing results such as autofocus calculation and bokeh calculation, resulting in poor image display effect in the interface and affecting user experience. Summary of the Invention

[0005] This application provides an image processing method, terminal device, chip system, and readable storage medium that can ensure the acquisition time of the RGB image is aligned with the acquisition time of the depth image as much as possible, thereby ensuring the subsequent image processing effect and improving the user experience.

[0006] In a first aspect, this application provides an image processing method applied to a terminal device, the terminal device including a camera, the camera including a color image sensor and a depth sensor, and a camera application running on the terminal device, the method including:

[0007] In response to a first operation, the depth sensor is controlled to acquire a depth image at a first frame rate, and the color image sensor is controlled to acquire a color image at a second frame rate. The first operation includes launching a camera application, and the color image is an RGB image in the red-green-blue color space. When the acquisition time of the depth image at the first frame rate is not aligned with the acquisition time of the color image at the second frame rate, the frame rate of the depth sensor is adjusted from the first frame rate to a third frame rate so that the depth sensor acquires the depth image at the third frame rate. Wherein, the acquisition time of the depth image at the third frame rate is aligned with the acquisition time of the color image at the second frame rate.

[0008] In the above method, if the acquisition time of the depth image based on the first frame rate is not aligned with the acquisition time of the color image based on the second frame rate, the frame rate of the depth sensor can be redefined so that the depth sensor switches from acquiring the depth image based on the first frame rate to acquiring the depth image based on the third frame rate. This ensures that the acquisition time of the depth image based on the third frame rate is aligned with the acquisition time of the color image based on the second frame rate. In this way, subsequent image processing can be better performed with the aligned color and depth images, which can improve the user experience.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the first operation further includes: activating the portrait mode of the camera application in an environment where the brightness value is below the brightness threshold.

[0010] In the above method, when the camera application is in portrait mode and the current environment is a low-light environment, by adjusting the frame rate of the depth sensor, the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate. This allows for bokeh processing based on accurate depth image depth data, thereby improving the bokeh effect in portrait mode.

[0011] In conjunction with the first aspect, in certain implementations of the first aspect, when the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate are not aligned, adjusting the frame rate of the depth sensor from the first frame rate to the third frame rate includes:

[0012] Determine whether the first difference between the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate is greater than or equal to a time threshold; if the first difference is greater than or equal to the time threshold, determine that the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate are not aligned; adjust the frame rate of the depth sensor from the first frame rate to the third frame rate.

[0013] In this application, the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate can be determined by the difference between the acquisition time of the depth image and the color image being greater than or equal to a preset threshold.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the acquisition time of the depth image is the time when the first row of data of the depth image begins to be read; the acquisition time of the color image is the time when the middle row of data of the color image is read out.

[0015] In the above method, the terminal device can determine whether the acquisition time of the depth image acquired according to the first frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate by determining whether the difference between the time when the first row of data of the depth image is read and the time when the middle row of data of the color image is read is greater than or equal to a time threshold.

[0016] In conjunction with the first aspect, in certain implementations of the first aspect, determining whether a first difference between the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate is greater than or equal to a time threshold includes:

[0017] Determine whether the first difference between the time when the first row of data in the depth image begins to be read and the time when the middle row of data in the color image is read is greater than or equal to a time threshold.

[0018] In the above method, the terminal device can determine that the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate are misaligned by determining that the first difference between the time when the first row of data of the depth image is read and the time when the middle row of data of the color image is read is greater than or equal to a time threshold.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0020] When the first difference is less than the time threshold, it is determined that the acquisition time of the depth image acquired according to the first frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate.

[0021] In the above method, the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate can be determined by the first difference being less than the time threshold.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the acquisition time of the depth image is the midpoint of the global exposure period of the depth image; the acquisition time of the color image is the moment when the midpoint row data of the color image is read out.

[0023] In this application, the terminal device can determine whether the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate are aligned by determining whether the difference between the acquisition time of the depth image being the midpoint of the global exposure period of the depth image and the midpoint of the data reading of the color image is greater than or equal to a time threshold.

[0024] In conjunction with the first aspect, in certain implementations of the first aspect, determining whether a first difference between the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate is greater than or equal to a time threshold includes:

[0025] Determine whether the first difference between the midpoint of the global exposure time of the depth image and the midpoint readout time of the color image is greater than or equal to a time threshold.

[0026] In the above method, the terminal device can determine that the acquisition time of the depth image based on the first frame rate and the acquisition time of the color image based on the second frame rate are misaligned by determining that the difference between the midpoint of the global exposure period of the depth image and the midpoint of the data readout of the color image is greater than or equal to a time threshold.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0028] When the first difference is less than the time threshold, it is determined that the acquisition time of the depth image acquired according to the first frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate.

[0029] In the above method, the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate can be determined by the first difference being less than the time threshold.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the third frame rate is determined based on the first difference and the first frame interval, where the first frame interval is the time interval between acquiring two depth images based on the first frame rate.

[0031] In the above method, the third frame rate can be quickly determined by the first difference between the acquisition time of the depth image acquired at the first frame rate and the acquisition time of the color image acquired at the second frame rate, and the time interval between acquiring two frames of depth images at the first frame rate.

[0032] In conjunction with the first aspect, in certain implementations of the first aspect, when the acquisition time of the depth image acquired according to the first frame rate is not aligned with the acquisition time of the color image acquired according to the second frame rate, the frame rate of the depth sensor is adjusted from the first frame rate to the third frame rate so that the depth sensor acquires the depth image according to the third frame rate. The method further includes:

[0033] When the acquisition time of the depth image based on the third frame rate is aligned with the acquisition time of the color image based on the second frame rate, the frame rate of the depth sensor is adjusted from the third frame rate to the first frame rate so that the depth sensor continues to acquire depth images based on the first frame rate.

[0034] In the above method, when the acquisition time of the depth image based on the third frame rate is aligned with the acquisition time of the color image based on the second frame rate, the frame rate of the depth sensor can be restored to the original first frame rate, thereby ensuring that the subsequent depth image and color image can be aligned.

[0035] In conjunction with the first aspect, in certain implementations of the first aspect, when the acquisition time of the depth image acquired according to the first frame rate is not aligned with the acquisition time of the color image acquired according to the second frame rate, the frame rate of the depth sensor is adjusted from the first frame rate to the third frame rate so that the depth sensor acquires the depth image according to the third frame rate. The method further includes:

[0036] Image processing is performed on the depth image and color image to obtain the target image. Image processing includes autofocus processing and blurring processing. The target image includes a focus frame and a blurred area. When the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate, the accuracy of the focus frame position is higher than that when they are not aligned. When the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate, the image sharpness of the focus frame position is higher than that when they are not aligned. When the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate, the accuracy of the blurred area position is higher than that when they are not aligned. A first interface is displayed, which includes the target image.

[0037] In the above method, by adjusting the frame rate of the depth sensor, the positional accuracy of the focus frame and the blurred area of ​​the target image in the first interface can be improved.

[0038] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0039] The depth image and color image are automatically focused to obtain the focus distance. When the acquisition time of the depth image based on the third frame rate is aligned with the acquisition time of the color image based on the second frame rate, the accuracy of the focus distance is higher than that when they are not aligned. Based on the focus distance, the lens position of the camera is adjusted. After the adjustment, the sharpness of the focus area of ​​the color image is higher than that of the focus area of ​​the color image before the adjustment.

[0040] In the above method, the autofocus calculation result, i.e., the focus distance, is obtained through autofocus processing. The camera lens can be adjusted according to the autofocus calculation result to improve the accuracy of frame-by-frame focus tracking.

[0041] In a second aspect, this application provides a terminal device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the terminal device to perform the method in the first aspect and any possible implementation of the first aspect.

[0042] Thirdly, this application provides a chip system applied to a terminal device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the terminal device to execute the methods in the first aspect and any possible implementation of the first aspect.

[0043] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a terminal device, cause the terminal device to perform the methods of the first aspect and any possible implementation thereof.

[0044] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the methods of the first aspect and any possible implementation thereof.

[0045] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0046] Figure 1 A schematic diagram of a depth image and an RGB image provided in an embodiment of this application;

[0047] Figure 2 A human-computer interaction interface diagram provided in one embodiment of this application;

[0048] Figure 3 A human-computer interaction interface diagram provided in one embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of a terminal device provided in one embodiment of this application;

[0050] Figure 5 This application provides a schematic diagram of the software architecture of a terminal device according to an embodiment of the present application.

[0051] Figure 6 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of data acquisition timing provided in an embodiment of this application;

[0053] Figure 8 A schematic diagram of a depth image and an RGB image provided in an embodiment of this application;

[0054] Figure 9 A human-computer interaction interface diagram provided in one embodiment of this application;

[0055] Figure 10 A human-computer interaction interface diagram provided in one embodiment of this application;

[0056] Figure 11 This is a schematic flowchart of an image processing method provided in an embodiment of this application. Detailed Implementation

[0057] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the first transformation relationship of the related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c alone can mean: a alone, b alone, c alone, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0058] For ease of understanding, the examples provided are for reference only and are related to the concepts in the embodiments of this application.

[0059] 1. Color image sensor

[0060] A color image sensor (red-green-blue sensor, RGB) is used to acquire color images. RGB represents three primary colors: red, green, and blue. An RGB sensor generates a color image (RGB image) by sensing the intensity of light of these three colors. The color information of each pixel is determined by the combination of these three colors.

[0061] 2. Depth sensor

[0062] A depth sensor (direct time-of-flight sensor, DTOF) is used to acquire depth images. DTOF calculates the distance from the sensor to the object by measuring the time it takes for light to travel from the sensor to the surface of the object being photographed and back to the sensor, thereby obtaining depth data and creating a depth image based on the depth data.

[0063] 3. Autofocus

[0064] Auto focus tracking (AF) is the process by which a camera continuously tracks a moving object and automatically adjusts the focus to keep the object in focus.

[0065] RGB can provide RGB images, while DTOF can provide depth images. In autofocus calculations, the focus area (the subject being photographed) can be determined using the RGB image, and the focus position can be determined more accurately using the depth data from the depth image. A focus frame is then generated at the focus position to visually display the focus area. Furthermore, the optimal focal length is determined, and the camera's focusing mechanism is adjusted via camera drive to improve the clarity of image acquisition.

[0066] Mobile phones, tablets and other terminal devices are usually equipped with cameras, which can capture images.

[0067] Currently, the camera in a terminal device is equipped with both a color image sensor (red-green-blue sensor, RGB) and a depth sensor (direct time-of-flight sensor, DTOF). RGB is used to capture color images, and DTOF is used to capture depth images. After the RGB image is captured and the DTOF image is captured, the terminal device can perform relevant image processing on the color image and depth image at the same time to display the interface.

[0068] However, in the above implementation, the RGB frame rate changes with the ambient brightness (the RGB frame rate decreases when the brightness value is below the brightness threshold, and increases when the brightness value is above the brightness threshold). This may result in the RGB image captured by RGB and the depth image captured by DTOF being out of sync in time. If the RGB image captured by RGB and the depth image captured by DTOF are out of sync in time, it will affect the subsequent image processing results such as autofocus calculation and bokeh calculation, resulting in poor image display in the interface and affecting the user experience.

[0069] The following explanation uses the portrait mode of a camera application as an example:

[0070] In this camera application, the shooting mode is portrait mode. When the current environment is bright (brightness value greater than or equal to the brightness threshold), the camera combines dual RGB image acquisition. Because the DTOF sensor actively emits light signals, it can operate under various lighting conditions, including low light and no light. In low light environments (brightness value less than the brightness threshold), DTOF can act as an active lighting device. The camera can combine RGB and DTOF to acquire images, improving the accuracy of object imaging and measurement in low-light environments.

[0071] When the camera application receives a start command, it notifies the camera to capture an image. RGB can capture RGB images, and DTOF can capture depth images. If the camera application activates portrait mode based on the user's instruction, and the current environment is a low-light environment, the application framework layer of the terminal device can determine the color image and depth image captured at the same time. It performs autofocus calculation based on the RGB image, making the face the focus subject, and blurs the background area other than the face based on the depth data of the depth image, so that the image with the blurred background area is displayed in the camera application.

[0072] If the RGB image and the depth image are not aligned in time, that is, the depth image corresponding to the RGB image at a certain moment is not the same as the depth image at that moment, for example, the depth image was acquired 200ms later than the RGB image, then the blurring result may be inaccurate when performing blurring processing based on the depth data of the depth image. In addition, if autofocus calculation is performed, the position of the generated focus frame may also be inaccurate.

[0073] Figure 1 (a) in the image is a depth image. Figure 1 (b) in the image is an RGB image. Figure 1 (c) in the diagram is a schematic of the fusion of the depth image and the RGB image. Figure 1 As can be seen in (c), some areas in the face region have the same color as the scenery region (background region), that is, the same depth value. This indicates that the depth image and the RGB image are misaligned in terms of acquisition time.

[0074] If the RGB image and depth image are not aligned in terms of acquisition time, when blurring a scene region based on the depth data from the depth image, the inaccurate depth data will result in an inaccurate blurred area. Figure 2 As can be seen, after blurring, when displaying the interface, the blurred area changes from including the entire background area except for the face area to including part of the face area and part of the background area, resulting in poor image display and affecting user experience.

[0075] Similarly, combining Figure 1For various shooting modes such as still mode and portrait mode, if the RGB image and depth image are misaligned in terms of acquisition time, when displaying the focus frame based on the depth data of the depth image, the inaccurate depth data will cause the position of the focus frame A to be displayed inaccurately. Figure 3 As can be seen from the camera mode interface, when the interface is displayed, the focus frame A changes from including the face area to including part of the face area and part of the non-face area, resulting in poor image clarity and affecting the user experience.

[0076] To address the aforementioned issues, this application provides an image processing method, terminal device, chip system, computer-readable storage medium, and computer program product. During the process of acquiring RGB images using RGB and depth images using DTOF, if the acquisition time difference between the RGB and depth images is greater than or equal to a time threshold, the frame rate of the DTOF can be adjusted to ensure that the acquisition time difference between the RGB and depth images is less than the time threshold. This means the RGB and depth images are aligned in time, thereby guaranteeing the accuracy of subsequent image processing results such as autofocus calculation and bokeh effects, ensuring the accuracy of the interface display, and improving the user's camera experience.

[0077] Among them, the aforementioned terminal devices can be terminal devices with display screen hardware and corresponding software support, as well as terminal devices with camera hardware and corresponding software support.

[0078] Among them, the aforementioned terminal devices can be mobile phones, tablets, in-vehicle devices, laptops, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart cars, smart TVs, robots, and other devices.

[0079] It should be noted that in some possible implementations, the terminal device (station) may also be referred to as electronic equipment, user equipment (UE), etc., and the embodiments of this application do not limit this.

[0080] For ease of explanation, Figure 4 In the example below, terminal device 100 is a mobile phone.

[0081] like Figure 4 As shown, in some embodiments, the terminal device 100 may include a processor 101, a communication module 102, a display screen 103, a camera 104, a sensor 105, an internal memory 106, a USB interface 107, an external memory interface 108, a charging management module 109, a power management module 110, and a battery 111, etc.

[0082] The processor 101 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processor, an image signal processor (ISP), a controller, a memory, a video stream codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors 101.

[0083] The controller can serve as the nerve center and command center of the terminal device 100. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0084] The processor 101 may also include a memory for storing instructions and data.

[0085] The communication module 102 may include antenna 1 and antenna 2, a mobile communication module, and / or a wireless communication module.

[0086] The display screen 103 is used to display images or videos in the human-computer interaction interface. The display screen 103 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or N display screens 103, where N is a positive integer greater than 1.

[0087] Camera 104 is used to capture still images or videos. The camera may include a color image sensor (red-green-blue sensor, RGB) and a depth sensor (direct time-of-flight sensor, DTOF). RGB is used to acquire color images, and DTOF is used to acquire depth images. In some embodiments, terminal device 100 may include one or N cameras 104, where N is a positive integer greater than 1.

[0088] Optionally, the terminal device 100 may also include peripheral devices such as a mouse, buttons, indicator lights, keyboard, speaker, microphone, etc.

[0089] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device 100.

[0090] In other embodiments, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0091] Please refer to Figure 5 This is a schematic diagram of the software architecture of a terminal device provided in an embodiment of this application. The image processing method provided in this embodiment is applied to... Figure 4 When the terminal device 100 is shown, the software in the terminal device 100 can be divided as follows: Figure 5 The application layer 201, application framework layer (FWK) 202, hardware abstraction layer (HAL) 203, and kernel layer 204 are shown.

[0092] Multiple applications can be installed in application layer 201, such as a camera application.

[0093] The application framework layer 202 provides a set of basic functions and services for the application layer 201 to call and use.

[0094] The application framework layer 202 may include a camera service for receiving data sent by the camera application in the application layer 201, sending data to the hardware abstraction layer 203, receiving data sent by the hardware abstraction layer 203, and sending data sent by the hardware abstraction layer 203 to the camera application.

[0095] The Hardware Abstraction Layer 203 is a software located between the operating system kernel and the hardware circuitry. It is typically used to abstract the hardware to enable interaction between the operating system and the hardware circuitry at the logic layer.

[0096] The hardware abstraction layer 203 includes a camera hardware abstraction sublayer (camera hal), which includes a non-camera sensor (NCS) module. The NCS includes a non-camera sensor service (NCS Service), an image sensor service (CSS Service), and a no image sensor use case (NIS use case).

[0097] When the camera application in application layer 201 starts, it can notify the camera hardware abstraction sublayer, which can create an RGB processing pipeline (red-green-blue sensor pipeline, RGBPipeline). The RGB Pipeline includes an image front end node (IFE Node) and a sensor node 1. The IFE Node includes a statistics module.

[0098] Subsequently, the statistics module is used to send registration requests to non-image sensor instances through the unconventional camera service and image sensor service. Non-image sensor instances can create a depth pipeline based on the registration request. The depth pipeline includes sensor node 2, lightweight image frontend lite (IFE Lite) node, and depth image processing node.

[0099] The statistics module is also used to receive RGB images and to send them to sensor node 2 via unconventional camera services, image sensor services, and non-image sensor instances.

[0100] The statistics module is also used to receive depth images and perform image processing such as autofocus calculation, blurring, and fusion based on RGB images and depth images.

[0101] Sensor node 1 is used to receive RGB images sent by the camera driver and send them to the statistics module.

[0102] Sensor node 2 is used to receive depth images sent by the camera driver, and also to send them to the statistics module through the lightweight image processing front-end node, the depth image processing node, the non-image sensor instance, the image sensor service, and the unconventional camera service.

[0103] Sensor node 2 is also used to receive RGB images, determine the acquisition time difference between the RGB images and the depth images based on the timestamps carried by the RGB images and the depth images, and redetermine the data acquisition frame rate of the depth sensor when the acquisition time difference is greater than or equal to the time threshold, and send the frame rate to the camera driver.

[0104] Multiple drivers can be installed in kernel layer 204 to drive hardware operations, such as camera drivers.

[0105] The camera driver is used to receive RGB images acquired by a color image sensor (red-green-blue sensor, RGB) and depth images acquired by a depth sensor (direct time-of-flight sensor, DTOF), and to send the RGB images to sensor node 1 and the depth images to sensor node 2.

[0106] The camera driver is also used to receive the frame rate sent by sensor node 2, and to send the frame rate to DTOF.

[0107] It should be noted that the application layer 201, application framework layer 202, hardware abstraction layer 203, and kernel layer 204 may also include other content, which is not specifically limited here.

[0108] In addition, the kernel layer 204 is connected to the hardware layer 205 of the terminal device 100. The hardware layer includes a camera 104, which includes RGB and DTOF.

[0109] RGB is used to acquire RGB images; DTOF is used to acquire depth images.

[0110] Based on the above scenario description, the image processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings and application scenarios.

[0111] Please see Figure 6 , Figure 6 A schematic flowchart of an image processing method provided in an embodiment of this application is shown.

[0112] like Figure 6 As shown, the image processing method provided in this application may include:

[0113] S11, Camera application receives operation 1.

[0114] The camera application can be a system camera application or a third-party camera application; this application does not limit the type of camera application.

[0115] In some embodiments, operation 1 is a startup operation for the camera application.

[0116] Launching a camera application can be an operation triggered by clicking the camera application icon; the terminal device includes a camera application (APP), and operation 1 is an operation triggered by the user to click, double-click, or long-press the icon of the camera application.

[0117] The operation to launch the camera application can be a voice operation related to launching the camera application; the terminal device includes a camera application and a voice assistant application. After receiving a specific wake-up word from the user's voice input, the voice assistant application can be woken up. Operation 1 is the user's voice input instruction to open the camera application after the voice assistant application is woken up.

[0118] In other embodiments, portrait mode is a sub-mode of the camera application, and operation 1 is the activation operation of the portrait mode of the camera application in an environment where the brightness value is lower than the brightness threshold (low light environment).

[0119] The operation to activate the portrait mode of the camera application can be an operation that triggers the portrait mode control in the interface of the camera application; the interface of the camera application includes the portrait mode control, and operation 1 is an operation that triggers the portrait mode control (such as clicking, double-clicking or long-pressing).

[0120] The operation to launch the portrait mode of the camera application can be a voice operation related to launching the portrait mode of the camera application; Operation 1 is the user's voice input instruction to open the portrait mode after the voice assistant application is woken up.

[0121] It should be understood that the above example of operation 1 is the operation to start the camera application, or the operation to start the portrait mode of the camera application in an environment where the brightness value is lower than the brightness threshold; operation 1 can also be other operations that require the acquisition of RGB images through a color image sensor and the acquisition of depth images through a depth sensor. This application does not limit operation 1 in any way.

[0122] It should be noted that if operation 1 is a startup operation for the camera application, then upon receiving the startup operation for the camera application, steps S12 and thereafter can be executed; if operation 1 is a startup operation for the portrait mode of the camera application in an environment where the brightness value is lower than the brightness threshold, then when the camera application is in the startup state, if a user-instructed startup operation for the portrait mode of the camera application is received, then when the portrait mode is turned on, if it is detected that the brightness value of the current environment is lower than the brightness threshold, then the color image sensor and depth sensor need to be activated. In this case, steps S12 and thereafter can also be executed.

[0123] S12, The camera application sends a creation request to the camera hardware abstraction sublayer.

[0124] The creation request is used to request the camera hardware abstraction sublayer to create an RGB processing flow; the RGB processing flow includes an image processing front-end node and sensor node 1.

[0125] Specifically, the camera application sends a creation request to the camera service in the application framework layer. After receiving the creation request, the camera service can send a creation request to the camera hardware abstraction sublayer.

[0126] S13, Camera Hardware Abstraction Sublayer creates RGB processing flow.

[0127] The RGB processing flow includes an image processing front-end node and a sensor node 1.

[0128] Upon receiving the creation request, the camera hardware abstraction sublayer can create an RGB processing flow that includes an image processing front-end node and sensor node 1, facilitating subsequent operations performed through the image processing front-end node and sensor node 1.

[0129] The image processing front-end node includes a statistics module. The operations performed by the subsequent image processing front-end node can be performed by the statistics module, which will not be elaborated here.

[0130] S14. The image processing front-end node sends a registration request to the non-image sensor instance through the unconventional camera service and the image sensor service.

[0131] The registration request is used to request non-image sensor instances to send to the image processing front-end node when they receive a depth image.

[0132] Specifically, the image processing front-end node sends a creation request to the non-image sensor instance through the unconventional camera service and the image sensor service to request the creation of a depth processing flow. The creation request may carry information 1, which is used to instruct the non-image sensor instance to send to the image processing front-end node when it receives a depth image.

[0133] S15, Creating a depth processing flow for non-image sensor instances.

[0134] The depth processing flow includes sensor node 2, lightweight image processing front-end node, and depth image processing node.

[0135] Upon receiving a registration request, a non-image sensor instance can create a depth processing flow that includes sensor node 2, a lightweight image processing front-end node, and a depth image processing node.

[0136] S16, The camera application sends a startup request to the camera driver.

[0137] The startup request is used to request the camera driver to start the camera.

[0138] In some embodiments, the camera application may send a startup request to the camera driver in response to operation 1.

[0139] In other embodiments, after the non-image sensor instance completes the depth processing flow, a success notification can be returned to the camera application. Upon receiving the notification, the camera application can send a start request to the camera driver.

[0140] S17, Camera driver starts the color image sensor.

[0141] S18, Camera driver starts the depth sensor.

[0142] Upon receiving a startup request, the camera driver can start the color image sensor and the depth sensor separately.

[0143] S19, The depth sensor acquires depth data based on frame rate 1.

[0144] Once powered on, the depth sensor can begin collecting depth data at a frame rate of 1.

[0145] Frame rate 1 is preset and is usually 60fps. The depth sensor can collect 60 frames of depth data in 1 second.

[0146] Each frame of depth data is used to represent the distance from the image sensor to the object being photographed.

[0147] When a depth sensor collects depth data for each frame, it can associate the collection time with each frame of depth data.

[0148] In some embodiments, when acquiring each frame of depth data, the depth sensor can associate each frame of depth data with a timestamp related to the time of acquisition. For example, the timestamp associated with the depth data of the start of frame (SOF) is a timestamp related to the time when the depth data of the start of frame was acquired.

[0149] S20, the depth sensor sends depth data to sensor node 2 via the camera driver.

[0150] The depth sensor can send each frame of depth data it acquires to the camera driver. After receiving each frame of depth data, the camera driver can send it to sensor node 2. Each frame of depth data is associated with the acquisition time of the depth data.

[0151] S21, The color image sensor acquires RGB images based on a frame rate of 2.

[0152] After startup, the color image sensor can begin acquiring RGB images at a frame rate of 2.

[0153] Frame rate 2 is preset and is usually 30fps. The color image sensor can acquire 30 frames of RGB images (color images) in 1 second.

[0154] Each RGB image frame is used to represent the color information of the image.

[0155] When a color image sensor acquires each frame of an RGB image, it can associate the acquisition time with each frame of the RGB image.

[0156] In some embodiments, when acquiring each frame of RGB image, the color image sensor can associate each frame with a timestamp related to the time of acquisition. For example, the timestamp associated with the starting frame RGB image is the timestamp related to the time when the starting frame RGB image was acquired.

[0157] S22, The color image sensor sends RGB images to the image processing front-end node through the camera driver and sensor node 1.

[0158] The color image sensor can send each frame of the acquired RGB image to the camera driver. After receiving each frame of the RGB image, the camera driver can send it to sensor node 1. After receiving each frame of the RGB image, sensor node 1 can send it to the image processing front-end node. Each frame of the RGB image is associated with the acquisition time of the RGB image.

[0159] S23. The image processing front-end node sends RGB images to sensor node 2 through unconventional camera services, image sensor services, and non-image sensor instances.

[0160] After receiving the RGB image, the image processing front-end node can send it to the unconventional camera service. After receiving the RGB image, the unconventional camera service can send it to the image sensor service. After receiving the RGB image, the image sensor service can send it to the non-image sensor instance. After receiving the RGB image, the non-image sensor instance can send it to sensor node 2.

[0161] In some embodiments, after receiving an RGB image, the non-image sensor instance can send it to the depth image processing node, the depth image processing node can send it to the lightweight image processing front-end node, and the lightweight image processing front-end node can send it to the sensor node 2.

[0162] Based on the above description, each time the image processing front-end node receives an RGB image sent by the color image sensor, it can send the frame information (acquisition time) in the form of a tag to the non-image sensor instance. Then, the non-image sensor instance can send the RGB image and its associated acquisition time to sensor node 2.

[0163] S24. Sensor node 2 determines whether the acquisition time difference between the RGB image and the depth data is greater than or equal to threshold 1.

[0164] The acquisition time difference between the RGB image and the depth data is the time difference between the acquisition time of the RGB image and the acquisition time of the depth data.

[0165] In some embodiments, combined with Figure 7 In (a), the acquisition time of the depth data recorded by the depth sensor is the time when the first row of depth data begins to be read; the acquisition time of the RGB image recorded by the color image sensor is the time when the middle row of data is read out.

[0166] In other embodiments, combined with Figure 7 In (b), the depth data recorded by the depth sensor is acquired at the midpoint of the global exposure period of the depth data; the RGB image recorded by the color image sensor is acquired at the moment when the middle row data is read out.

[0167] It should be noted that depth sensors typically use global shutter technology, meaning that all pixels of the entire sensor are exposed at the same time, and then the data is read out line by line. Therefore, the acquisition time of depth data can be defined as the moment when the first line of data begins to be read, or it can be defined as the middle moment of the global exposure period. Color image sensors typically use rolling shutter technology, meaning that all pixels of the entire sensor are exposed and read out line by line. Therefore, the acquisition time of RGB images can be defined as the moment when the middle line of data is read out.

[0168] If the frame rate 1 of the depth sensor is 60fps and the frame rate 2 of the color image sensor is 30fps, under normal circumstances, the depth data collected by the two depth sensors corresponds to the RGB image collected by the color image sensor in one frame. The depth data collected at the time closer to the RGB image is used for autofocus calculation, and the other depth data is used to correct the autofocus calculation result.

[0169] It should be understood that the depth data used to calculate the acquisition time difference with the RGB image is the depth data of the two frames of depth data whose acquisition time is closer to the acquisition time of the RGB image.

[0170] For example, the depth data calculated from the acquisition time difference of the first frame of RGB image acquired by the color image sensor is the depth data of the first frame acquired by the depth sensor.

[0171] Threshold 1 is a preset threshold used to avoid situations where the RGB image and depth data are not aligned. The value of threshold 1 can range from 2 milliseconds (ms) to 4 ms. For example, the value of threshold 1 is 3 ms.

[0172] In some embodiments, the acquisition time of the depth data recorded by the depth sensor is the time when the first row of depth data begins to be read; the acquisition time of the RGB image recorded by the color image sensor is the time when the middle row of data is read out; determining whether the acquisition time difference between the RGB image and the depth data is greater than or equal to a threshold 1 includes: whether the time difference between the time when the first row of depth data begins to be read and the time when the middle row of RGB image is read out is greater than or equal to a threshold 1.

[0173] In other embodiments, the acquisition time of the depth data recorded by the depth sensor is the midpoint of the global exposure period of the depth data; the acquisition time of the RGB image recorded by the color image sensor is the midpoint of the data readout time; determining whether the acquisition time difference between the RGB image and the depth data is greater than or equal to a threshold 1 includes: whether the time difference between the midpoint of the global exposure period of the depth data and the midpoint of the data readout time of the RGB image is greater than or equal to a threshold 1.

[0174] If sensor node 2 determines that the acquisition time difference between the RGB image and the depth data is greater than or equal to threshold 1, sensor node 2 can determine that the RGB image and the depth data are not aligned.

[0175] For example, if the threshold value is 3ms, the acquisition time of the first frame of RGB image is 10:00:53s, and the acquisition time of the first frame of depth data is 10:00:53:003ms, then the acquisition time difference between the RGB image and the depth data is equal to the threshold value 1, and sensor node 2 can determine that the RGB image and the depth data are not aligned.

[0176] If sensor node 2 determines that the acquisition time difference between the RGB image and the depth data is less than the threshold 1, sensor node 2 can determine that the RGB image and the depth data are aligned.

[0177] For example, if the threshold value is 3ms, the acquisition time of the first frame of RGB image is 10:00:53s, and the acquisition time of the first frame of depth data is 10:00:53:002ms, then the acquisition time difference between the RGB image and the depth data is less than the threshold value 1, and sensor node 2 can determine that the RGB image and the depth data are aligned.

[0178] When sensor node 2 determines that the acquisition time difference between the RGB image and the depth data is greater than or equal to threshold 1, sensor node 2 can execute S25; when sensor node 2 determines that the acquisition time difference between the RGB image and the depth data is less than threshold 1, sensor node 2 can determine that the RGB image and the depth data are aligned and does not perform any operation.

[0179] It should be noted that when the image processing front-end node receives an RGB image, it can actively send RGB data to the sensor node 2, as described in S23. Alternatively, when the sensor node 2 receives depth data, it can obtain the RGB image from the image processing front-end node, calculate the acquisition time difference with the current frame depth data, and perform subsequent data synchronization calculations.

[0180] S25, Sensor node 2 determines the frame rate 3 of the depth sensor.

[0181] The frame rate of 3 is used to ensure that the time difference between the acquisition of RGB image and depth data is less than the threshold of 1, which means that the RGB image and depth data are aligned.

[0182] If sensor node 2 determines that the time difference between the acquisition of the RGB image and the depth data is greater than or equal to threshold 1, the frame rate of the depth sensor can be re-determined to obtain frame rate 3. Frame rate 3 is used to ensure that the depth data subsequently acquired by the depth sensor is in time aligned with the RGB image subsequently acquired by the color image sensor.

[0183] For example, sensor node 2 can set the frame rate of the depth sensor to 50fps. By lowering the frame rate of the depth sensor, the depth data subsequently acquired by the depth sensor is aligned with the RGB image subsequently acquired by the color image sensor in time.

[0184] In some embodiments, combined with Figure 7 In (a), the acquisition time of the depth data recorded by the depth sensor is the time when the first row of depth data begins to be read; the acquisition time of the RGB image recorded by the color image sensor is the time when the middle row of data is read out; the depth data and the RGB image are aligned in time, that is, the time when the first row of depth data begins to be read out and the time when the middle row of RGB image is read out are the same.

[0185] In other embodiments, combined with Figure 7 In (b), the acquisition time of the depth data recorded by the depth sensor is the midpoint of the global exposure period of the depth data; the acquisition time of the RGB image recorded by the color image sensor is the moment when the middle row data is read out; the depth data and the RGB image are aligned in time, that is, the midpoint of the global exposure period of the depth data and the moment when the middle row data of the RGB image is read out are the same moment.

[0186] In some embodiments, frame rate 3 is determined based on the time difference between the acquisition of RGB image and depth data when the frame rate of the depth sensor is 1, and the time interval between acquiring two frames of depth data at frame rate 1.

[0187] Specifically, the formula for calculating frame rate 3 can be: current frame rate of depth frame (depthCurrentFps) = NanoSecondsPerSecond / (deltaNs + m_latestPerFrameInfo.frameDuration).

[0188] Here, nanosecondsperSecond = 1,000,000,000 ns / s, which is a constant; 1 second equals 1,000,000,000 nanoseconds. deltaNs is a variable representing the time difference between the acquisition of the RGB image and the depth data, that is, the time difference between the start of frame (SOF) of the currently acquired RGB image and the SOF of the depth data, in nanoseconds (ns). m_latestPerFrameInfo.frameDuration is the frame interval of the currently acquired depth data, that is, the time interval between acquiring two frames of depth data at a frame rate of 1.

[0189] It should be noted that if the frame rate of the depth sensor is set to 3, and the time difference between the acquisition of the RGB image and the depth data is still greater than or equal to the threshold 1, sensor node 2 can determine the frame rate of the depth sensor to be 4. The frame rate 4 is determined based on the time difference between the acquisition of the RGB image and the depth data when the frame rate of the depth sensor is 3, and the time interval between acquiring two frames of depth data according to the frame rate 3.

[0190] S26, Sensor node 2 sends frame rate 3 to the depth sensor via camera driver.

[0191] After receiving frame rate 3, sensor node 2 can send frame rate 3 to the camera driver. After receiving frame rate 3, the camera driver can send frame rate 3 to the depth sensor, so that the depth sensor can adjust the acquisition frame rate of depth data according to frame rate 3.

[0192] In some embodiments, after obtaining frame rate 3, sensor node 2 can send a frame rate adjustment request to the camera driver in the kernel mode driver (KMD) of the kernel layer. The frame rate adjustment request carries frame rate 3. After receiving frame rate 3, the camera driver can set the register value related to frame rate 3 through the camera register map (CRM) of the camera driver. The camera driver can write frame rate 3 into the depth sensor according to the register value in the CRM.

[0193] S27, The depth sensor adjusts the frame rate from frame rate 1 to frame rate 3.

[0194] Here, frame rate 1 is the frame rate of the depth sensor before the current moment, which is also the default frame rate of the depth sensor.

[0195] When a frame rate of 3 is received, the depth sensor can adjust the acquisition frame rate of the depth data from frame rate 1 to frame rate 3 so that the depth sensor can acquire depth data according to frame rate 3.

[0196] For example, if frame rate 1 is 60fps and frame rate 3 is 50fps, the depth sensor can adjust the depth data acquisition frame rate from 60fps to 50fps.

[0197] S28, The depth sensor acquires depth data based on frame rate 3.

[0198] Among them, S28 and Figure 6 The implementation of S19 in the illustrated embodiment is similar and will not be repeated here.

[0199] S29. The depth sensor adjusts the frame rate from 3 to 1.

[0200] S29 is an optional step.

[0201] When the time difference between the acquisition of RGB image and depth data is less than the threshold of 1, the depth sensor can adjust the frame rate from frame rate 3 to frame rate 1, that is, restore the frame rate to the default frame rate.

[0202] After the depth sensor adjusts the frame rate from frame rate 3 to frame rate 1, the depth sensor can continue to execute S19 and subsequent steps.

[0203] Among them, S29 and Figure 6 The implementation of S27 in the illustrated embodiment is similar and will not be repeated here.

[0204] S30, sensor node 2 sends depth data to the depth image processing node through a lightweight image processing front-end node.

[0205] After S20, sensor node 2 can also execute S30.

[0206] Based on S20, after the depth sensor sends depth data to sensor node 2 via camera driver, sensor node 2 can send depth data to lightweight image processing front-end node. After receiving the depth data, the lightweight image processing front-end node can send depth data to depth image processing node.

[0207] S31. The depth image processing node sends depth images to the image processing front-end node through non-image sensor instances, image sensor services, and unconventional camera services.

[0208] After receiving the depth data, the depth image processing node can map the depth data of each pixel to a grayscale value or a color value (usually, the larger the depth value, the darker the mapped grayscale or color). The mapped values ​​are then combined into a two-dimensional image, which is the depth image.

[0209] In this way, the depth image processing node can obtain a depth image. After obtaining the depth image, the depth image processing node can send the depth image to the image processing front-end node through non-image sensor instances, image sensor services, and unconventional camera services.

[0210] S32. The image processing front-end node performs image processing based on the depth image and RGB image to obtain the target image.

[0211] Each time the image processing front-end node obtains a depth image and an RGB image, it needs to execute S32.

[0212] In some embodiments, image processing includes image fusion; the depth image processing node can fuse the RGB image and the depth image to obtain image 1, and use image 1 as the target image.

[0213] The depth image and its acquisition time, as well as the RGB image and its acquisition time, can be stored in the NCS buffer of the unconventional camera module. For SOF, the image processing front-end node can select the depth data corresponding to the SOF of the RGB image for autofocus calculation based on the autofocus function of the terminal device.

[0214] In other embodiments, image processing includes image fusion and autofocus calculation; after obtaining image 1, the depth image processing node can also determine the focus area through an autofocus algorithm, and determine the foreground and background areas based on the depth data of the depth image. In this way, the depth image processing node can optimize the focus area based on the foreground area to obtain an accurate focus area; the depth image processing node can also generate a focus frame in the focus area to obtain image 2, and use image 2 with the focus frame as the target image.

[0215] When the depth image and the RGB image are aligned, the accuracy of the focus frame position is higher than that when the depth image and the RGB image are not aligned, and the image sharpness is also higher than that when they are not aligned.

[0216] In other embodiments, if operation 1 is the activation operation of the portrait mode of the camera application in an environment where the brightness value is lower than the brightness threshold (low light environment), the image processing may also include blurring processing. After obtaining the accurate focus area, the depth image processing node may also perform blurring processing on the area outside the focus area according to the depth data of the depth image to obtain image 3. Image 3 with focus frame and blurring area is used as target image.

[0217] Among them, the accuracy of the blurred region position is higher when the depth image and the color image are aligned than when the depth image and the color image are not aligned.

[0218] In other embodiments, image processing includes image fusion and autofocus calculation; the depth image processing node can determine the focus distance based on the depth data of the RGB image and the depth image, and send the focus distance to the camera driver. The camera driver can adjust the lens of the camera based on the focus distance to ensure that the subject being photographed is clearer.

[0219] Let's take the current RGB image as the Nth frame RGB image and the depth image as the Nth frame depth image as an example:

[0220] The depth image processing node can determine the detection box (focus box) of the Nth frame that can surround the subject being photographed, and based on the RGB image, depth image, and the motion speed of the subject being photographed, it determines the prediction box (focus box) of the N+1th frame through runtime estimation and motion compensation processing. This prediction box can surround the subject being photographed in the N+1th frame.

[0221] Once the detection box of frame N and the prediction box of frame N+1 are obtained, the depth image processing node can determine the focus distance based on the detection box of frame N, the prediction box of frame N+1, and the depth image corresponding to frame N. This focus distance is the distance the camera lens moves when autofocusing on frame N+1, predicted based on the focus box of frame N. The depth image processing node can send this focus distance to the camera driver, which can then adjust the camera lens according to the focus distance to ensure that the subject is captured more clearly.

[0222] S33, the depth image processing node sends the target image to the camera application.

[0223] S34, The camera application displays interface 1 based on the target image.

[0224] After receiving the target image, the camera application can display Interface 1 in the camera application, which includes the target image.

[0225] If the target image is image 1, then interface 1 includes image 1.

[0226] If the target image is image 3 with a blurred area, the interface 1 includes image 3, in which part of the area is clear and part of the area is blurred.

[0227] Figure 8 (a) in the image is a depth image. Figure 8 (b) in the image is an RGB image. Figure 8 (c) in the diagram is a schematic of the fusion of the depth image and the RGB image. Figure 8As can be seen in (c), the color of the face region is different from that of the scenery region (background region), that is, the depth value is different. This indicates that the depth image and the RGB image are aligned in terms of acquisition time.

[0228] from Figure 9 It can be seen that if the RGB image and the depth image are aligned in terms of acquisition time, when blurring the scene area based on the depth data of the depth image, the depth data is accurate, the blurring result is accurate, the blurred area includes all background areas except for the face area, and the accuracy of the blurred area position is higher.

[0229] If the target image is image 2 with a focus frame, interface 1 includes image 2, and image 2 displays the focus frame; the focus area is displayed intuitively through the focus frame, and the user can intuitively see the subject in focus.

[0230] Combination Figure 7 The description, from Figure 10 The camera mode interface shows that if the RGB image and depth image are captured at the same time, autofocus is calculated based on the depth data from the depth image. Therefore, when the focus frame A is displayed, the depth data is accurate, the position of focus frame A is accurately displayed, and focus frame A includes the face area. Figure 3 In comparison, the image in focus frame A has higher sharpness.

[0231] In the image processing method of this application, if the acquisition time difference between the RGB image acquired by the color image sensor at frame rate 2 and the depth data acquired by the depth sensor at frame rate 1 is greater than or equal to a threshold 1, it can be determined that the RGB image and the depth data are not aligned. Then, the frame rate of the depth sensor can be re-determined, so that the depth sensor switches from acquiring depth data at frame rate 1 to acquiring depth data at frame rate 3. This ensures that the acquisition time difference between the RGB image acquired by the color image sensor at frame rate 2 and the depth data acquired by the depth sensor at frame rate 3 is less than the threshold 1, that is, the RGB image and the depth data are aligned. In this way, the aligned RGB image and depth image can be used to better perform autofocus calculation, making it easier to accurately display the position of the focus frame on the interface based on the focus calculation result. Furthermore, the camera lens can be adjusted based on the autofocus calculation result, thereby improving the accuracy of frame-by-frame tracking focus and enhancing the user experience.

[0232] Furthermore, when the camera application is in portrait mode and the current environment is low light, by aligning the RGB image and depth data in terms of acquisition time, blurring can be performed based on accurate depth data, which can improve the blurring effect of portrait mode.

[0233] Based on the foregoing embodiments, the image processing method provided in this application is described below.

[0234] For example, this application provides an image processing method.

[0235] For ease of explanation, the image processing method of this application is applied to a terminal device, which includes a camera, a color image sensor and a depth sensor, and a camera application running on the terminal device; the image processing method of this application can be executed by the terminal device 100.

[0236] Please see Figure 11 , Figure 11 A schematic flowchart of an image processing method provided in an embodiment of this application is shown.

[0237] like Figure 11 As shown, the image processing method provided in this application may include:

[0238] S301, in response to the first operation, control the depth sensor to acquire a depth image according to a first frame rate, and control the color image sensor to acquire a color image according to a second frame rate, the first operation including the startup operation of the camera application, the color image being an RGB image in the red-green-blue color space.

[0239] For details on the implementation of the first operation, please refer to [link / reference]. Figure 6 The relevant description of operation 1 in S11.

[0240] For details on how the first frame rate is implemented, please refer to [link / reference]. Figure 6 The description of frame rate 1 in S19; for the specific implementation of depth images, please refer to Figure 6 Description of medium-depth images.

[0241] For details on how the second frame rate is implemented, please refer to [link / reference]. Figure 6 The description of frame rate 2 in S21; for the specific implementation of color images, please refer to Figure 6 The relevant description of RGB images in S21.

[0242] For details on the implementation of S301, please refer to [link / reference]. Figure 6 The relevant descriptions of S11-S23 in the document will not be elaborated here.

[0243] S302. When the acquisition time of the depth image based on the first frame rate is not aligned with the acquisition time of the color image based on the second frame rate, the frame rate of the depth sensor is adjusted from the first frame rate to the third frame rate so that the depth sensor acquires the depth image based on the third frame rate; wherein, the acquisition time of the depth image based on the third frame rate is aligned with the acquisition time of the color image based on the second frame rate.

[0244] For details on the implementation of the third frame rate, please refer to [link / reference]. Figure 6 The relevant description of frame rate 3 in S25.

[0245] For details on the implementation of S302, please refer to [link / reference]. Figure 6 The relevant descriptions of S24-S28 in the document will not be elaborated here.

[0246] In the image processing method of this application, if the acquisition time of the depth image acquired according to the first frame rate is not aligned with the acquisition time of the color image acquired according to the second frame rate, the frame rate of the depth sensor can be re-determined so that the depth sensor switches from acquiring the depth image at the first frame rate to acquiring the depth image at the third frame rate. This ensures that the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate. In this way, subsequent image processing can be better performed with the color image and depth image in an aligned state, which can improve the user experience.

[0247] In some embodiments, the first operation further includes: activating the portrait mode of the camera application in an environment where the brightness value is below a brightness threshold.

[0248] In this application, when the camera application is in portrait mode and the current environment is a low-light environment, by adjusting the frame rate of the depth sensor, the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate. This allows for bokeh processing based on accurate depth image depth data, thereby improving the bokeh effect in portrait mode.

[0249] In some embodiments, the image processing method provided in this application may include:

[0250] Determine whether the first difference between the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate is greater than or equal to a time threshold; if the first difference is greater than or equal to the time threshold, determine that the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate are not aligned; adjust the frame rate of the depth sensor from the first frame rate to the third frame rate.

[0251] In this application, the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate can be determined by the difference between the acquisition time of the depth image and the color image being greater than or equal to a preset threshold.

[0252] In some embodiments, the acquisition time of the depth image is the time when the first row of data of the depth image begins to be read; the acquisition time of the color image is the time when the middle row of data of the color image is read out.

[0253] The acquisition time of the depth image can be found in [reference needed]. Figure 6 The description of the depth image acquisition time in S24 is provided; the acquisition time for the color image can be found in [reference needed]. Figure 6 The description in S24 regarding the acquisition time of the RGB image.

[0254] In some embodiments, the image processing method provided in this application may include:

[0255] Determine whether the first difference between the time when the first row of data in the depth image begins to be read and the time when the middle row of data in the color image is read is greater than or equal to a time threshold.

[0256] In this application, the terminal device can determine that the acquisition time of the depth image acquired at the first frame rate and the acquisition time of the color image acquired at the second frame rate are misaligned by determining that the first difference between the time when the first row of data of the depth image is read and the time when the middle row of data of the color image is read is greater than or equal to a time threshold; and determine that the acquisition time of the depth image acquired at the first frame rate and the acquisition time of the color image acquired at the second frame rate are aligned by determining that the first difference is less than the time threshold.

[0257] In some embodiments, the depth image is acquired at the midpoint of the global exposure period of the depth image; the color image is acquired at the moment when the middle row of data of the color image is read out.

[0258] The acquisition time of the depth image can be found in [reference needed]. Figure 6 The description of the depth image acquisition time in S24 is provided; the acquisition time for the color image can be found in [reference needed]. Figure 6 The description in S24 regarding the acquisition time of the RGB image.

[0259] In some embodiments, the image processing method provided in this application may include:

[0260] Determine whether the first difference between the midpoint of the global exposure time of the depth image and the midpoint readout time of the color image is greater than or equal to a time threshold.

[0261] In this application, the terminal device can determine that the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate are misaligned by determining that the difference between the acquisition time of the depth image and the midpoint of the global exposure period of the depth image and the midpoint of the color image data readout time is greater than or equal to a time threshold; and determine that the acquisition time of the depth image acquired according to the first frame rate and the acquisition time of the color image acquired according to the second frame rate are aligned by determining that the first difference is less than the time threshold.

[0262] In some embodiments, the third frame rate is determined based on the first difference and the first frame interval, where the first frame interval is the time interval between acquiring two depth images based on the first frame rate.

[0263] The method for determining the third frame rate can be found in [reference needed]. Figure 6 The relevant description of how frame rate 3 is determined in S25 is not elaborated here.

[0264] In some embodiments, the image processing method provided in this application may include:

[0265] When the acquisition time of the depth image based on the third frame rate is aligned with the acquisition time of the color image based on the second frame rate, the frame rate of the depth sensor is adjusted from the third frame rate to the first frame rate so that the depth sensor continues to acquire depth images based on the first frame rate.

[0266] For details on how to implement the above steps, please refer to [link / reference]. Figure 6 The relevant descriptions of S29 in the document will not be elaborated here.

[0267] In some embodiments, the image processing method provided in this application may include:

[0268] Image processing is performed on the depth image and color image to obtain the target image. Image processing includes autofocus processing and blurring processing. The target image includes a focus frame and a blurred area. When the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate, the accuracy of the focus frame position is higher than that when they are not aligned. When the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate, the image sharpness of the focus frame position is higher than that when they are not aligned. When the acquisition time of the depth image acquired according to the third frame rate is aligned with the acquisition time of the color image acquired according to the second frame rate, the accuracy of the blurred area position is higher than that when they are not aligned. A first interface is displayed, which includes the target image.

[0269] For details on how to implement the above steps, please refer to [link / reference]. Figure 6 The relevant descriptions of S32-S34 in the document will not be elaborated here.

[0270] In some embodiments, the image processing method provided in this application may include:

[0271] The depth image and color image are automatically focused to obtain the focus distance. The accuracy of the focus distance is higher when the acquisition time of the depth image based on the third frame rate is aligned with the acquisition time of the color image based on the second frame rate than when they are not aligned. Based on the focus distance, the lens position of the camera is adjusted. The sharpness of the focus area of ​​the color image after adjustment is higher than that of the focus area of ​​the color image before adjustment.

[0272] For details on how to implement the above steps, please refer to [link / reference]. Figure 6 The relevant descriptions regarding the adjustment of the camera lens in S32 are not elaborated here.

[0273] For example, this application provides a terminal device including a processor; when the processor executes computer code or instructions in memory, the terminal device performs the image processing method described in the preceding embodiments.

[0274] For example, this application provides a terminal device, including: a memory and a processor; the memory is coupled to the processor, and the memory is used to store program code or instructions; the processor is used to call the program code or instructions in the memory to cause the terminal device to execute the image processing method in the preceding embodiments.

[0275] For example, this application provides a chip system applied to a terminal device including a memory, a display screen, and sensors; the chip system includes: one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the interface circuits are used to receive signals from the memory and send signals to the processors, the signals including computer code or instructions stored in the memory; when the processor executes the computer code or instructions, the terminal device executes the image processing method in the preceding embodiments.

[0276] For example, this application provides a computer-readable storage medium storing code or instructions that, when executed on a terminal device, cause the terminal device to implement the image processing method described in the preceding embodiments.

[0277] For example, this application provides a computer program product that, when run on a computer, enables a terminal device to implement the image processing method described in the preceding embodiments.

[0278] In the above embodiments, all or part of the functionality can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented wholly or partially in the form of a computer program product. A computer program product includes one or more computer codes or instructions. When the computer program code or instructions are loaded and executed on a computer, all or part of the flow or functionality according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer code or instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0279] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image processing method, characterized by, The method is applied to a terminal device, the terminal device comprising a camera, the camera comprising a color image sensor and a depth sensor, and a camera application running on the terminal device, the method comprising: in response to a first operation, controlling the depth sensor to capture a depth image according to a first frame rate, and controlling the color image sensor to capture a color image according to a second frame rate, the first operation comprising a start operation of the camera application, the color image being an image in a red-green-blue color space RGB; when the capturing time of the depth image according to the first frame rate and the capturing time of the color image according to the second frame rate are not aligned, adjusting the frame rate of the depth sensor from the first frame rate to a third frame rate, so that the depth sensor captures the depth image according to the third frame rate; wherein the capturing time of the depth image according to the third frame rate and the capturing time of the color image according to the second frame rate are aligned.

2. The method of claim 1, wherein, The first operation further comprises: in an environment where the luminance value is lower than a luminance threshold, a start operation of a portrait mode of the camera application.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: determining whether a first difference between the capturing time of the depth image according to the first frame rate and the capturing time of the color image according to the second frame rate is greater than or equal to a time threshold value; when the first difference is greater than or equal to the time threshold value, determining that the capturing time of the depth image according to the first frame rate and the capturing time of the color image according to the second frame rate are not aligned; adjusting the frame rate of the depth sensor from the first frame rate to the third frame rate.

4. The method of claim 3, wherein, The capturing time of the depth image is the time when the first row data of the depth image starts to be read; and the capturing time of the color image is the time when the middle row data of the color image is read out.

5. The method of claim 4, wherein, The method further comprises: determining whether a first difference between the capturing time of the depth image according to the first frame rate and the capturing time of the color image according to the second frame rate is greater than or equal to a time threshold value; 6. The method of claim 3, wherein, when the first difference is greater than or equal to the time threshold value, determining that the capturing time of the depth image according to the first frame rate and the capturing time of the color image according to the second frame rate are not aligned; 7. The method of claim 6, wherein, adjusting the frame rate of the depth sensor from the first frame rate to the third frame rate. The capturing time of the depth image is the middle time of a global exposure period of the depth image; and the capturing time of the color image is the time when the middle row data of the color image is read out.

8. The method according to any one of claims 3 to 7, characterized in that, The method further comprises: determining whether a first difference between the capturing time of the depth image according to the first frame rate and the capturing time of the color image according to the second frame rate is greater than or equal to a time threshold value; when the first difference is greater than or equal to the time threshold value, determining that the capturing time of the depth image according to the first frame rate and the capturing time of the color image according to the second frame rate are not aligned; adjusting the frame rate of the depth sensor from the first frame rate to the third frame rate. When the first difference is less than the time threshold, it is determined that the acquisition time of the depth image according to the first frame rate is aligned with the acquisition time of the color image according to the second frame rate.

9. The method according to any one of claims 3 to 8, characterized in that, The third frame rate is determined according to the first difference and the first frame interval, and the first frame interval is a time interval for acquiring two frames of the depth image according to the first frame rate.

10. The method according to any one of claims 1 to 9, characterized in that, When the acquisition time of the depth image according to the first frame rate is not aligned with the acquisition time of the color image according to the second frame rate, the frame rate of the depth sensor is adjusted from the first frame rate to a third frame rate, so that after the depth sensor acquires the depth image according to the third frame rate, the method further comprises: When the acquisition time of the depth image according to the third frame rate is aligned with the acquisition time of the color image according to the second frame rate, the frame rate of the depth sensor is adjusted from the third frame rate to the first frame rate, so that the depth sensor continues to acquire the depth image according to the first frame rate.

11. The method according to any one of claims 1 to 10, characterized in that, When the acquisition time of the depth image according to the first frame rate is not aligned with the acquisition time of the color image according to the second frame rate, the frame rate of the depth sensor is adjusted from the first frame rate to a third frame rate, so that after the depth sensor acquires the depth image according to the third frame rate, the method further comprises: The depth image and the color image are image processed to obtain a target image, the image processing includes automatic focusing processing and / or blurring processing, the target image includes a focusing frame and / or a blurring area, the accuracy of the focusing frame position when the acquisition time of the depth image according to the third frame rate is aligned with the acquisition time of the color image according to the second frame rate is higher than that when the acquisition time of the depth image according to the third frame rate is not aligned with the acquisition time of the color image according to the second frame rate, the image definition of the focusing frame position when the acquisition time of the depth image according to the third frame rate is aligned with the acquisition time of the color image according to the second frame rate is higher than that when the acquisition time of the depth image according to the third frame rate is not aligned with the acquisition time of the color image according to the second frame rate, and the accuracy of the blurring area position when the acquisition time of the depth image according to the third frame rate is aligned with the acquisition time of the color image according to the second frame rate is higher than that when the acquisition time of the depth image according to the third frame rate is not aligned with the acquisition time of the color image according to the second frame rate; A first interface is displayed, and the first interface includes the target image.

12. The method of claim 11, wherein, The method further comprises: The depth image and the color image are automatically focused to obtain a focusing distance, and the accuracy of the focusing distance when the acquisition time of the depth image according to the third frame rate is aligned with the acquisition time of the color image according to the second frame rate is higher than that when the acquisition time of the depth image according to the third frame rate is not aligned with the acquisition time of the color image according to the second frame rate; According to the focusing distance, the lens position of the camera is adjusted, and the definition of the focusing area of the color image after adjustment is higher than that of the focusing area of the color image before adjustment.

13. A terminal device, comprising: The terminal device comprises one or more processors and a memory; the memory is coupled with the one or more processors, the memory is used to store computer program codes, the computer program codes comprise computer instructions, and the one or more processors invoke the computer instructions to enable the terminal device to perform the method in any one of claims 1 to 12.

14. A chip system, characterized by The chip system is applied to a terminal device, and the chip system comprises one or more processors configured to invoke computer instructions to enable the terminal device to perform the method in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises instructions which, when executed on a terminal device, enable the terminal device to perform the method in any one of claims 1 to 12.

Images