Image acquisition methods and image acquisition equipment

Abstract

An image acquisition method and device are provided. The method includes: periodically arranging time periods for acquiring color images; and within the time periods for acquiring color images, arranging time periods for acquiring depth images and infrared images, wherein the time periods for acquiring depth images and infrared images do not overlap, and the total duration of the time periods for acquiring depth images and infrared images is less than the total duration of the time periods for acquiring color images. This method shortens the time interval between capturing infrared and depth images, thereby improving the recognition efficiency and accuracy of face recognition technology.

Description

Technical Field

[0001] This disclosure relates to the field of cloud technology, and more specifically, to an image acquisition method, device, and computer-readable storage medium. Background Technology

[0002] Due to its reliability and user-friendliness, image-based facial recognition technology has been widely used in personal identification systems. However, significant security issues remain, posing security risks to systems that utilize facial recognition, such as payment systems.

[0003] For example, systems using facial recognition technology need to determine whether the object to be identified is a real person / living object, a photo or video, or a head model, etc.

[0004] Commonly used image types for facial recognition technology include color images, infrared images, and depth images. In the facial recognition process, it is typically required to capture images of the same object within the shortest possible time interval to achieve the highest possible recognition accuracy within the shortest possible image acquisition time. However, since the light source used to capture depth images and the light source used to capture infrared images are usually the same, simultaneously acquiring depth and infrared images can cause interference to both. Therefore, time-division multiplexing of depth and infrared image acquisition is necessary. For this purpose, as... Figure 2 As shown, the traditional arrangement of time slots for acquiring color, depth, and infrared images results in a one-frame interval between the depth and infrared image acquisition sessions for the color image. Therefore, for time alignment, either the color and depth images must be time-aligned (i.e., color and depth image acquisition occur within the same color image acquisition time frame), or the color and infrared images must be time-aligned (i.e., color and infrared image acquisition occur within the same color image acquisition time frame). If the color and depth images are time-aligned, there will inevitably be a one-frame interval between them and the infrared image acquisition, typically 40ms. Similarly, if the color and infrared images are time-aligned, there will also be a one-frame interval between them and the depth image acquisition, typically 40ms. Consequently, if the face to be detected cannot remain completely still, one of the three acquired images will inevitably show a difference between the face to be detected in the other two images, making face recognition difficult and reducing the efficiency and accuracy of face recognition technology. Summary of the Invention

[0005] To address the aforementioned problems, this disclosure provides an image acquisition method, an image acquisition device, and a computer-readable storage medium.

[0006] The embodiments of this disclosure provide an image acquisition method, including: periodically arranging time periods for acquiring color images; and arranging time periods for acquiring depth images and infrared images during the time periods for acquiring color images, wherein the time periods for acquiring depth images and the time periods for acquiring infrared images do not overlap, and the total duration of the time periods for acquiring depth images and the time periods for acquiring infrared images is less than the total duration of the time periods for acquiring color images.

[0007] Embodiments of this disclosure provide an image acquisition device, including a color sensor and an infrared sensor, wherein the color sensor is configured to periodically arrange time periods for acquiring color images; the infrared sensor is configured to arrange time periods for acquiring depth images and time periods for acquiring infrared images during the time periods for acquiring color images; wherein the time periods for acquiring depth images and the time periods for acquiring infrared images do not overlap, and the total duration of the time periods for acquiring depth images and the time periods for acquiring infrared images is less than the total duration of the time periods for acquiring color images.

[0008] The color sensor is further configured to send a frame synchronization signal to the infrared sensor, the frame synchronization signal indicating the start time of the time period for acquiring the color image. The infrared sensor utilizes the frame synchronization signal sent by the color sensor to ensure that both the time period for acquiring the depth map and the time period for acquiring the infrared image fall within the time period for acquiring the color image.

[0009] The infrared sensor is further configured to: receive a frame synchronization signal sent by the color sensor; and generate a first trigger frame and a second trigger frame based on the frame synchronization signal. The first trigger frame is used to schedule a time period for acquiring a depth map, and the second trigger frame is used to schedule a time period for acquiring an infrared image. The time interval between the first trigger frame and the second trigger frame is greater than or equal to the processing time for acquiring the depth map, or the time interval between the first trigger frame and the second trigger frame is greater than or equal to the processing time for acquiring the infrared image.

[0010] The number of time periods used to acquire color images is the first number, the number of time periods used to acquire depth images is the second number, and the number of time periods used to acquire infrared images is the third number. The second number is equal to the third number, and the sum of the second number and the third number is less than or equal to the first number.

[0011] The color sensor uses a frame synchronization clock to generate a frame synchronization signal, and the infrared sensor uses a pixel clock to generate a first trigger frame and a second trigger frame. The frame rate of the pixel clock is more than three times the frame rate of the frame synchronization clock.

[0012] The image acquisition device further includes a power management circuit, a speckle infrared light source, and a diffuse infrared light source. The infrared sensor is configured to: transmit a first strobe signal to the power management circuit based on a first trigger frame, and / or transmit a second strobe signal to the power management circuit based on a second trigger frame. The power management circuit is further configured to: control the speckle infrared light source to emit speckle infrared light during the exposure period of the time period used for acquiring the depth map based on the first strobe signal, and / or control the diffuse infrared light source to emit diffuse infrared light during the exposure period of the time period used for acquiring the infrared image based on the second strobe signal.

[0013] The infrared sensor is further configured to: acquire the depth map during the exposure period of the time period for acquiring the depth map, and store the depth map in a memory during the data readout period of the time period for acquiring the depth map; and / or acquire the infrared image during the exposure period of the time period for acquiring the infrared image, and store the infrared image in a memory during the data readout period of the time period for acquiring the infrared image.

[0014] Specifically, the color sensor acquires the color image using a line-by-line rolling exposure method during the time period for acquiring the color image; the infrared sensor acquires the depth image using a global exposure method during the time period for acquiring the depth image; and the infrared sensor acquires the infrared image using a global exposure method during the time period for acquiring the infrared image.

[0015] Specifically, during the period for acquiring color images, the color sensor acquires a color image corresponding to the object to be detected; during the period for acquiring depth images, the infrared sensor acquires a depth image corresponding to the object to be detected; during the period for acquiring infrared images, the infrared sensor acquires an infrared image corresponding to the object to be detected; and based on the acquired color image, depth image, and infrared image, the image acquisition device determines the identity of the object to be detected.

[0016] The process of identifying the object to be detected further includes: determining the coordinate transformation relationship between the color image and the infrared image, and between the color image and the depth image, based on the acquired color image, infrared image, and depth image; calibrating the infrared image and the depth image based on the coordinate transformation relationship between the color image and the infrared image, and between the color image and the depth image; and determining the identity of the object to be detected based on the color image and the calibrated infrared image and depth image.

[0017] Embodiments of this disclosure also provide a computer-readable storage medium having instructions stored thereon that, when executed by a processor, are used to implement the method described above.

[0018] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of this disclosure.

[0019] Based on the image acquisition method, image acquisition device, and computer-readable storage medium provided in the embodiments of this disclosure, the time interval between capturing infrared images and depth images is shortened by adjusting the way the face recognition system drives the light source and the frame rate and frame synchronization method of triggering the infrared sensor, thereby improving the recognition efficiency and recognition accuracy of face recognition technology. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1 A terminal device including the above-described image acquisition device is shown according to an embodiment of this disclosure.

[0022] Figure 2 This illustrates the traditional arrangement of time periods for acquiring color images, depth images, and infrared images.

[0023] Figure 3 A flowchart of an image acquisition method according to an embodiment of the present disclosure is shown.

[0024] Figure 4 A structural diagram of an image acquisition device according to an embodiment of the present disclosure is shown.

[0025] Figure 5 A comparison diagram is shown between the arrangement of time periods for acquiring color images, depth images, and infrared images according to embodiments of the present disclosure and the conventional arrangement of time periods for acquiring color images, depth images, and infrared images.

[0026] Figure 6 Another structural diagram of an image acquisition device according to an embodiment of the present disclosure is shown.

[0027] Figure 7 A flowchart illustrating spatial alignment of a color image, a depth image, and an infrared image according to an embodiment of the present disclosure is shown.

[0028] Figure 8 A schematic diagram of the registration process according to an embodiment of the present disclosure is shown.

[0029] Figure 9 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0030] Figure 10 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown.

[0031] Figure 11 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0033] In this specification and accompanying drawings, substantially the same or similar steps and elements are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0034] The facial recognition technology mentioned in the embodiments of this disclosure can be specifically implemented by training a machine learning model with corresponding functions based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions.

[0035] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] Optionally, facial recognition technology also involves computer vision (CV). Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data.

[0037] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0038] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0039] The embodiments of this disclosure relate to technologies such as computer vision in artificial intelligence. For ease of understanding, some basic concepts related to the embodiments of this disclosure are first introduced below.

[0040] A camera module (also simply called a camera) is a system comprised of devices capable of facial recognition. Camera modules can also be used to recognize QR codes, and can be positioned either in the front or rear of a facial recognition device. Cameras can be monocular, binocular, RGB-D (Red-Green-Blue-Deep), or 3D (3-Dimensional) cameras such as 3D structured light cameras. A camera module consists of a lens, an optical sensor, and a sensor processor. The sensor processor can be a digital signal processing (DSP) chip, such as an image signal processing (ISP). The lens is the device that projects the image onto the sensor; it can consist of several lenses, which can be plastic or glass lenses, etc. The digital signal processing chip optimizes the digital image signal through a series of complex mathematical algorithms. It should be noted that, in the embodiments according to this disclosure, the digital signal processing chip may or may not be integrated into the camera. For example, in the embodiments according to this disclosure, the camera may only include multiple optical sensors, while the digital signal processing chip is integrated into the terminal's processor; for example, in some other embodiments according to this disclosure, the camera may only include multiple optical sensors, while the digital signal processing chip exists independently of the camera and the terminal processor.

[0041] Three-image alignment: Face recognition technology requires color images, depth images, and infrared images to be as close as possible in time and space, with the same resolution, the same field of view (FOV), and the pixels in the three images to be aligned one by one.

[0042] The optimization process involves selecting a set of color images, depth images, and infrared images that meet the prerequisites for liveness detection and comparison recognition algorithms. Optionally, the optimization process optimizes the color images based on factors such as face angle, face size, face centering, and color image clarity. The optimization process also optimizes the infrared images based on their brightness. Optionally, the optimization process optimizes the depth images based on their completeness.

[0043] Preferred frames: A set of color images, depth images, and infrared images selected through a preferred process. These can be used for liveness detection and comparative identification.

[0044] Liveness detection: A method to determine whether the person using facial recognition is a real person, a photo, or a head model. Generally, depth maps are used to determine if it is a photo, and the brightness of infrared images is used to determine if it is a silicone head model.

[0045] Comparison and Recognition: The facial recognition system identifies which user is being scanned. This is typically achieved by extracting five key similarities from a color image and then using a depth image to further compare the three-dimensional similarities of these five key similarities.

[0046] According to embodiments of this disclosure, an image acquisition method is proposed to shorten the time interval between capturing infrared and depth images. The method includes: periodically arranging time periods for capturing color images; and arranging time periods for capturing depth and infrared images during the color image acquisition time periods, wherein the depth and infrared image acquisition time periods do not overlap, and the total duration of the depth and infrared image acquisition time periods is less than the total duration of the color image acquisition time period. Embodiments of this disclosure shorten the time interval between capturing infrared and depth images by adjusting the way the face recognition system drives the light source and the frame rate and frame synchronization method of triggering the infrared sensor, thereby improving the recognition efficiency and accuracy of face recognition technology.

[0047] According to embodiments of this disclosure, an image acquisition device is also proposed, including a color sensor and an infrared sensor, wherein the color sensor is configured to periodically arrange time periods for acquiring color images; the infrared sensor is configured to arrange time periods for acquiring depth images and time periods for acquiring infrared images during the time periods for acquiring color images; wherein the time periods for acquiring depth images and the time periods for acquiring infrared images do not overlap, and the total duration of the time periods for acquiring depth images and the time periods for acquiring infrared images is less than the total duration of the time periods for acquiring color images.

[0048] The following combination Figures 1 to 11 The image acquisition method and image acquisition device according to embodiments of the present disclosure will be described in detail to improve the recognition efficiency and accuracy of face recognition technology.

[0049] Figure 1 A terminal device including the above-described image acquisition device is shown according to an embodiment of this disclosure.

[0050] The image acquisition device can be integrated into a terminal with a camera, such as a mobile phone, facial recognition payment device, tablet computer, monitoring equipment, self-service checkout machine, smart Bluetooth device, laptop computer, or personal computer (PC). In some embodiments, the image acquisition device can also be integrated into multiple terminals; for example, the image acquisition device can be integrated into multiple security inspection devices, with multiple security inspection devices implementing the image acquisition method of the present invention.

[0051] Optionally, the image acquisition device can also be integrated into the camera module. The camera can have built-in functional components such as sensors, sensor processors, and memory. The built-in memory of the camera module can store the image acquisition method provided in the embodiments of the present invention. The sensor processor can read from the memory and control the sensor to execute the image acquisition method provided in the embodiments of the present invention.

[0052] The following is Figure 1 The facial recognition payment device shown is used as an example for illustration. This device may include a sensor module, which may include depth sensors, temperature sensors, color sensors, gravity sensors, accelerometers, gyroscopes, fingerprint sensors, position sensors, etc., as well as a sensor processor for controlling these sensors, etc. Those skilled in the art should understand that this disclosure is not limited thereto.

[0053] For example, the terminal could be a plug-and-play facial recognition payment device for manual checkout counters, supporting both QR code payment and facial recognition payment simultaneously. For instance, when receiving a QR code payment instruction, the facial recognition payment device can capture a QR code image using its built-in camera, decode and recognize the QR code image, and process the payment based on the decoding result. Conversely, when receiving a facial recognition payment instruction, the device can capture a color image, depth image, and infrared image of the user to be paid using its built-in camera. These images are then processed using algorithms to align them. The device can then perform facial recognition based on the alignment of the three images to determine the user's identity. Once the identity is verified, the facial recognition payment device can proceed with the subsequent payment processing.

[0054] For example, facial recognition payment devices can identify facial feature points in a color image to obtain facial location information. For instance, a facial recognition payment device can identify the contours of a face, hairstyle, ears, eyes, nose, mouth, and other facial features, and then determine the positional information between these features, such as the distance between the eyes, the angle between the nose and eyes, or the distance between the mouth and nose. Alternatively, computer devices can extract facial feature maps from a color image using machine learning models, identify individual facial features based on these feature maps, and then determine the coordinates of each feature individually or relative to each other. Specifically, the machine learning model can be an R-CNN model (Region-Convolutional Neural Networks, a type of object detection model) or a YOLO model (You Only Look Once, an object detection model), etc., and this application does not limit the specific implementation of these models.

[0055] Figure 3A flowchart of an image acquisition method 300 according to an embodiment of the present disclosure is shown. The image acquisition method 300 can be performed by an image acquisition device 400. Figure 4 A structural diagram of an image acquisition device 400 according to an embodiment of the present disclosure is shown. The image acquisition device 400 includes a color sensor 401 and an infrared sensor 402.

[0056] Image acquisition method 300 is suitable for applications such as... Figure 1 Image acquisition is performed in the terminal shown. Optionally, after receiving an image acquisition start command, the image acquisition device 400 begins executing the image acquisition method 300. The image acquisition start command is a command triggered by the user or the terminal, used to instruct the image acquisition device 400 to perform image acquisition. The image acquisition start command may include multiple operation codes specifying the type of operation or parameters to be performed by the command, as well as address codes specifying the content of the operation object or the address of its storage unit, etc. Of course, in some special scenarios, the image acquisition device 400 may also start without receiving an image acquisition start command. For example, when the image acquisition device 400 is used as part of a monitoring device to monitor in real time whether a criminal appears in a certain area, the image acquisition device 400 may repeatedly execute the image acquisition method 300 without any image acquisition start command. Those skilled in the art should understand that this disclosure is not limited thereto.

[0057] The image acquisition method 300 may include the following two steps S301 and S302.

[0058] In step S301, time periods for acquiring color images are periodically arranged.

[0059] Optionally, the color sensor 401 can be periodically arranged to acquire color images over time. Optionally, the color sensor can be a sensor within a camera module used to sense color and acquire color information. For example, the color sensor may include a sensor processor. The sensor processor is a core component that reads, decodes, and executes acquisition commands, and has functions such as controlling the sensor to acquire data, controlling sensor acquisition parameters, and performing image processing. The color sensor 401 and the sensor processor can use an internal clock to instruct each photosensitive element in the color sensor to perform exposure, data storage, and data reading processes.

[0060] Optionally, a color image refers to data captured during the natural light imaging process by a color sensor (such as the aforementioned ordinary camera). Color images are generally used in the selection and comparison recognition processes in face recognition systems. A color image is an image located in a color space, which can present different colors based on different pixel values. Specifically, a color image can be an image corresponding to multiple channels, such as an RGB image (R represents Red; G represents Green; B represents Blue), a CMYK image (C represents Cyan; M represents Magenta; Y represents Yellow; K represents Black), or a YUV image (Y represents Luminance; U and Y represent Chrominance), etc. Those skilled in the art should understand that this disclosure is not limited thereto.

[0061] Optionally, the color sensor 401 acquires the color image using a rolling shutter method during the time period for acquiring the color image. During the rolling shutter process, the color sensor 401 can first expose the first row of photosensitive elements, then begin exposing the second row only after the first row's exposure time has ended, and so on until all rows of photosensitive elements have been exposed. This exposure method allows the acquired color image to include more information, and the exposure of adjacent rows does not interfere with each other.

[0062] Optionally, the time period for acquiring the color image includes the total exposure time of all photosensitive elements. Optionally, the time period for acquiring the color image also includes the time required for the photosensitive elements to perform data storage and data retrieval processes. That is, the time period for acquiring the color image can begin at the moment when the first row of photosensitive elements begins exposure and end at the moment when all data in the last row of photosensitive elements has been read out. As another embodiment, the time period for acquiring the color image can begin at the moment when the first row of photosensitive elements begins exposure and end at the moment when all data of the color image has been processed. This disclosure is not limited thereto.

[0063] In step S302, during the time period for acquiring color images, time periods for acquiring depth images and time periods for acquiring infrared images are arranged, wherein the time periods for acquiring depth images and time periods for acquiring infrared images do not overlap, and the total duration of the time periods for acquiring depth images and time periods for acquiring infrared images is less than the total duration of the time period for acquiring color images.

[0064] Optionally, the infrared sensor 402 can be configured to acquire depth images during specific time periods and infrared images during specific time periods. Optionally, the infrared sensor can be a sensor in a camera module that acquires data using infrared light (including speckle infrared light and ambient infrared light). In some embodiments, the infrared sensor may also include a sensor processor. The sensor processor is a core component that reads, decodes, and executes acquisition commands, and has functions such as controlling the sensor to acquire data, controlling sensor acquisition parameters, and performing image processing.

[0065] Optionally, the sensor processor in the infrared sensor 402 can be composed of a CPU and ISP built into the image acquisition device 400. The infrared sensor 402 can act as a slave computer, receiving instructions from a master computer via an I / O interface to acquire images with different image parameters and send these images to the master computer. As an example, the master computer can be the color sensor 401. Optionally, the infrared sensor 402 utilizes a frame synchronization signal sent by the color sensor 401 to ensure that the time period for acquiring the depth map and the time period for acquiring the infrared image both fall within the time period for acquiring the color image.

[0066] Optionally, the infrared sensor 402 acquires depth maps using a global shutter method during the depth map acquisition period. Similarly, the infrared sensor 402 acquires infrared images using a global shutter method during the infrared image acquisition period. During the global shutter process, all photosensitive elements in the infrared sensor 402 are driven simultaneously. That is, all photosensitive elements begin and end exposure simultaneously during the depth map or infrared image imaging process. This results in higher global shutter efficiency and avoids distortion caused by movement of the subject's face. Optionally, although all photosensitive elements in the infrared sensor 402 perform exposure operations simultaneously, the data stored in these photosensitive elements can be read out all at once, sequentially row by row, or sequentially in batches later.

[0067] Optionally, the time period for acquiring the depth map can begin at the moment when all photosensitive elements in the infrared sensor 402 begin exposure and end at the moment when all data from all photosensitive elements have been read out. As another embodiment, the time period for acquiring the depth map can begin at the moment when all photosensitive elements in the infrared sensor 402 begin exposure and end at the moment when all data from the dark image has been processed. This disclosure is not limited thereto. The time period for acquiring the infrared image is similar and will not be described again here.

[0068] Optionally, a depth image, also known as a range image, is an image that uses the distance (depth) from the depth sensor to various points in the scene as pixel values. This depth image can directly reflect the geometry of the visible surfaces of objects. A speckle infrared light emitter can emit multiple speckle spots. These speckle spots are projected onto the face to be detected, and the face reflects the spots. The infrared sensor collects the reflected infrared light from the speckle structure, and the depth unit analyzes the speckle to obtain the depth image. In 3D computer graphics and computer vision, a depth image can also indicate an image or image channel containing information related to the distance from the surface of a scene object to the viewpoint. Each pixel in the depth image represents the vertical distance between the depth camera plane and the plane of the object being photographed, typically represented by 16 bits in millimeters. Those skilled in the art should understand that this disclosure is not limited thereto. In face recognition technology, depth images can optionally be used in the liveness detection process and / or to assist in the comparison and recognition process.

[0069] Optionally, the flood-infrared light emitter emits low-power infrared light invisible to the human eye. An infrared sensor collects the returned infrared light, thus obtaining an infrared image formed by the flood-infrared light. The infrared image is unaffected by lighting conditions; even when lighting conditions are too dim to obtain a clear color image, valid information can still be acquired through the infrared image. In facial recognition technology, the infrared image can optionally be used for the liveness detection process.

[0070] Optionally, the aforementioned speckle infrared light emitter and the general infrared light emitter are integrated within the same light-emitting element. Optionally, the aforementioned speckle infrared light emitter and the general infrared light emitter can also be the same device, emitting different types of infrared light under the control of different commands.

[0071] To address the technical problem in existing technologies where only one image (depth map or infrared image) can be aligned with the color image within a single frame, embodiments of this disclosure modify the timing of infrared sensor placement for depth map acquisition and infrared image acquisition, enabling both depth and infrared images to align with the color image within a single frame. The following references... Figure 5 This illustrates the arrangement of time periods for acquiring color images, depth maps, and infrared images according to embodiments of this disclosure. Those skilled in the art should understand... Figure 5 This is merely one example of the present disclosure, and the disclosure is not limited thereto.

[0072] Figure 5 A comparison diagram is shown between the arrangement of time periods for acquiring color images, depth images, and infrared images according to embodiments of the present disclosure and the conventional arrangement of time periods for acquiring color images, depth images, and infrared images.

[0073] like Figure 5 As shown, the arrangement of the time periods for acquiring the color image, depth image, and infrared image according to the embodiments of this disclosure ensures that both the depth image and the infrared image are located within the time period used for the color image. Therefore, in terms of time alignment, the depth image and infrared image can be acquired using a global exposure method within one frame of acquiring the color image in a line-by-line scrolling manner.

[0074] Optionally, the duration of the time slot used for acquiring the depth map (infrared map) can be changed from being greater than half the duration of the time slot used for acquiring the color map (e.g., 30ms) to being less than half the duration of the time slot used for acquiring the color map (e.g., 10ms). Of course, the duration of the time slot used for acquiring the depth map can also remain greater than half the duration of the time slot used for acquiring the color map (e.g., 30ms), but in this case, the duration of the time slot used for acquiring the infrared map needs to be relatively small to ensure that the time slots used for acquiring the depth map and the time slots used for acquiring the infrared map do not overlap, and that the total duration of the time slots used for acquiring the depth map and the time slots used for acquiring the infrared map is less than the total duration of the time slot used for acquiring the color map. The reverse is also true, and this will not be elaborated further in this disclosure.

[0075] In some embodiments, where the processing speed of certain terminals or sensor processors is not high, in order to shorten the time period for acquiring depth maps to less than half the duration of the time period for acquiring color maps (e.g., 10ms), the time period for acquiring depth maps may begin at the moment when all photosensitive elements in the infrared sensor 402 begin exposure and end at the moment when the data in all photosensitive elements is read out (e.g., stored in the data storage area).

[0076] In some embodiments, when the computing speed of certain terminals or sensor processors is high enough, half the duration of the time period used to acquire the color image is sufficient for the terminal to complete the data processing of the depth map (e.g., filtering, denoising, optimization, depth information extraction, distortion correction, etc.). In this case, the time period used to acquire the depth map can start from the moment when all photosensitive elements in the infrared sensor 402 begin to be exposed and end at the moment when the data in all photosensitive elements has been processed.

[0077] Those skilled in the art should understand that the operations performed during the time period used for acquiring depth maps are merely examples. The specific operations to be performed by the infrared sensor during that time period can be determined based on one or more constraints such as the computing speed of the terminal or sensor processor, the image processing algorithm, and the image processing hardware. This disclosure does not impose any limitations on these aspects.

[0078] Since the photosensitive elements in the infrared sensor are used to acquire both depth and infrared images, and these two images cannot be acquired simultaneously, the infrared sensor 402 can only begin acquiring infrared images after all data from its photosensitive elements have been read out. Of course, the depth image can also be acquired after the infrared image acquisition is complete; this disclosure does not limit the order in which depth and infrared images are acquired. This disclosure is not limited to this.

[0079] Similar to the method for acquiring depth maps, the related operations performed during the infrared image acquisition period can also be determined based on one or more constraints such as the processing speed of the terminal or sensor processor, the image processing algorithm, and the image processing hardware. These will not be elaborated upon further in this disclosure.

[0080] exist Figure 5 Although the example uses 10ms as the time period for both depth map acquisition and infrared map acquisition, those skilled in the art should understand that the duration of the time period for depth map acquisition and infrared map acquisition can be the same or different, and either of them can be greater than 10ms or less than 10ms.

[0081] exist Figure 5 Although only one time period for acquiring dark images and one time period for acquiring infrared images are arranged within one frame of the color image acquisition duration, this disclosure is not limited to this. For example, in other embodiments of this disclosure, multiple time periods for acquiring dark images and multiple time periods for acquiring infrared images can be arranged alternately within one frame of the color image acquisition duration.

[0082] Optionally, the number of time periods used for acquiring color images is a first number, the number of time periods used for acquiring depth images is a second number, and the number of time periods used for acquiring infrared images is a third number. The second number equals the third number, and the sum of the second and third numbers is less than or equal to the first number. For example, in Figure 5 In this method, depth and infrared images are acquired only once every other time period used for acquiring color images. Since acquiring depth and infrared images requires exposure to a corresponding light source, and processing depth and infrared images also consumes a certain amount of power, this setting allows the embodiments of this disclosure to maintain the same power consumption or even reduce power consumption compared to traditional acquisition methods.

[0083] according to Figure 5The arrangement of time slots for acquiring color, depth, and infrared images shown in the diagram is compared with the traditional arrangement of time slots for acquiring color, depth, and infrared images. The embodiments of this disclosure achieve the acquisition of both depth and infrared images within one frame of color image acquisition, thereby achieving three-image alignment, shortening the time interval between capturing infrared and depth images, and thus improving the recognition efficiency and accuracy of face recognition technology.

[0084] Figure 6 Another structural diagram of an image acquisition device 400 according to an embodiment of the present disclosure is shown.

[0085] like Figure 6 As shown, the image acquisition device 400 also includes a power control circuit, which is electrically connected to the infrared sensor, the speckle infrared light source, and the ambient infrared light source. The power control circuit, upon receiving a trigger frame emitted by the infrared sensor, distributes the power voltage to the speckle infrared light source or the ambient infrared light source, causing the speckle infrared light source or the ambient infrared light source to emit light.

[0086] Optionally, the power control circuit can provide an enable signal to the speckle infrared light source (or diffuse infrared light source) to drive it to emit laser light. Optionally, the infrared sensor is connected to the power control circuit via an I2C bus. Of course, the infrared sensor can also be connected to the power control circuit in other ways, and this disclosure is not limited thereto.

[0087] Optionally, when the infrared sensor is used in conjunction with a speckle infrared light source (or a diffuse infrared light source), in some embodiments of this disclosure, the infrared sensor can control the projection timing of the speckle infrared light source (or diffuse infrared light source) via a strobe signal (also known as a strobe signal), wherein the strobe signal is generated based on a trigger frame generated by the infrared sensor. The timing of acquiring the depth map and the infrared map is determined at least in part based on a frame synchronization signal transmitted from the color sensor to the infrared sensor.

[0088] For example, after receiving the frame synchronization signal sent by the color sensor 401, the infrared sensor 402 generates a first trigger frame and a second trigger frame. The first trigger frame is used to schedule the time period for acquiring the depth map, and the second trigger frame is used to schedule the time period for acquiring the infrared image. Optionally, the time interval between the first trigger frame and the second trigger frame is greater than or equal to the processing time for acquiring the depth map. In this case, the depth map is acquired first, followed by the infrared image. The time interval between the first trigger frame and the second trigger frame can be determined by one or more constraints such as the processing speed of the terminal or sensor processor, the image processing algorithm, and the image processing hardware. This disclosure is not limited thereto.

[0089] Compared to traditional infrared sensors that generate only one trigger frame after receiving a frame synchronization signal, the embodiments of this disclosure generate at least two trigger frames after receiving a frame synchronization signal. This ensures that the time period for acquiring depth maps and the time period for acquiring infrared maps do not overlap, and that the total duration of the time period for acquiring depth maps and the time period for acquiring infrared maps is less than the total duration of the time period for acquiring color maps.

[0090] Optionally, based on the first trigger frame, the infrared sensor 402 transmits a first strobe signal to the power management circuit. Based on the first strobe signal, the power management circuit controls the speckle infrared light source to emit speckle infrared light during the exposure period of the time period used to acquire the depth map, so that the infrared sensor 402 acquires the depth map; and during the data readout period of the time period used to acquire the depth map, the infrared sensor 402 stores the depth map in the memory.

[0091] Optionally, if the first strobe signal is high, the power control circuit drives the speckle infrared light source to emit laser light into the scene; if the first strobe signal is low, the power control circuit prevents the speckle infrared light source from emitting laser light. Alternatively, the power control circuit can drive the speckle infrared light source to emit laser light into the scene when the first strobe signal is low, and prevent the speckle infrared light source from emitting laser light when the first strobe signal is high. This disclosure is not limited thereto. Similarly, the power control circuit can also drive the infrared light source based on the second strobe signal in the same / similar manner, which will not be described in detail here.

[0092] Optionally, based on the second trigger frame, the infrared sensor 402 transmits a second strobe signal to the power management circuit. Based on the second strobe signal, the power management circuit controls the flood infrared light source to emit flood infrared light during the exposure period of the time period used to acquire the infrared image, so that the infrared sensor 402 acquires the infrared image; and during the data readout period of the time period used to acquire the infrared image, the infrared sensor 402 stores the infrared image in the memory.

[0093] Optionally, the strobe signal can be an electrical signal with alternating high and low levels, and the speckle infrared light source (or diffuse infrared light source) projects laser according to the laser projection timing indicated by the strobe signal.

[0094] Optionally, the color sensor uses a frame synchronization clock to generate a frame synchronization signal, while the infrared sensor uses a pixel clock to generate a first trigger frame and a second trigger frame, wherein the frame rate of the pixel clock is more than three times the frame rate of the frame synchronization clock.

[0095] For example, color sensors can also use HDR (High Dynamic Range) mode. The HDR process can involve capturing two or more color images at different exposure levels, as well as combining different color images. In other words, HDR can involve capturing different color images of the same scene at different shutter speeds and aperture combinations to produce a set of color images with different luminosities and depths of field. The terminal can then post-process this set of color images to combine the color images, thereby creating a color image that includes the most focused, well-lit, and clearly defined facial features of the person to be detected.

[0096] Therefore, the frame synchronization clock of the color sensor can alternately and continuously generate high-exposure frames (corresponding to high exposure levels and low shutter speeds) and low-exposure frames (corresponding to low exposure levels and high shutter speeds) to acquire color images. As an example, the ratio of exposure time for high-exposure frames to exposure time for low-exposure frames is 16:1. As another example, the frame rate of the color sensor's frame synchronization clock can be 25 FPS, the duration of each frame (i.e., the time interval between adjacent frame synchronization clocks) is 40 ms, the longest exposure time is 40 ms, and the shortest exposure time is 10 ms.

[0097] Optionally, when the color sensor uses HDR mode, the duration of the high-exposure frame can be longer than the duration of the low-exposure frame. Therefore, the time slots for acquiring the depth map and the infrared image can be arranged only within the high-exposure frame to ensure sufficient duration for acquiring both. As another example, the duration of the high-exposure frame can be equal to the duration of the low-exposure frame. In this case, the color sensor may have more blank time during line-by-line reading of the low-exposure frame. In this case, the time slots for acquiring the depth map and the infrared image can be arranged only within the low-exposure frame to reduce potential interference during the acquisition of the depth map and infrared image. The above descriptions are merely examples, and those skilled in the art should understand that this disclosure is not limited thereto.

[0098] As an example, the frame rate of the pixel clock of an infrared sensor can be set to 120 FPS. Compared to a conventional pixel clock using a frame rate of 30 FPS, the infrared sensor can reduce the duration of the time periods used for depth map acquisition and infrared map acquisition, ensuring that the time periods for depth map acquisition and infrared map acquisition do not overlap, and that the total duration of the time periods for depth map acquisition and infrared map acquisition is less than the total duration of the time periods for color map acquisition. The above description is merely illustrative, and those skilled in the art should understand that this disclosure is not limited thereto.

[0099] Therefore, by adjusting the way the face recognition system drives the light source and the frame rate and frame synchronization method of triggering the infrared sensor, the time interval between capturing the infrared image and the depth image is shortened, thereby improving the recognition efficiency and accuracy of the face recognition technology.

[0100] Figure 7 A flowchart illustrating spatial alignment of a color image, a depth image, and an infrared image according to an embodiment of the present disclosure is shown.

[0101] like Figure 7 As shown, during factory production, the internal and external parameters can be calibrated first, and then actual testing can be conducted using checkerboard calibration. For example, the calibration of internal and external parameters can use the Zhang Zhengyou calibration method to determine the internal and external parameters of the color sensor, as well as the internal and external parameters of the infrared sensor. Of course, other calibration methods can also be used to determine the internal and external parameters, and this disclosure does not impose any limitations.

[0102] Optionally, the process of conducting actual testing using checkerboard calibration may include the following operations: collecting sample points using a checkerboard pattern, identifying the coordinates of the collected sample points, and finally determining the pixel correspondence between the depth map, infrared map, and color map based on the identified sample points. For example, a sample of the checkerboard pattern can be pasted on a wall, and the terminal device can be fixed 1.2 meters away from the wall. Then, the method according to the embodiments of this disclosure can be used to collect color maps, depth maps, and infrared maps respectively. For example, during the time period for collecting color maps, a color map corresponding to the checkerboard pattern is collected; during the time period for collecting depth maps, a depth map corresponding to the checkerboard pattern is collected; and during the time period for collecting infrared maps, an infrared map corresponding to the checkerboard pattern is collected. Finally, the coordinates of the sample points on these three maps are identified, and the coordinate transformation relationship between the color map and the infrared map, and the coordinate transformation relationship between the color map and the depth map are determined based on these coordinates. Of course, other calibration methods can also be used to determine the coordinate transformation relationship between the color map and the infrared map, and the coordinate transformation relationship between the color map and the depth map, and this disclosure does not impose any limitations.

[0103] like Figure 7 As shown, in practical use, the intrinsic and extrinsic parameters can be calibrated first, and then the coordinate transformation relationship between the color image and the infrared image, as well as the coordinate transformation relationship between the color image and the depth image, can be registered using a registration algorithm. The calibration of intrinsic and extrinsic parameters can be based on the estimation of multiple color images, depth images, and infrared images obtained during historical use, and then the intrinsic and extrinsic parameters determined in the factory production process can be adjusted based on the estimated intrinsic and extrinsic parameters.

[0104] The following is for reference Figure 8 The following describes a registration algorithm according to embodiments of the present disclosure. Optionally, the registration algorithm identifies the same series of pixels in the color image and the infrared image (depth map), and then calculates the parameters of the quadratic curve through quadratic curve fitting to generate a quadratic equation for coordinate transformation between the color image and the infrared image (depth map). The correspondence between each pixel in the color image and the infrared image (depth map) can be represented by this quadratic curve equation.

[0105] Figure 8 A schematic diagram of the registration process according to an embodiment of the present disclosure is shown.

[0106] The coordinate transformation relationships between color images and infrared images, as well as between color images and depth images, can both be represented by a single mapping matrix. The following explanation uses the determination of the coordinate transformation relationship between color images and depth images as an example; the coordinate transformation relationship between color images and infrared images can be determined similarly. Those skilled in the art should understand that this disclosure is not limited thereto.

[0107] refer to Figure 8 Where f is the focal length of the color sensor and the infrared sensor, which are the same. R is the image distance applied by the color sensor and the infrared sensor when detecting an object. The dashed line represents the light received by the infrared sensor, and the solid line represents the light received by the color sensor. d1 represents the length of the virtual image of the object formed by the infrared sensor in the x-direction, and d2 represents the length of the virtual image of the object formed by the color sensor in the x-direction. dz is the disparity value of the depth map. x is the disparity value of the color map.

[0108] For example, considering the computational complexity, and provided a certain level of accuracy is met, the coordinate transformation relationship between the color map and the depth map can be represented in the following form:

[0109] x r =x0+dx+βdz (1)

[0110] y r =y0+dy (2)

[0111] Among them, parameters x0 represents the tolerance of the parameters in the x-axis direction during the registration calculation, and y0 represents the tolerance of the parameters in the y-axis direction during the registration calculation. r and y r The coordinates of the depth map after alignment with the color map are represented, where dx and dy are the original coordinates in the depth map. Optionally, in some embodiments of this disclosure, dx and dy can be quadratic functions of a combination of x0 and y0. For these embodiments, the unknown parameters in the quadratic function (e.g., coefficients for x0 or y0, and coefficients for x0) can be solved by multiple measurements. 2 or y0 2 (coefficients, constant coefficients, etc.). Optionally, for Figure 8 In the example shown, dx and dz can be obtained in the following way.

[0112] refer to Figure 8 By using the similar triangle theorem, the following formulas (3) and (4) can be designed.

[0113]

[0114]

[0115] Therefore, formula (5) can be derived.

[0116]

[0117] Formula (5) calculates the relationship between x and dz. Substituting dz into formula (6) yields delta(x). Optionally, delta(x) may be equivalent to dx in some embodiments. Optionally, delta(x) may not be equivalent to dx in other embodiments, and this disclosure is not limited thereto.

[0118] delta(x)=(d1-d2 / d1)*dz (6)

[0119] Therefore, the coordinate transformation relationships between the color image and the infrared image, as well as between the color image and the depth image, can be determined. Then, based on these coordinate transformation relationships, the infrared image and the depth image can be calibrated. Finally, based on the color image and the calibrated infrared and depth images, the identity of the object to be detected can be determined.

[0120] Therefore, the image acquisition method, image acquisition device, and computer-readable storage medium provided by the embodiments of this disclosure shorten the time interval between capturing infrared images and depth images by adjusting the way the face recognition system drives the light source and the frame rate and frame synchronization method of triggering the infrared sensor, thereby improving the recognition efficiency and recognition accuracy of face recognition technology.

[0121] According to another aspect of this disclosure, an electronic device is also provided for implementing the method according to embodiments of this disclosure. Figure 9 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0122] like Figure 9 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the image acquisition method described above.

[0123] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 or ARM architecture.

[0124] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0125] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 10 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 10 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 10 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 10 One or more components in the computing device shown.

[0126] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 11 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0127] like Figure 11As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the method described with reference to the above figures according to embodiments of the present disclosure can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0128] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of this disclosure.

[0129] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0131] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. An image acquisition method, comprising: Color sensors are periodically arranged to collect color images during specific time periods; as well as After the infrared sensor receives the frame synchronization signal sent by the color sensor, during the time period for acquiring the color image, the infrared sensor generates a first trigger frame and a second trigger frame to arrange the time period for acquiring the depth image and the time period for acquiring the infrared image. The first trigger frame is used to arrange the time period for acquiring the depth image, and the second trigger frame is used to arrange the time period for acquiring the infrared image. Both the time periods for acquiring the depth image and the time periods for acquiring the infrared image are located within the time period for acquiring the color image. The time periods for acquiring depth maps and infrared maps do not overlap, and the total duration of the time periods for acquiring depth maps and infrared maps is less than the total duration of the time periods for acquiring color maps.

2. The image acquisition method according to claim 1, in, The time interval between the first trigger frame and the second trigger frame is greater than or equal to the processing time for acquiring the depth map, or the time interval between the first trigger frame and the second trigger frame is greater than or equal to the processing time for acquiring the infrared map.

3. The image acquisition method according to claim 1, wherein, The number of time periods used to acquire color images is the first number, the number of time periods used to acquire depth images is the second number, and the number of time periods used to acquire infrared images is the third number. The second number is equal to the third number, and the sum of the second number and the third number is less than or equal to the first number.

4. The image acquisition method according to claim 2, wherein, The color sensor uses a frame synchronization clock to generate a frame synchronization signal. The infrared sensor uses a pixel clock to generate a first trigger frame and a second trigger frame, and the frame rate of the pixel clock is more than three times the frame rate of the frame synchronization clock.

5. The image acquisition method according to claim 2, further comprising: Based on the first trigger frame, the infrared sensor transmits a first strobe signal to the power management circuit. Based on the first strobe signal, the power management circuit controls the speckle infrared light source to emit speckle infrared light during the exposure period in the time period used to acquire the depth map, so that the infrared sensor can acquire the depth map. as well as During the data readout period of the time period used to acquire the depth map, the infrared sensor stores the depth map in the memory.

6. The image acquisition method according to claim 2 further includes: Based on the second trigger frame, the infrared sensor transmits a second strobe signal to the power management circuit. Based on the second strobe signal, the power management circuit controls the flood infrared light source to emit flood infrared light during the exposure period of the time period used to acquire the infrared image, so that the infrared sensor can acquire the infrared image; and During the data readout period of the infrared sensor used to acquire the infrared image, the infrared image is stored in the memory.

7. The image acquisition method according to claim 1, further comprising: The color sensor acquires color images using a line-by-line rolling exposure method during the time period used for acquiring color images; The infrared sensor acquires depth maps using a global exposure method during the time period used for depth map acquisition. as well as The infrared sensor acquires infrared images using a global exposure method during the time period in which it is used to acquire infrared images.

8. The image acquisition method according to claim 1, further comprising: During the time period used for acquiring color images, acquire the color image corresponding to the object to be detected; During the time period used for acquiring depth maps, a depth map corresponding to the object to be detected is acquired; During the period used for acquiring infrared images, infrared images corresponding to the object to be detected are acquired; and The identity of the object to be detected is determined based on the collected color image, depth image, and infrared image.

9. The image acquisition method according to claim 8, wherein determining the identity of the object to be detected based on the acquired color image, depth image, and infrared image further includes: Based on the acquired color image, infrared image, and depth image, determine the coordinate transformation relationship between the color image and the infrared image, as well as the coordinate transformation relationship between the color image and the depth image; The infrared image and the depth image are calibrated based on the coordinate transformation relationship between the color image and the infrared image, and the coordinate transformation relationship between the color image and the depth image. The identity of the object to be detected is determined based on the color image, as well as the calibrated infrared and depth images.

10. An image acquisition device, comprising a color sensor and an infrared sensor, wherein, The color sensor is configured to periodically arrange time periods for acquiring color images and send frame synchronization signals to the infrared sensor; The infrared sensor is configured to receive a frame synchronization signal sent by the color sensor; During the time period for acquiring color images, a first trigger frame and a second trigger frame are generated based on the frame synchronization signal to arrange the time period for acquiring depth images and the time period for acquiring infrared images, wherein the first trigger frame is used to arrange the time period for acquiring depth images, the second trigger frame is used to arrange the time period for acquiring infrared images, and both the time period for acquiring depth images and the time period for acquiring infrared images are located within the time period for acquiring color images. The time periods for acquiring depth maps and infrared maps do not overlap, and the total duration of the time periods for acquiring depth maps and infrared maps is less than the total duration of the time periods for acquiring color maps.

11. The image acquisition device according to claim 10, wherein, The frame synchronization signal indicates the start time of the time period used to acquire the color image.

12. The image acquisition device according to claim 10, wherein, The number of time periods used to acquire color images is the first number, the number of time periods used to acquire depth images is the second number, and the number of time periods used to acquire infrared images is the third number. The second number is equal to the third number, and the sum of the second number and the third number is less than or equal to the first number.

13. The image acquisition device according to claim 11 further includes a power management circuit, a speckle infrared light source, and a diffuse infrared light source, wherein, The infrared sensor is also configured to: Based on the first trigger frame, a first strobe signal is transmitted to the power management circuit, and / or Based on the second trigger frame, the infrared sensor transmits a second strobe signal to the power management circuit; The power management circuit is further configured to: Based on the first strobe signal, the speckle infrared light source is controlled to emit speckle infrared light during the exposure period in the time period used for acquiring the depth map, and / or Based on the second strobe signal, the power management circuit controls the infrared light source to emit infrared light during the exposure period in the time period used to acquire the infrared image.

14. The image acquisition device according to claim 13, wherein the infrared sensor is further configured as: During the exposure period of the time period used for acquiring the depth map, the depth map is acquired, and during the data readout period of the time period used for acquiring the depth map, the depth map is stored in memory; and / or During the exposure period of the time period used for acquiring infrared images, the infrared images are acquired, and during the data readout period of the time period used for acquiring infrared images, the infrared images are stored in the memory.

Images