Image output method, image sensor, and image output system

Abstract

The present invention discloses an image output method, an image sensor, and an image output system. The image output method includes: receiving an image control instruction, the image control instruction including operating mode information and a corresponding configuration parameter set, wherein the configuration parameter set includes at least one configuration parameter combination; performing exposure based on the operating mode information to obtain a target image frame corresponding to each configuration parameter combination in the configuration parameter set, wherein the target image frame is generated by projecting a corresponding light source beam into a target space based on the operating mode information; configuring output parameters based on the configuration parameter set, and outputting target image frames corresponding to all configuration parameter combinations. The present invention eliminates the need to obtain external configuration parameters of the image sensor via a bus in real time, configures output parameters based on the configuration parameter set, and outputs target image frames corresponding to all configuration parameter combinations, thereby improving the output efficiency and accuracy of the target image frames.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image output method, an image sensor, and an image output system. Background Art

[0002] With the advancement of science and technology, especially computer vision, depth images are widely used in various scenarios, such as 3D face recognition and 3D object tracking. In one application scenario, a depth camera captures multiple near-infrared (NIR) and depth images of the object to be recognized and outputs them to a 3D recognition algorithm for simultaneous 3D face recognition. During the depth camera's acquisition of depth images and calculations, ambient light may interfere with the depth camera. Therefore, it may be necessary to eliminate ambient light before performing depth calculations.

[0003] In the prior art, when acquiring an image corresponding to a light source, it is necessary to first configure the image sensor using corresponding parameters (such as exposure time, gain, data switch, and other instructions). Before each image is captured, it is necessary to obtain the corresponding configuration parameters in real time from outside the image sensor of the depth camera. For example, the configuration parameters input by the user in real time are obtained from outside the image sensor through a data transmission interface (such as a bus). That is, it is necessary to continuously obtain the configuration parameters from outside the image sensor in real time through the bus. Data transmission on the bus is prone to errors due to interference caused by the environment and the operating status of the processor, which is not conducive to improving the efficiency and accuracy of image acquisition, and may further affect the efficiency and accuracy of subsequent operations such as 3D face recognition. Summary of the Invention

[0004] The main purpose of this application is to provide an image output method, an image sensor and an image output system, aiming to solve the problem in the prior art that configuration parameters need to be continuously acquired in real time through a bus, and that data transmission on the bus is prone to errors due to interference caused by the environment and processor operating conditions. The output parameters are configured based on a configuration parameter set, and the target image frames corresponding to all configuration parameter combinations are output, which is conducive to improving the output efficiency and accuracy of the target image frames.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides an image output method, wherein the above-mentioned method includes: receiving an image control instruction, the image control instruction includes working mode information and a corresponding configuration parameter set, wherein the configuration parameter set includes at least one group of configuration parameter combinations; performing exposure based on the working mode information, obtaining a target image frame corresponding to each group of configuration parameter combinations in the configuration parameter set, and the target image frame is generated by projecting a corresponding light source beam to the target space based on the working mode information; configuring output parameters according to the configuration parameter set, and outputting the target image frames corresponding to all configuration parameter combinations.

[0006] The second aspect of the present application provides an image sensor, a control and data transmission interface for receiving image control instructions, the image control instructions including working mode information and a corresponding configuration parameter set, wherein the configuration parameter set includes at least one group of configuration parameter combinations; a pixel array unit for receiving image control instructions transmitted by the control and data transmission interface, and performing exposure based on the working mode information to obtain a target image frame corresponding to each group of configuration parameter combinations in the configuration parameter set, the target image frame being generated by projecting a corresponding light source beam based on the working mode information; an image signal processor for configuring output parameters according to the configuration parameter set, and outputting target image frames corresponding to all configuration parameter combinations.

[0007] The third aspect of the present application provides an image output system, comprising: a processor, configured to configure working mode information and a corresponding configuration parameter set to generate an image control instruction, wherein the configuration parameter set includes at least one group of configuration parameter combinations; a transmitting module, configured to receive and project a corresponding light source beam to a target space based on the working mode information in the image control instruction to obtain target image frames corresponding to all configuration parameter combinations; a receiving module, comprising the image sensor of the second aspect, configured to receive and perform exposure based on the working mode information in the image control instruction, obtain target image frames corresponding to each group of configuration parameter combinations in the configuration parameter set, and configure output parameters according to the configuration parameter set, and output target image frames corresponding to all configuration parameter combinations.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, on which an image output program is stored. When the image output program is executed by a processor, the steps of any one of the image output methods in the first aspect are implemented.

[0009] As can be seen from the above, in this application, exposure is performed by receiving an image control instruction carrying operating mode information and a corresponding configuration parameter set, obtaining a target image frame corresponding to each configuration parameter combination, configuring output parameters based on the configuration parameter set, and outputting target image frames corresponding to all configuration parameter combinations. This eliminates the need to obtain configuration parameters external to the image sensor via a bus in real time, helping to avoid interference and data errors caused by the environment and processor operating conditions, thereby improving the output efficiency and accuracy of the target image frames, and thus improving the efficiency and accuracy of subsequent operations such as 3D face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a structural diagram of an image output system provided by an embodiment of the present invention;

[0012] Figure 2 is a structural diagram of an image sensor provided by an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of cyclic output of an image frame group provided by an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of a corresponding relationship between a configuration parameter combination and a subframe provided by an embodiment of the present invention;

[0015] Figure 5 This is a schematic diagram of a corresponding relationship between a configuration parameter set and multiple target image frames provided by an embodiment of the present invention;

[0016] Figure 6 Schematic diagram of the inter-frame interval and the time interval between frame groups provided by an embodiment of the present invention;

[0017] Figure 7 It is a flowchart of an image output method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0019] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0020] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0023] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0025] In the prior art, when transmitting images, the image output system needs to first use corresponding parameters (such as exposure time, gain, data switch and other instructions) to configure the image sensor in the receiving module. Before each image is captured, the corresponding configuration parameters need to be obtained from the processor, for example, through the I2C bus, I3C bus, SPI bus or control and data transmission interface. The problem with the prior art is that it is necessary to continuously obtain configuration parameters from the processor through the bus or control and data transmission interface. The data transmission on the bus is prone to errors due to interference caused by the environment and the operating status of the processor, which is not conducive to improving the efficiency and accuracy of image acquisition, and thus affects the efficiency and accuracy of subsequent operations such as 3D face recognition. Among them, the data errors on the I2C (or other) bus are inconsistent with expectations, such as exposure time mismatch, gain mismatch, sensor (Sensor) switch status mismatch, etc.

[0026] Furthermore, for specific requirements such as 3D recognition, simultaneous input of near-infrared and depth images is often required. The smaller the time interval between the two image streams and the higher the synchronization rate, the better the recognition performance. Therefore, the smaller the interval between the two acquired image frames (e.g., the near-infrared image and the speckle laser image), the better the synchronization and, theoretically, the better the algorithm's recognition performance. In existing technologies, due to limitations such as the Mobile Industry Processor Interface (MIPI) transmission protocol, most implementations employ different light sources and employ image sensors that alternately output near-infrared image subframes and speckle laser images. However, the bus requires a large amount of parameter data to be transmitted (e.g., exposure time, gain, data switch commands, etc.). The greater the amount of data required to be transmitted, the slower the bus transmission efficiency may be, resulting in delayed acquisition of the corresponding configuration parameters. Furthermore, each capture requires waiting for the configuration parameters to be fully transmitted before the corresponding image can be acquired and output. This results in a longer inter-frame interval between the two acquired frames, and consequently, a longer time interval between the output infrared and depth images, impacting recognition efficiency and accuracy.

[0027] In order to solve at least one problem existing in the prior art, the present invention designs an image output system, such as Figure 1 As shown, the image output system includes a transmitting module 1, a receiving module 2, and a processor 3, wherein the receiving module 2 includes an image sensor 21. The processor 3 is used to configure the working mode information and the corresponding configuration parameter set, wherein the configuration parameter set includes at least one configuration parameter combination. The processor can obtain the configuration parameter set by selecting at least one configuration parameter combination from a plurality of configuration parameter combinations as a cyclic unit based on the working mode information to form a configuration parameter set. The configuration parameter combination includes an exposure time, and the working mode information includes an exposure mode. The exposure time and exposure mode can be used to expose to obtain a target image frame corresponding to each configuration parameter combination. The target image frame is generated by controlling the light source to be turned on or off using the working mode information to project a corresponding light source beam into the target space. The target image frame has a corresponding relationship with the above-mentioned configuration parameter combination. A set of configuration parameter combinations can correspond to the transmission of a target image frame of any frame.

[0028] In one embodiment, the configuration parameter combination may further include exposure time, gain, and the number of transmission frames N, wherein a set of configuration parameter combinations corresponds to matching N target image frames, where N is greater than or equal to 1. For example, a set of configuration parameter combinations may correspond to the transmission of up to 8 target image frames. The processor 3 is configured to configure the operating mode information and the corresponding configuration parameter set to generate an image control instruction.

[0029] In one embodiment, the image control instruction is an initialization instruction. The receiving module receives the configuration parameter set in the control instruction and stores it in register 1021 for storage. When data transmission is required, a start control instruction is sent to transmitting module 1 and receiving module 2 to activate the light source for image acquisition and data transmission. Before data transmission according to the new operating mode information and the corresponding configuration parameter set is performed, processor 3 sends an end instruction to transmitting module 1 and receiving module to control the current data acquisition and data transmission.

[0030] Furthermore, the processor 3 retransmits the new operating mode information and the image control instructions of the corresponding configuration parameter set to perform the next data acquisition and data transmission. In one embodiment, the image control instructions include a start control instruction, and the transmitting module 1 and the receiving module 2 directly execute the start control instruction to perform the current data acquisition and data transmission.

[0031] The transmitting module 1 is used to receive and project the corresponding light source beam to the target space based on the working mode information in the image control instruction to obtain the target image frames corresponding to all configuration parameter combinations. In one embodiment, the working mode information includes a light source control signal and an exposure mode. The transmitting module 1 receives the light source control signal and sequentially projects the light source beam corresponding to the light source to the target space. For example, if the light source control signal is a near-infrared floodlight light source control signal, the floodlight beam is projected to the target space by the near-infrared floodlight light source. The receiving module 2 performs exposure according to the exposure mode in the obtained working mode information to obtain the target image frame corresponding to each configuration parameter combination in the configuration parameter set, and then configures the output parameters according to the configuration parameter set, and uses the output parameters to output the target image frames corresponding to all configuration parameter combinations. Among them, the output parameters are some parameters in the configuration parameter combination, which are used for the output of the target image frame. There is no need to configure them separately according to each frame, which improves the transmission efficiency.

[0032] Furthermore, the light source control signal includes at least two sub-light source control signals. For example, if the transmitting module 1 includes a laser light source and a near-infrared floodlight light source, the light source control signal includes on / off control logic and a loop sequence for the laser light source and the near-infrared floodlight light source. The signal can be used to activate the near-infrared floodlight light source to project a floodlight beam toward the target space, allowing the receiving module 2 to obtain a near-infrared image subframe; activate the laser light source to project a speckle beam toward the target space, allowing the receiving module 2 to obtain a speckle image subframe; and deactivate the near-infrared floodlight light source and the laser light source to project ambient light toward the target space, allowing the receiving module 2 to obtain a background image subframe. For another example, if the transmitting module 1 includes a light source signal for a first wavelength band, a light source signal for a second wavelength band, and a light source signal for a third wavelength band, these signals are used to activate the light sources in the first wavelength band, the second wavelength band, and the third wavelength band, respectively, thereby obtaining multispectral data for liveness detection.

[0033] Receiving module 2 not only receives the aforementioned images but also outputs target image frames corresponding to all configuration parameter combinations based on the configuration parameter set. For example, if transmitting module 1 includes a laser light source and a near-infrared floodlight light source, then all target image frames refer to all images generated by cyclically turning the laser light source and the near-infrared floodlight light source on and off according to the light source control signal.

[0034] In one embodiment, the image output system is a depth camera, which has a built-in processor 3 and a depth calculation chip 4. The depth calculation chip 4 is used to perform depth calculation based on the received light beam information to obtain a depth image.

[0035] Optionally, the image output system is a structured light depth camera. Transmitter module 1 includes a laser light source for projecting an encoded structured light patterned speckle beam into the target space. Receiving module 2 captures the structured light pattern and outputs it to depth calculation chip 4, which performs depth calculation on the structured light pattern to obtain a depth image of the target space. Transmitter module 1 may include a light source, a lens, and a diffractive optical element. The structured light pattern it emits is an infrared speckle pattern with a relatively uniform distribution of particles but high degree of uncorrelation. Receiving module 2 is an infrared camera including image sensor 21. The structured light pattern may also take other forms, such as stripes or two-dimensional patterns. When the laser light source is turned on, image sensor 21 captures a speckle image formed by the laser light source and infrared light from the environment projected onto the target space. When the laser light source is turned off, image sensor 21 captures a background image formed by only ambient light projected onto the target space. The target space is the object for which a depth image is to be captured and obtained. This can be a person, an object, or a scene, without specific limitation.

[0036] In one embodiment, the image output system may be a binocular structured light depth camera, etc. In the following description, the structured light depth camera will be used as an example, and its principle can be applied to other types of image output systems.

[0037] In one embodiment, the transmitting module 1 includes a floodlight for emitting near-infrared light and a laser light source for emitting speckle laser light. The transmitting module 1 is connected to the processor via a depth calculation chip 4 or directly. The processor 3 can configure operating mode information and a corresponding set of configuration parameters to generate image control instructions. The operating mode information is transmitted to the transmitting module 1 to control the on / off of the near-infrared floodlight and laser light source, thereby acquiring the corresponding target image frame. The near-infrared wavelength band used can be an 850-nanometer wavelength band or a 940-nanometer wavelength band.

[0038] In one embodiment, the image output system is equipped with an LED surface light source using a near-infrared frequency band, referred to as a floodlight, which is used to supplement the illumination in the near-infrared frequency band so that the image sensor can generate an image of appropriate brightness for conventional image recognition and liveness detection operations. The laser light source is composed of a dot matrix laser light source in the near-infrared frequency band, which is used to mark the current environment so that the image sensor can generate a speckle image subframe with sufficient texture information, and then perform depth calculation to obtain a corresponding depth image. The near-infrared image subframe is a near-infrared image (reflected by the scene) obtained under a surface light source (ideally a uniform surface light source). When the near-infrared floodlight is turned on and the laser light source is turned off, the near-infrared image can be captured by the corresponding image sensor and output to the algorithm used for image recognition (such as face recognition and positioning) and liveness detection.

[0039] The processor 3 is used to control the entire image output system. The processor 3 can be a single processor or multiple processor units, including but not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a neural network processor (NPU), an image signal processor (ISP), etc. In some embodiments, the processor 3 can be an integrated system on a chip (SoC) or an application-specific integrated circuit (ASIC), including a processor such as a CPU, on-chip memory, a processor, a communication interface, etc.

[0040] In one embodiment, the processor 3 can select information such as exposure time and working mode information from the corresponding configuration parameter set from the memory 5 to generate an image control instruction. The transmission control instruction includes the working mode information and the corresponding configuration parameter set, and is sent to the processor 3 through the control and data transmission interface between the image sensor 21 to generate the image control instruction, so as to control the image sensor to obtain the target image frame according to the preset exposure time and exposure mode and cyclically output all target image frames based on the configuration parameter set.

[0041] In one embodiment, the image control instruction includes the exposure time in the configuration parameter combination and the exposure mode in the operating mode information. The processor can send the image control instruction to the image sensor, which then controls the pixel array to perform exposure according to the exposure mode and exposure time, thereby obtaining the target image frame. If the scene is background reduction, the exposure mode for the speckle image subframe is global shutter exposure, and the exposure mode for the background image subframe is rolling shutter exposure. If the scene is not background reduction and is directly output, the exposure mode for the speckle image subframe and the background image subframe can be global shutter exposure. The exposure mode for the near-infrared subframe can be global shutter exposure.

[0042] The configuration parameter set is formed by at least one configuration parameter combination selected by the processor 3 from a plurality of configuration parameter combinations based on the working mode information as a loop unit. For example, the plurality of configuration parameter combinations are preset multiple context groups, such as three groups of context1, context2, and context3. In one embodiment, the processor selects at least one of the configuration parameter combinations as needed, for example, selects two groups of configuration parameter combinations, and there are three groups of options, (context1, context2), (context2, context3), or (context1, context3). One of the groups, such as (context1, context3), is selected as a parameter loop unit. Assuming that the number of transmission frames corresponding to context1 is 2 and the number of transmission frames corresponding to context3 is 5, then one parameter loop unit corresponds to 7 transmission frames, and the loop is performed every 7 frames. Among them, the configuration parameters in context1, context2, and context3 can be set according to actual conditions.

[0043] In one embodiment, the processor sequentially selects at least one configuration parameter combination as needed, for example, three combinations of context 1, context 2, and context 3. Two configuration parameter combinations are sequentially selected, namely (context 1, context 2), which serve as parameter loop units. Assuming that context 1 corresponds to 2 transmission frames and context 2 corresponds to 3 transmission frames, one parameter loop unit corresponds to 5 transmission frames, and the loop is repeated every 5 frames.

[0044] In one embodiment, the image output system also includes a memory 5, which is used to store processor control instructions, configuration information, and data storage, such as system data, application data, parameter data, etc., wherein the system data includes operating systems such as Android and Linux, and the parameter data includes internal and external parameters of the image output system.

[0045] The above-mentioned image output system does not need to obtain the external configuration parameters of the image sensor in real time through the bus. The image control instructions can be generated by the processor 3, and the image control instructions carrying the working mode information and the corresponding configuration parameter set are transmitted to the transmitting module 1 and the receiving module 2, controlling the transmitting module to emit the corresponding light source beam, and controlling the receiving module 2 to cyclically output the corresponding target image frame, thereby improving the efficiency and accuracy of obtaining the target image frame.

[0046] Figure 2 FIG. 1 is a structural diagram of an image sensor provided by an embodiment of the present invention. Figure 2As shown, the image sensor 21 includes a pixel array unit 101, an image signal processor 102, and a control and data transmission interface 103. The processor 3 sends an image control instruction to the image sensor 21, which is received by the control and data transmission interface 10 of the image sensor 21 and further transmitted to the pixel array unit 101. The image control instruction includes working mode information and a corresponding configuration parameter set, wherein the configuration parameter set includes at least one group of configuration parameter combinations. The pixel array unit 101 receives the image control instruction transmitted by the control and data transmission interface, and performs exposure based on the working mode information to obtain a target image frame corresponding to each group of configuration parameter combinations in the configuration parameter set. The target image frame is generated based on the corresponding light source beam projected by the emission module 1 according to the working mode information. The image signal processor 102 is used to configure the output parameters according to the configuration parameter set, and output the target image frames corresponding to all configuration parameter combinations.

[0047] In one embodiment, the image signal processor 102 may further include a register 1021. The control and data transmission interface 103 is configured to receive image control instructions sent by the processor 3, and transmit the operating mode information and corresponding configuration parameter set in the image control instructions to the register 1021 for storage, so as to be called by the image signal processor 102 and the pixel array unit 101.

[0048] In one embodiment, the operating mode information includes a light source control signal and an exposure mode. The light source control signal is at least one light source signal. The image sensor can support exposure of multiple target image frames according to corresponding exposure modes. Furthermore, the operating mode information includes on / off control logic and a cycling sequence for the laser light source and the near-infrared flood light source. The pixel array unit 101 performs exposure according to the exposure time in the configuration parameter set and the exposure mode in the operating mode information to obtain a near-infrared image subframe, a speckle image subframe, or a background image subframe. A near-infrared image subframe is obtained by turning on the near-infrared flood light source and projecting a flood beam into the target space. A speckle image subframe is obtained by projecting a speckle beam into the target space using both ambient light and an on-state laser light source. A background image subframe is obtained by turning off the laser light source and the near-infrared flood light source and projecting ambient light into the target space. A background-subtracted image subframe is obtained by performing background subtraction calculations on the speckle image subframe and the background image subframe by the image signal processor 102.

[0049] In one embodiment, five types of image frame groups may be obtained according to the working mode information. The first type of image frame group includes a near-infrared image subframe and a speckle image subframe, the second type of image frame group includes a speckle image subframe and a background image subframe, the third type of image frame group includes a background-subtracted image subframe, the fourth type of image frame group includes a near-infrared image subframe, a speckle image subframe, and a background image subframe, and the fifth type of image frame group includes a near-infrared image subframe and a background-subtracted image subframe.

[0050] Furthermore, a corresponding image frame group can be selected from the above five image frame groups according to the working mode information for cyclic output. Figure 3 As shown, assuming that the fourth frame group (MODE) is used as the cyclic output, NIR represents the near-infrared image subframe, TG represents the speckle image subframe, and BG represents the background image subframe. Assuming that the target transmission data is subframe 0 to subframe 8, subframe 0 corresponds to NIR0, subframe 1 corresponds to TG0, subframe 2 corresponds to BG0, subframe 3 corresponds to NIR1, subframe 4 corresponds to TG1, subframe 5 corresponds to BG1, subframe 6 corresponds to NIR2, subframe 7 corresponds to TG2, and subframe 8 corresponds to BG2.

[0051] Furthermore, the controller can pre-set the configuration parameter combination in the configuration parameter set as a parameter cycle unit for transmission, such as Figure 4 As shown, the configuration parameter set is composed of three configuration parameter combinations (context0, context1, context2) as parameter cycle units (GROUP). Assuming that the target transmission data is subframe 0-subframe 4, the preset context0 transmission frame number is 2, corresponding to subframe 0 and subframe 1; the context1 transmission frame number is 1, corresponding to subframe 2; the context2 transmission frame number is 2, corresponding to subframe 3 and subframe 4.

[0052] Furthermore, a corresponding image frame group can be selected from the above five image frame groups according to the working mode information for cyclic output, and a configuration parameter combination of the cycle can be pre-set, such as Figure 5As shown, two configuration parameter combinations within the configuration parameter set can be sequentially selected as context0 and context1, which serve as parameter loop units (GROUPs). Image signal processor 102 generates a fifth image frame group (MODE) as a loop output. The fifth image frame group includes near-infrared image subframes and background-subtracted image subframes. NIR represents a near-infrared image subframe, TG represents a speckle image subframe, BG represents a background image subframe, and MG represents a background-subtracted image subframe. MG is obtained by image signal processor 102 by subtracting speckle image subframe TG from the corresponding background image subframe BG. To facilitate background subtraction, speckle image subframe TG and background image subframe BG must share the same context. Configuration parameter combinations include exposure time, gain, frame length, and the number of transmitted frames N. One configuration parameter combination corresponds to matching N target image frames, where N is greater than or equal to 1. In the figure, context0 has a transmission frame number of 1, corresponding to NIR; context1 has a transmission frame number of 1, corresponding to MG.

[0053] In one embodiment, the image signal processor 102 cyclically outputs target image frames at equal intervals according to a timing sequence such as a frame length set in the configuration parameter set.

[0054] In some embodiments, in a background subtraction scenario, it is required that the speckle image subframes and the background image subframes are output as synchronously as possible. The smaller the interval between subframes, the better the depth calculation effect. The synchronous output of the near-infrared image subframes, the speckle image subframes, and the background image subframes is also beneficial to depth calculation and face recognition. Therefore, the interval between subframes in each group can be controlled by adjusting the subframe length through a combination of configuration parameters of the image signal processor 102.

[0055] like Figure 6As shown in the figure, taking the transmission of the fourth type of image frame group as an example, NIR represents the near-infrared image subframe, TG represents the speckle image subframe, and BG represents the background image subframe. (NIR0, TG0, BG0) is the first frame group, and (NIR1, TG1, BG1) is the second frame group. L1, L2, and L3 represent the time intervals between subframes (e.g., between NIR0 and TG0, between TG0 and BG0, and between BG0 and NIR1). L3 is the time interval between the last subframe in a frame group (e.g., BG0) and the time interval between two frame groups. L4 represents the time (or period) of the frame group. By shortening the time intervals (e.g., L1 and L2) between the first few subframes in a frame group (e.g., NIR0, TG0, BG0), depth calculation and face recognition performance are improved. The time interval (L3) of the last frame in a frame group (e.g., BG0) is appropriately increased to increase the time interval between frame groups, ensuring that the frame group period (L4) or frame rate (1 / L4) meets the requirements. Furthermore, the time interval between subframes can be controlled by using an exposure mode, for example, by alternately using a global shutter exposure mode and a rolling shutter exposure mode to expose TG and BG respectively, thereby reducing the time interval between TG and BG.

[0056] In one embodiment, the light source control signal may include a laser light source control signal, i.e., a laser light source is used for projection and corresponding speckle image subframes are obtained. The speckle image subframes are generated by at least one speckle image subframe. For example, there are three configuration parameter combinations available for selection: context0, context1, and context2, corresponding to the number of transmitted frames being 1, 2, and 2, respectively. Optionally, two of them, such as context1 and context2, may be selected as the current configuration parameter combination. In this case, a set of context1 + context2 corresponds to the transmission of four speckle image subframes, and then context1 and context2 are cyclically transmitted to transmit the corresponding speckle image subframes. Optionally, a configuration parameter combination in the configuration parameter set sequentially selects context0 and context1 as the current configuration parameter combination. In this case, a set of context0 + context1 corresponds to the transmission of three speckle image subframes, and then context1 and context2 are cyclically transmitted to transmit the corresponding speckle image subframes.

[0057] Furthermore, when the background reduction control information indicates that background reduction is not performed, the pixel array unit 101 may use a global shutter to expose the speckle image subframe and the background image subframe, and control the distance between the two frames by the frame length. When the background reduction control information indicates that background reduction is not performed, the pixel array unit 101 may use a global shutter to expose the speckle image subframe and a rolling shutter to expose the background image subframe. Compared with conventional global shutter exposure, this can effectively shorten the time interval between the two exposures (especially the first half of the image), thereby reducing the impact of noise and motion, and improving depth quality in outdoor environments with strong interference.

[0058] Optionally, the pixel array unit 101 is specifically provided with two image data storage nodes, which can continuously capture two image subframes and then output the two image subframes simultaneously. Specifically, the pixel array unit 101 first exposes the scene with the laser light source plus ambient light (global shutter exposure), then does not output the data and stores it in the image data storage node, and immediately exposes the pure ambient scene (i.e., the scene with the laser light source turned off) (rolling shutter exposure). The data of the pure ambient scene can be stored in another image data storage node, and the above two data (i.e., background-subtracted image subframes) are output simultaneously for depth calculation. Compared with conventional global shutter exposure, the time interval between the two exposures can be effectively shortened (especially in the first half of the image) to reduce the impact of noise and motion, thereby improving the depth quality in outdoor environments with strong interference. It should be noted that in conventional image sensors and their exposure control processes, after one exposure, the data corresponding to the exposure must be output before the next exposure can be performed, which is not conducive to shortening the time interval between the two exposures. Based on the solution of this embodiment, it is beneficial to shorten the time interval between the two exposures. Furthermore, the global shutter exposure followed by the rolling shutter exposure can achieve the effect of shortening the exposure time interval between the speckle image subframe and the background image subframe at a relatively low cost.

[0059] In one embodiment, the image sensor can also implement grouped exposure of multiple frames (a group can include global shutter exposure and rolling shutter exposure). Within a group, the exposure time interval of the multiple frames is minimized, while the time interval between different groups can be adjusted (e.g., slightly increased) according to actual needs. In this way, the time interval between two (or more) global shutter exposures can be shortened as much as possible at a lower total output frame rate. It is also possible to adjust the frame rate without changing the time interval between multiple frames within a group. For example, the exposure time can be adjusted without changing the time interval between the speckle image subframes and background image subframes in the same group.

[0060] The above exposure control method can optimize the depth calculation effect and algorithm recognition effect.

[0061] In one embodiment, a method for outputting an image to the image sensor 21 is provided. Figure 7 FIG. 1 is a flow chart of an image output method provided by an embodiment of the present invention, such as Figure 7 As shown, the above image output method specifically includes the following steps:

[0062] Step S100: receiving an image control instruction, the image control instruction including operating mode information and a corresponding configuration parameter set, wherein the configuration parameter set includes at least one set of configuration parameter combinations;

[0063] Step S200 , performing exposure based on the working mode information to obtain a target image frame corresponding to each configuration parameter combination in the configuration parameter set, where the target image frame is generated by projecting a corresponding light source beam onto a target space based on the working mode information;

[0064] Step S300: configuring output parameters according to the configuration parameter set, and outputting target image frames corresponding to all configuration parameter combinations.

[0065] Specifically, the aforementioned image control instructions can be input by a processor, and the image sensor receives the image control instructions to obtain operating mode information. It can then perform exposure based on the operating mode information to obtain an exposure mode. Furthermore, based on the operating mode information, it can control the corresponding light source to project the corresponding light beam into the target space, thereby obtaining a target image frame. The image type in the target image frame is also determined by the light source control signal controlling the corresponding light source to project. After obtaining the corresponding configuration parameter set, the image sensor can perform parameter matching on multiple consecutive target image frames based on the configuration parameter set, without the need to receive single-frame configuration parameters transmitted by the controller via a bus or control and data transmission interface in real time, or to read the configuration parameters for each frame from a register each time.

[0066] In some embodiments, a configuration parameter set is a set formed by selecting at least one configuration parameter combination from a plurality of configuration parameter combinations as a cyclic unit based on the operating mode information. The configuration parameter combination may further include a gain, a frame length, and a number of transmission frames N. A set of configuration parameter combinations corresponds to N target image frames in a matching target image, where N is greater than or equal to 1. The frame length is used to adjust the inter-frame spacing between two frames.

[0067] The operating mode information also includes a light source control signal, which is used to control the projection of the corresponding light source. Further, the operating mode information may include at least two sub-light source control signals, each sub-light source control signal is used to control the corresponding light source to project cyclically in a preset order.

[0068] In one embodiment, the light source control signal may include on / off control logic and a loop sequence for a laser light source and a near-infrared flood light source. The corresponding generated target image frames include near-infrared image subframes, speckle image subframes, background image subframes, or background-subtracted image subframes. The near-infrared image subframes may be obtained by turning on the near-infrared flood light source to project a flood light beam into the target space; the speckle image subframes may be obtained by turning on the laser light source to project a speckle beam into the target space; the background image subframes may be obtained by turning off the laser light source and the near-infrared flood light source and projecting ambient light into the target space; and the background-subtracted image subframes may be obtained by performing background subtraction calculation on the speckle image subframes and the background image subframes. Furthermore, the exposure mode of the near-infrared image subframe is global shutter exposure, and the light source control signal also includes background subtraction control information. If the background subtraction control information indicates no background subtraction, that is, when the speckle image subframe and the background image subframe are directly output without background subtraction, the global shutter exposure mode can be used to expose the speckle image subframe and the background image subframe. If the background subtraction control information indicates background subtraction, that is, when on-chip background subtraction is performed before output, the global shutter exposure mode can be used to expose the speckle image subframe, and the rolling shutter exposure mode can be used to expose the background image subframe, and background subtraction calculation can be performed to obtain the background-subtracted image subframe.

[0069] In one embodiment, five image frame groups can be obtained according to the working mode information: the first image frame group includes a near-infrared image subframe and a speckle image subframe, the second image frame group includes a speckle image subframe and a background image subframe, the third image frame group includes a background-subtracted image subframe, the fourth image frame group includes a near-infrared image subframe, a speckle image subframe, and a background image subframe, and the fifth image frame group includes a near-infrared image subframe and a background-subtracted image subframe. The above five image frame groups correspond to five transmission modes, respectively:

[0070] The first transmission mode alternates between outputting near-infrared image subframes and speckle image subframes. In this mode, two frames of data are output as a single cycle. Configuration parameters such as exposure time, gain, and DPC on / off for each frame (the two preceding and succeeding frames) can be set based on actual needs. The interframe interval (interframe interval) between near-infrared and speckle image subframes can be fine-tuned using parameters such as the frame length in the configuration parameter set.

[0071] The second transmission mode can be used to alternately output speckle image subframes and background image subframes. In this mode, the image sensor exposes the speckle image subframe and background image subframe using the global shutter exposure mode. The two frames are output in a loop, and the speckle image subframe and background image subframe are used for off-chip background subtraction and depth data calculation. In this mode, the exposure time, gain, DPC switch, and other configuration parameters corresponding to different frames (the previous and next frames) must be completely consistent. The interframe interval (interframe interval) of the speckle image subframe and background image subframe can be fine-tuned using parameters such as the frame length in the configuration parameter set.

[0072] The third transmission mode outputs a background-subtracted image subframe. In this mode, the image sensor exposes a speckle image subframe using a global shutter exposure mode and a background image subframe using a rolling shutter exposure mode. The background-subtracted image subframe is then subtracted from the speckle image subframe and output. The exposure time, gain, and DPC switch settings for the speckle and background image subframes must be identical.

[0073] The fourth transmission mode can output speckle image subframes, background image subframes, and near-infrared image subframes. Similar to the second transmission mode, this mode allows for configurations such as exposure time, gain, and DPC (Digital Pitch Control) switches for the near-infrared, speckle, and background image subframes to be customized (subject to variation).

[0074] The fifth transmission mode can output background-subtracted image subframes and near-infrared image subframes. This mode is similar to the third transmission mode. Specifically, the exposure time, gain, DPC switch, and other configurations of the near-infrared image subframe and the speckle / background image subframe can be set according to individual needs (which may vary).

[0075] In addition to the above modes, data can also be transmitted in normal mode. That is, in normal output scenarios, the output mode adjusts the inter-frame spacing according to the pre-set frame length, etc. For example, the output of near-infrared image subframes or speckle image subframes, and its exposure time, gain, DPC switch and other parameters can be set according to the transmission requirements.

[0076] As can be seen from the above, this method does not require real-time acquisition of external image sensor configuration parameters via the bus, which helps avoid interference and data errors caused by the environment and processor operating conditions, thereby improving the efficiency and accuracy of image acquisition. This, in turn, can improve the efficiency and accuracy of subsequent operations such as 3D face recognition, which helps enhance the user experience. By adjusting the time interval between subframes and the time interval between frame groups, as well as shortening the exposure time, the 3D recognition effect and depth calculation effect can be improved.

[0077] An embodiment of the present invention further provides a computer-readable storage medium, on which an image output program is stored. When the image output program is executed by a processor, the steps of any one of the image output methods provided in the embodiments of the present invention are implemented.

[0078] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a specific order of execution; the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0080] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0081] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0082] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented.

[0083] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An image output method, characterized in that: include: receiving an image control instruction, the image control instruction including operating mode information and a corresponding configuration parameter set, wherein the configuration parameter set includes at least one set of configuration parameter combinations; Performing exposure based on the working mode information to obtain a target image frame corresponding to each combination of the configuration parameters in the configuration parameter set, wherein the target image frame is generated by projecting a corresponding light source beam onto a target space based on the working mode information; Configuring output parameters according to the configuration parameter set, and outputting target image frames corresponding to all combinations of the configuration parameters; The configuration parameter set is a set formed by selecting at least one configuration parameter combination from a plurality of configuration parameter combinations as a cyclic unit based on the working mode information; The configuration parameter combination includes exposure time, gain, frame length, number of transmission frames N and output parameters, wherein the set of configuration parameter combinations corresponds to matching N frames of target image frames, N is greater than or equal to 1, the frame length is used to control the inter-frame spacing between two target image frames, and the output parameters are used to control the output of the target image frame corresponding to the configuration parameter combination.

2. The method according to claim 1, characterized in that The working mode information includes a light source control signal and an exposure mode, and the light source control signal is used to control the projection of a corresponding light source.

3. The method according to claim 2, characterized in that The light source control signal includes at least two sub-light source control signals, and each sub-light source control signal is used to control the corresponding light source to cyclically project in a preset order.

4. The method according to claim 3, characterized in that The light source control signal includes the laser light source and the near-infrared flood light light source switch control logic and the cycle sequence; The target image frame includes a near-infrared image subframe, a speckle image subframe, a background image subframe, or a background-subtracted image subframe, wherein the near-infrared image subframe is an image subframe obtained by turning on the near-infrared floodlight light source to project a floodlight beam toward the target space, the speckle image subframe is an image subframe obtained by turning on the laser light source to project a speckle beam toward the target space, the background image subframe is an image subframe obtained by turning off the near-infrared floodlight light source and the laser light source and projecting ambient light toward the target space, and the background-subtracted image subframe includes an image subframe obtained by performing a background subtraction calculation on the speckle image subframe and the background image subframe.

5. The method according to claim 4, characterized in that The light source control signal also includes background subtraction control information; The exposure mode of the near-infrared image subframe is global shutter exposure; When the background reduction control information indicates not reducing the background, the exposure mode of the speckle image subframe and the background image subframe is the global shutter exposure; When the background subtraction control information is background subtraction, the exposure mode of the speckle image subframe is global shutter exposure, the exposure mode of the background image subframe is rolling shutter exposure, and the speckle image subframe and the background image subframe are used for background subtraction calculation to obtain a background subtracted image subframe.

6. An image sensor, characterized in that: The image sensor comprises: A control and data transmission interface, configured to receive an image control instruction, wherein the image control instruction includes operating mode information and a corresponding configuration parameter set, wherein the configuration parameter set includes at least one configuration parameter combination; a pixel array unit, configured to receive image control instructions transmitted by the control and data transmission interface, and perform exposure based on the operating mode information to obtain a target image frame corresponding to each combination of configuration parameters in the configuration parameter set, wherein the target image frame is generated by projecting a corresponding light source beam into a target space based on the operating mode information; an image signal processor, configured to configure output parameters according to the configuration parameter set, and output target image frames corresponding to all combinations of the configuration parameters; The configuration parameter set is a set formed by selecting at least one configuration parameter combination from a plurality of configuration parameter combinations as a cyclic unit based on the working mode information; The configuration parameter combination includes exposure time, gain, frame length, number of transmission frames N and output parameters, wherein the set of configuration parameter combinations corresponds to matching N frames of target image frames, N is greater than or equal to 1, the frame length is used to control the inter-frame spacing between two target image frames, and the output parameters are used to control the output of the target image frame corresponding to the configuration parameter combination.

7. The image sensor according to claim 6, wherein: The image sensor further includes a register; The register is used to store the working mode information and the corresponding configuration parameter set in the image control instruction received by the control and data transmission interface for calling by the image signal processor.

8. The image sensor according to claim 7, wherein: The working mode information includes the switching control logic and cycle sequence of the laser light source and the near-infrared floodlight light source; The pixel array unit is specifically configured to obtain a near-infrared image subframe, a speckle image subframe, and a background image subframe according to the configuration parameter set, wherein the near-infrared image subframe is an image subframe obtained by turning on a near-infrared floodlight light source to project a floodlight beam toward a target space, the speckle image subframe is an image subframe obtained by turning on the laser light source to project a speckle beam toward the target space, and the background image subframe is an image subframe obtained by turning off the near-infrared floodlight light source and the laser light source and projecting ambient light toward the target space.

9. The image sensor according to claim 8, wherein: The working mode information also includes background reduction control information; when the background reduction control information is not to reduce the background, The pixel array unit is further configured to obtain the speckle image subframe and the background image subframe through global shutter exposure.

10. The image sensor according to claim 9, wherein: When the background reduction control information is background reduction, The pixel array unit is further configured to obtain a speckle image subframe through global shutter exposure and a background image subframe through rolling shutter exposure; The image signal processor is further configured to receive and perform background subtraction calculation according to the speckle image subframe and the background image subframe to obtain a background subtracted image subframe.

11. An image output system, characterized in that: include: a processor, configured to configure operating mode information and a corresponding configuration parameter set to generate an image control instruction, wherein the configuration parameter set includes at least one configuration parameter combination, the configuration parameter combination includes an exposure time, and the operating mode information includes an exposure mode; a transmitting module, configured to receive and project a corresponding light source beam to a target space based on the working mode information in the image control instruction to obtain a target image frame corresponding to all the configuration parameter combinations; a receiving module, comprising the image sensor according to any one of claims 6 to 10, configured to receive and perform exposure based on the operating mode information in the image control instruction, obtain a target image frame corresponding to each configuration parameter combination in the configuration parameter set, configure output parameters according to the configuration parameter set, and output the target image frames corresponding to all the configuration parameter combinations; The processor is specifically configured to select at least one group of configuration parameter combinations from a plurality of configuration parameter combinations as a cyclic unit to form a configuration parameter set based on the working mode information; The configuration parameter combination also includes gain, frame length, number of transmission frames N and output parameters, wherein the set of configuration parameter combinations corresponds to matching N frames of target image frames, N is greater than or equal to 1, the frame length is used to control the inter-frame spacing between two target image frames, and the output parameters are used to control the output of the target image frame corresponding to the configuration parameter combination.

12. The image output system according to claim 11, wherein: The working mode information also includes a light source control signal; the transmitting module is specifically used to receive the light source control signal and project a light source beam corresponding to the light source to the target space.

13. The image output system according to claim 12, wherein: The light source control signal includes at least two sub-light source control signals; the emission module includes the at least two sub-light sources, and is specifically used to receive the at least two sub-light source control signals and cyclically project the light source light beams corresponding to the at least two sub-light sources to the target space in a preset order.

14. The image output system according to claim 13, wherein: The light source control signal includes the laser light source and the near-infrared flood light light source switch control logic and the cycle sequence; The transmitting module includes the laser light source and the near-infrared floodlight light source, and is specifically configured to, according to the switching control logic and cycle sequence of the laser light source and the near-infrared floodlight light source, respectively turn on the near-infrared floodlight light source to project a floodlight beam toward the target space so that the receiving module obtains a near-infrared image subframe, turn on the laser light source to project a speckle beam toward the target space so that the receiving module obtains a speckle image subframe, and turn off the near-infrared floodlight light source and the laser light source to project ambient light toward the target space so that the receiving module obtains a background image subframe.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an image output program, and when the image output program is executed by the processor, the steps of the image output method according to any one of claims 1 to 5 are implemented.

Images