Camera system, image acquisition method, and related device

Abstract

A camera system, an image acquisition method, and a related device. The camera system comprises a light supplementing unit, a sensor, and a filter unit. The light supplementing unit is configured to emit non-visible light. The filter unit is configured to switch to a first mode for transmitting non-visible light or a second mode for transmitting visible light. The sensor is configured such that: when the filter unit is in the first mode, the sensor generates first image data on the basis of the time from when the non-visible light is emitted from the light supplementing unit to when the non-visible light is reflected back to the sensor, and when the filter unit is in the second mode, the sensor generates second image data on the basis of the visible light. Therefore, the camera system can generate a depth image and a raw image at an extremely short time interval on one sensor by means of fast switching of the filter unit and on the basis of directly measured time of flight, thereby avoiding the operations of time alignment and spatial registration, and reducing the complexity of the camera system.

Description

Camera system, image acquisition method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 29, 2023, with application number 202311867313.2 and application name “A camera system, image acquisition method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of camera technology, and specifically to a camera system, an image acquisition method, and related equipment. Background Art

[0003] With the rapid development of camera technology, the application of camera systems is becoming more and more extensive.

[0004] For example, in a traffic camera capture system, the camera system needs to simultaneously capture the scene's real-world image and distance information (i.e., depth image) for different objects in the scene. Currently, cameras are typically used to capture the real-world image, while radar is used to capture the depth image. Through data fusion, objects in the scene are detected in real time and their speed and other information are analyzed.

[0005] However, fusing data from independent cameras and radar requires strict temporal and spatial registration. Otherwise, the fusion effect will show significant misalignment, affecting detection and analysis. Furthermore, using two independent devices simultaneously increases the overall complexity of the camera system.

[0006] Summary of the Invention

[0007] The present invention provides a camera system for simultaneously acquiring an original image and a depth image of a scene, while reducing the complexity of the camera system. The present invention also provides a corresponding image acquisition method, a computing device, and a computer-readable storage medium.

[0008] In a first aspect, the present application provides a camera system comprising a fill light unit, a sensor, and a filter unit. The fill light unit is configured to emit non-visible light, and the filter unit is configured to switch between a first mode and a second mode, wherein the first mode is configured to transmit non-visible light, and the second mode is configured to transmit visible light. The sensor is configured to generate first image data based on the flight time of the non-visible light when the filter unit is in the first mode, and to generate second image data based on the visible light when the filter unit is in the second mode, wherein the flight time of the non-visible light is the time from when the non-visible light is emitted from the fill light unit to when it is reflected back to the sensor.

[0009] In the present application, the camera system is used to capture a shooting scene and obtain first image data and second image data of the shooting scene within a very short time interval. The filter unit can quickly switch between two states: transmitting visible light and transmitting narrow-band non-visible light. The fill light unit is a fill light source in the non-visible light band, emitting non-visible light in the form of nanosecond pulses to illuminate the shooting scene. For example, the fill light unit is a non-visible strobe light. Visible light is the ambient light in the shooting scene, that is, a naturally occurring visible light source. The sensor is used to convert the received visible light signal or non-visible light signal into a digital signal output.

[0010] In the present application, when there is a scene with insufficient ambient light, the camera system can also be additionally provided with a fill light source in the visible light band to serve as ambient light. The control unit is used to control the fill light source in the visible light band to operate in a third time period. The fill light source can also be coupled with the fill light unit to form a device.

[0011] In the present application, the camera system also includes a lens. When the filter unit is switched to the first mode and the fill light unit emits non-visible light, the non-visible light flies from the fill light unit to the shooting scene, and after being reflected by an object in the shooting scene, it passes through the lens and the filter unit in sequence and enters the sensor. At this time, the sensor uses a method of directly measuring the time of flight dToF to generate first image data based on the time from when the non-visible light is emitted from the fill light unit to when it is reflected back to the sensor.

[0012] In the present application, when the filter unit is switched to the second mode, ambient light, which is visible light, continues to illuminate the captured scene. This ambient light passes through the lens and the filter unit and enters the sensor. The sensor then generates second image data based on the photon count integration of the visible light. It should be understood that the second image data can be a single-frame image or a video image.

[0013] In this first aspect, the fill light unit is configured to emit non-visible light, and the filter unit is configured to switch between a first mode for transmitting non-visible light or a second mode for transmitting visible light. The sensor is configured to generate first image data based on the time it takes for non-visible light to be emitted from the fill light unit and reflected back to the sensor when the filter unit is in the first mode, and to generate second image data based on visible light when the filter unit is in the second mode. This camera system can thus generate a depth image and a raw image on a single sensor in extremely short time intervals by rapidly switching the filter unit and directly measuring the time of flight, avoiding the need for temporal alignment and spatial registration, and reducing the complexity of the camera system.

[0014] In a possible implementation manner of the first aspect, the camera system further includes a control unit, configured to control the fill light unit, the sensor, and the filter unit based on a preset working sequence.

[0015] In this possible implementation, a control unit may be provided in the camera system to control the working sequence of each unit in the camera system, thereby obtaining raw images and depth images in a short time interval, thereby improving the feasibility of the solution.

[0016] In a possible implementation of the first aspect, the operating sequence includes a first time period, a second time period, a third time period, and a fourth time period that are continuous and cyclic. During the first time period, the control unit is configured to control the filter unit to switch to a first mode, control the fill light unit to emit non-visible light, and control the sensor to generate first image data based on the time-of-flight of the non-visible light. During the second time period, the control unit is configured to control the filter unit to switch to a second mode and control the sensor to output the first image data. During the third time period, the control unit is configured to control the sensor to generate second image data based on visible light. During the fourth time period, the control unit is configured to control the filter unit to switch to the first mode and control the sensor to output the second image data.

[0017] In this possible implementation, the working sequence can be specifically divided into four time periods, which improves the feasibility of the solution.

[0018] In a possible implementation manner of the first aspect, the starting time period of the working sequence is the first time period or the third time period.

[0019] In this possible implementation, the working sequence can start from the first time period or the third time period, which improves the feasibility of the solution.

[0020] In a possible implementation manner of the first aspect, the filter unit is a rotating filter, and the rotating filter includes a first area and a second area, the first area is used to transmit non-visible light, and the second area is used to transmit visible light.

[0021] In this possible implementation, the filter unit is a rotating filter that can rotate rapidly. The rotating filter includes a first area that transmits visible light and a second area that transmits non-visible light, so that the filter unit can be switched between the first mode and the second mode, thereby improving the feasibility of the solution.

[0022] In a possible implementation manner of the first aspect, the rotating filter further includes a third area, and the third area is opaque.

[0023] In this possible implementation, the rotating filter may further be provided with a third light-proof area, thereby improving the feasibility of the solution.

[0024] In a possible implementation manner of the first aspect, the filter unit includes a first filter and a second filter, the first filter is used to transmit non-visible light, and the second filter is used to transmit visible light.

[0025] In this possible implementation, the filter unit can separate the first area and the second area into a first filter and a second filter, thereby improving the feasibility of the solution.

[0026] In a possible implementation manner of the first aspect, the filter unit is a liquid crystal filter.

[0027] In this possible implementation, the liquid crystal filter can adjust the arrangement of liquid crystal molecules through voltage, produce polarization modulation of light, and control the transmission of light of specific wavelengths, thereby improving the feasibility of the solution.

[0028] In a possible implementation manner of the first aspect, the first image data is a depth image, and the second image data is an original raw image.

[0029] In this possible implementation, the raw image can be a single-frame image or a video image, which can be output after image processing, thereby improving the feasibility of the solution.

[0030] A second aspect of the present application provides an image acquisition method, which is applied to a camera system, which includes a fill light unit, a sensor, a filter unit and a control unit. The method includes: within a first time period, the control unit controls the filter unit to switch to a first mode, controls the fill light unit to emit non-visible light, and controls the sensor to generate first image data based on the flight time of the non-visible light, wherein the first mode is used to transmit non-visible light, and the flight time of the non-visible light is the time from the non-visible light being emitted from the fill light unit to being reflected back to the sensor; within a second time period, the control unit controls the filter unit to switch to a second mode, and controls the sensor to output the first image data, and the second mode is used to transmit visible light; within a third time period, the control unit controls the sensor to generate second image data based on visible light; within a fourth time period, the control unit controls the filter unit to switch to the first mode, and controls the sensor to output the second image data; wherein the first time period, the second time period, the third time period and the fourth time period are continuous and cyclic.

[0031] In a possible implementation manner of the second aspect, the first image data is a depth image, and the second image data is an original raw image.

[0032] In a third aspect, the present application provides a computing device comprising a processor, a memory, and a computer-readable storage medium storing a computer program; the processor is coupled to the computer-readable storage medium, and a computer-executable instruction running on the processor, when the computer-executable instruction is executed by the processor, the processor executes the method as described in the second aspect or any possible implementation of the second aspect. Optionally, the computing device may further include an input / output (I / O) interface, and the computer-readable storage medium storing the computer program may be a memory.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the second aspect or any possible implementation of the second aspect.

[0034] In a fifth aspect, the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the second aspect or any possible implementation of the second aspect.

[0035] In a sixth aspect of the present application, a chip system is provided, which includes at least one processor and an interface for receiving data and / or signals, and at least one processor for supporting a computer device to implement the functions involved in the above-mentioned second aspect or any possible implementation of the second aspect. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the computer device. The chip system may be composed of a chip or may include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of a traffic camera capture system scenario;

[0037] FIG2 is a schematic diagram of an embodiment of a camera system provided in an embodiment of the present application;

[0038] FIG3 is a schematic diagram of the working sequence of the camera system provided in an embodiment of the present application;

[0039] FIG4 is a schematic diagram of the structure of a sensor provided in an embodiment of the present application;

[0040] 5A and 5B are schematic diagrams of another embodiment of a camera system provided in an embodiment of the present application;

[0041] FIG6 is a schematic diagram of an embodiment of an image acquisition method provided in an embodiment of the present application;

[0042] FIG7 is a schematic diagram of an embodiment of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0044] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0045] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0046] In addition, numerous specific details are provided in the detailed description below to better illustrate the present application. Those skilled in the art will appreciate that the present application can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present application.

[0047] The following is an explanation of some terminology concepts involved in the embodiments of this application.

[0048] (1) Time of flight (ToF):

[0049] A technology that measures the distance by measuring the time it takes for an object, particle or wave to travel a certain distance in a fixed medium.

[0050] (2) Single photon avalanche diode (SPAD):

[0051] A photodetection avalanche diode with single-photon detection capability.

[0052] (3) Time to digital converter (TDC):

[0053] A device commonly used to measure time intervals and convert them into a digital (binary) output.

[0054] (4) Direct measurement of time of flight (dToF):

[0055] A method or device for measuring distance by directly measuring the time it takes light to propagate through a medium. The core components of this device typically include a vertical-cavity surface-emitting laser (VCSEL) as a light source, a SPAD, and a time-dependent detector (TDC).

[0056] (5) Indirect measurement of time of flight (iToF): The working principle of the iToF camera is to illuminate the scene with modulated light of a specific waveform and infer the depth of each pixel by collecting multiple images with different phases.

[0057] (6) Depth image:

[0058] Also known as a range image, it refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value.

[0059] (7) Original (raw) image:

[0060] The "raw" data collected by the camera image sensor during exposure needs to be further converted into images or videos suitable for human eye observation habits through image processing.

[0061] The following examples illustrate the application scenarios involved in the embodiments of the present application.

[0062] With the rapid development of camera technology, the application of camera systems is becoming more and more extensive. For scenarios such as autonomous driving or traffic capture, the camera system needs to simultaneously capture the real-life image of the scene and the distance information of different objects in the scene (i.e., depth image).

[0063] As shown in FIG1 , for example, in a camera system 100 used for capturing traffic camera images, a camera 101 is usually used to obtain real-time images, and a lidar 102 is used to obtain depth images. Data fusion is used to detect the content of objects in the scene in real time and analyze information such as their speed.

[0064] However, the fusion of data from the independent camera 101 and radar 102 requires strict time alignment and spatial registration, otherwise the fusion effect will show obvious misalignment, affecting the detection and analysis results. In addition, the use of two independent devices at the same time will increase the overall complexity of the camera system 100. Based on this, an embodiment of the present application provides a camera system for simultaneously acquiring the original image and depth image of the shooting scene and reducing the complexity of the camera system. The embodiment of the present application also provides corresponding image acquisition methods, computing devices, and computer-readable storage media. Each of these is described in detail below.

[0065] The camera system provided in the embodiment of the present application is described below in combination with the above-mentioned terminology concepts and application scenarios.

[0066] As shown in FIG2 , an embodiment of the present application provides a camera system, which includes a fill light unit 110 , a sensor 120 , and a filter unit 130 .

[0067] Among them, the fill light unit 110 is used to emit non-visible light, and the filter unit 130 is used to switch to a first mode or a second mode, the first mode is used to transmit non-visible light, and the second mode is used to transmit visible light; the sensor 120 is used to generate first image data based on the flight time of non-visible light when the filter unit 130 is in the first mode, and to generate second image data based on visible light when the filter unit 130 is in the second mode. The flight time of non-visible light is the time from the non-visible light being emitted from the fill light unit 110 to being reflected back to the sensor 120.

[0068] Specifically, the camera system is used to shoot a shooting scene, and obtain first image data and second image data of the shooting scene within a very short time interval, where the first image data is a depth image and the second image data is a raw image.

[0069] The filter unit 130 can switch between a first mode for transmitting non-visible light and a second mode for transmitting visible light, i.e., it can rapidly switch between transmitting visible light and transmitting narrow-band non-visible light. The fill light unit 110 is a fill light source in the non-visible light band, emitting non-visible light in nanosecond pulses to illuminate the captured scene. For example, the fill light unit 110 is a non-visible strobe light. Visible light refers to the ambient lighting in the captured scene, i.e., naturally occurring visible light sources. The sensor 120 is used to convert received visible light signals or non-visible light signals into digital signals for output.

[0070] Optionally, the camera system also includes a lens 140. When the filter unit 130 is switched to the first mode and the fill light unit 110 emits non-visible light, the non-visible light flies from the fill light unit 110 to the shooting scene, and after being reflected by an object in the shooting scene, it passes through the lens 140 and the filter unit 130 in sequence and enters the sensor 120. At this time, the sensor 120 adopts the dToF method to generate first image data based on the time from the non-visible light being emitted from the fill light unit 110 to being reflected back to the sensor 120.

[0071] When the filter unit 130 switches to the second mode, the ambient light, which is visible light, continues to illuminate the captured scene. This light passes through the lens 140 and the filter unit 130, and then enters the sensor 120. At this point, the sensor 120 generates second image data based on the photon integration of the visible light. It should be understood that the second image data can be a raw image of a single frame or a raw image of a video.

[0072] Optionally, the camera system further includes a control unit 150, which is configured to control the fill light unit 110, the sensor 120, and the filter unit 130 based on a preset operating sequence. That is, the control unit 150 has high timing accuracy and can accurately control the fill light unit 110, the filter unit 130, and the sensor 120 to operate at a specific moment.

[0073] Exemplarily, as shown in FIG3 , the working sequence includes a first time period, a second time period, a third time period and a fourth time period that are continuous and cyclic.

[0074] During the first time period, the control unit is configured to control the filter unit to switch to the first mode, control the fill light unit to emit non-visible light, and control the sensor to generate first image data based on the time-of-flight of the non-visible light. That is, at the beginning of the first time period, the control unit simultaneously controls the filter unit, the fill light unit, and the sensor.

[0075] Specifically, the control unit controls the filter unit to switch to a narrow-band non-visible light transmission mode, i.e., the first mode. At the same time, the control unit controls the fill light unit to perform pulsed flash on the shooting scene, and the pulse half-width time is in the nanosecond level. Therefore, the fill light unit finishes working first. At the same time, the control unit controls the sensor to start counting the flight time of non-visible light photons reflected from the shooting scene from the flash to the reflection back to the sensor, calculates the peak value of the photon distribution histogram, obtains the photon flight time, and simultaneously generates a depth image of the shooting scene, i.e., the first image data.

[0076] It should be understood that the first image data is a depth image of the shooting scene within a first time period, and does not involve other time periods.

[0077] During the second time period, the control unit is used to control the filter unit to switch to the second mode and control the sensor to output the first image data. That is, when the second time period begins, the control unit controls the filter unit and the sensor to work simultaneously.

[0078] Specifically, the control unit controls the filtering unit to quickly switch from the narrow-band non-visible light transmission mode to the visible light transmission mode, that is, switches the filtering unit from the first mode to the second mode. At the same time, the control unit controls the sensor to complete the depth image, that is, the calculation of the first image data, to prepare for the next imaging.

[0079] During the third time period, the control unit is configured to control the sensor to generate the second image data based on the visible light. That is, at the beginning of the third time period, the control unit only controls the sensor to operate, and the filter unit can already receive the visible light.

[0080] Specifically, the control unit controls the sensor to start collecting visible light photons reflected by the shooting scene, and obtains a raw image of the shooting scene, i.e., the second image data, by internally integrating the number of photons within the exposure time. It should be understood that the raw image needs to be further processed to obtain a single-frame image or video image of the shooting scene.

[0081] It should be understood that the second image data is a raw image of the shooting scene in the third time period, and does not involve other time periods.

[0082] During the fourth time period, the control unit is configured to control the filter unit to switch to the first mode and control the sensor to output the second image data. That is, at the beginning of the fourth time period, the control unit controls the filter unit and the sensor to operate simultaneously.

[0083] Specifically, the control unit controls the filter unit to switch from the visible light transmission mode back to the narrow-band non-visible light transmission mode, that is, to switch the filter unit from the second mode to the first mode. At the same time, the control unit controls the sensor to complete the output of the raw image of the shooting scene, that is, the second image data, to prepare for the next imaging.

[0084] By alternating the above operations, the depth image and the raw image of the captured scene can be continuously obtained, and the interval between the depth image and the raw image is very short, that is, the interval between the first time period and the third time period is the second time period. At this time, the control unit only needs to switch the filter unit to the second mode. The second time period can be controlled to be very short, so the image content is basically aligned.

[0085] It should be understood that, during the second cycle, if the filter unit has been switched to the first mode in the fourth time period of the previous cycle, then the filter unit does not need to be switched to the first mode in the first time period of the next cycle. The filter unit only needs to be switched to the first mode in the first time period for the first time.

[0086] Optionally, the starting time period of the operating sequence is the first time period or the third time period. That is, the control unit can control the operating sequence of the camera system to cycle through the first time period, the second time period, the third time period, and the fourth time period in sequence, in which case the camera system first acquires the first image data and then the second image data. Alternatively, the control unit can control the operating sequence of the camera system to cycle through the third time period, the fourth time period, the first time period, and the second time period in sequence, in which case the camera system first acquires the second image data and then the first image data.

[0087] Exemplarily, the length of the first time period is hundreds of microseconds to several milliseconds, the length of the second time period is hundreds of microseconds to several milliseconds, the length of the third time period is several milliseconds, and the length of the fourth time period is tens of milliseconds. The specific length of each time period can be determined based on actual conditions and user needs. The embodiments of the present application do not limit the specific length of each time period.

[0088] It should be understood that the camera system may also not be provided with a control unit, but the control function may be integrated into the fill light unit, sensor and filter unit, so that the fill light unit, sensor and filter unit automatically work according to the above-mentioned working sequence, or the fill light unit, sensor and filter unit may work according to the above-mentioned working sequence through other means. For example, after the control unit controls the sensor to start counting the flight time of visible light, the sensor automatically starts to output the first image data without the need for secondary control of the control unit. The embodiments of the present application do not limit this.

[0089] Optionally, when there is a scene with insufficient ambient light, the camera system can also additionally set up a fill light source in the visible light band to serve as ambient light. The control unit is used to control the fill light source in the visible light band to operate in a third time period. The fill light source can also be coupled with the fill light unit to form a device.

[0090] Exemplarily, as shown in Figure 4, the sensor in the embodiment of the present application is a single photon avalanche diode (SPAD) area array sensor, which includes a pixel array, a readout circuit, a time to digital converter (TDC), a histogram generator and a clock generator, etc., wherein a specific filter is added above the pixel array, and the filter above each pixel array maintains high transmittance in the non-visible light band, and transmits red, green, blue bands or all of them in the visible light band.

[0091] During a first time period, the sensor collects non-visible light photons reflected from the captured scene. These photons undergo photoelectric conversion in the pixel array of the SPAD area array sensor, generating electrons. These electrons are then amplified by avalanche multiplication. The TDC receives the electrons output by the pixel array and measures the number of electrons and the time of reception. The TDC measures the time of flight of each received photon after it is reflected back to the sensor after the strobe. A histogram generator generates a time-of-flight distribution histogram of all photons at each pixel. The TDC receives a high-resolution clock signal generated by a clock generator, quantifies the number of received electrons during each clock cycle, and stores this information in a histogram generator. This ultimately yields a histogram of the number of electrons at each time interval. Finally, based on the time period of the peak reflected photon count in the histogram, the depth information of each pixel is calculated, resulting in a depth image of the captured scene.

[0092] During the third time period, the sensor begins collecting visible light photons reflected from the scene at the set starting exposure time. Each pixel is filtered through different filters to obtain photons in the red, green, blue, or full visible band. The photon count for each time period is counted and stored in a histogram generator. The histogram for the specified exposure time range is then integrated to produce a raw image of the scene. Further image processing can be performed to obtain a single-frame image or video of the scene.

[0093] In the embodiment of the present application, there are multiple possible implementations of the filter unit, which are described below respectively.

[0094] 1. The filter unit is a filter

[0095] (1) The rotating filter includes a first region and a second region

[0096] The filter unit is a rotating filter that can achieve high-speed rotation. The rotating filter includes a first area and a second area. The first area is used to transmit non-visible light, and the second area is used to transmit visible light.

[0097] Specifically, the area ratio of the first area and the second area can be adjusted according to needs. The shape of the rotating filter can be circular, fan-shaped or ring-shaped, etc. The rotating filter can be placed between the lens and the camera, or in front of the lens.

[0098] (2) The rotating filter also includes a third area

[0099] Compared with the above-mentioned case (1), the only difference of the filter unit is that the rotating filter also includes a third area, the third area is not transparent, and the filter unit can be switched to the third area in the fourth time period or the non-working time period.

[0100] For example, as shown in Figure 5A, the shooting scene is a vehicle, the fill light unit is an invisible flash light, the control unit is a controller, which is specifically a central processing unit (CPU), and the filtering unit is a rotating filter. The rotating filter includes a first area, a second area and a third area. The first area is a narrow-band non-spatial light transmission area, the second area is a visible light transmission area, and the third area is an opaque area.

[0101] (3) First filter and second filter

[0102] The filter unit includes a first filter and a second filter, the first filter is used to transmit non-visible light, and the second filter is used to transmit visible light.

[0103] Specifically, the filter unit can be understood as separating the first area and the second area in the above-mentioned case (1), with the first area serving as a separate first filter and the second area serving as a separate second filter. It should be understood that the filter unit can also include a third filter that is opaque.

[0104] 2. The filter unit is a liquid crystal filter

[0105] The liquid crystal filter can adjust the arrangement of liquid crystal molecules through voltage to produce polarization modulation of light and control the transmission of light of specific wavelengths, such as controlling the transmission of visible light or controlling the transmission of invisible light, thereby switching the filter unit to the first mode and the second mode.

[0106] For example, as shown in FIG5B , the shooting scene is a vehicle, the fill light unit is an invisible flash light, the control unit is a controller, which is specifically a CPU, and the filter unit is a liquid crystal filter, which can also be understood as a liquid crystal filter device.

[0107] In the embodiments of the present application, the camera system can be applied to traffic scenes to capture the vehicle conditions and speeds in the image. When used in traffic scenes, the camera system transmits the acquired raw images (e.g., video images) and depth images to the main control chip, where the video image processing and depth image processing are performed respectively. The main control chip then uses intelligent algorithms to identify vehicle information in the video images and calculate information such as vehicle distance and speed, which is then transmitted to the back-end platform for big data analysis and detection.

[0108] Furthermore, the camera system can also be used as an onboard vehicle detector, combining the functions of an onboard camera and a LiDAR radar to provide accurate, all-weather information about surrounding road conditions and vehicles, enabling assisted driving and autonomous driving. When used in an onboard device, the camera system transmits the acquired raw and depth images to the main control chip, where they are processed separately. An intelligent algorithm module analyzes both the depth information and the video image in real time to assist in determining whether braking or other maneuvers are necessary, and provides driver prompts via the display and voice.

[0109] It should be understood that the camera system can also be used in various scenarios that require simultaneous acquisition of raw images and depth images, such as gaming entertainment, augmented reality (AR), and three-dimensional reconstruction, and the embodiments of the present application are not limited to this.

[0110] In an embodiment of the present application, the camera system uses a fast-switching filter unit combined with a wide-spectrum SPAD sensor, and uses one lens and one sensor to simultaneously acquire raw images and depth images, avoiding the installation, alignment and other problems currently encountered when two independent devices are used to acquire raw images and depth images respectively, while reducing the complexity and equipment cost of the camera system.

[0111] In addition, a high-precision control unit controls the fill light unit, filter unit, and sensor to operate according to a preset working sequence, achieving ultra-short time intervals for raw and depth images, which is more conducive to image registration and synthesis. The depth image is acquired using dToF, which only takes a single shot to obtain the depth image. Compared to the multiple shots required by the iToF solution, the interval between the raw and depth images is shorter, and the misalignment caused by the movement of objects in the shooting scene is smaller, eliminating the need for alignment.

[0112] The camera system provided in the embodiment of the present application is introduced above. The image acquisition method and related equipment provided in the embodiment of the present application are introduced below with reference to the accompanying drawings.

[0113] As shown in FIG6 , an embodiment of the present application further provides an image acquisition method, which is applied to the camera system shown in FIG2 , wherein the camera system includes a fill light unit, a sensor, a filter unit, and a control unit. The method includes:

[0114] 601. Within a first time period, the control unit controls the filter unit to switch to a first mode, controls the fill light unit to emit non-visible light, and controls the sensor to generate first image data based on the flight time of the non-visible light.

[0115] The first mode is used to transmit non-visible light, and the flight time of the non-visible light is the time from when the non-visible light is emitted from the fill light unit to when it is reflected back to the sensor.

[0116] 602. In a second time period, the control unit controls the filter unit to switch to a second mode, and controls the sensor to output first image data.

[0117] The second mode is used to transmit visible light.

[0118] 603. In a third time period, the control unit controls the sensor to generate second image data based on visible light.

[0119] 604. In a fourth time period, the control unit controls the filter unit to switch to the first mode, and controls the sensor to output second image data.

[0120] Among them, the first time period, the second time period, the third time period and the fourth time period are continuous and cyclic, that is, step 604 can be followed by step 601, and the execution order of the method can be step 601, step 602, step 603 and step 604, or step 603, step 604, step 601 and step 602, and step 602 can be followed by step 603.

[0121] In the embodiment of the present application, the image acquisition method can refer to the working timing of the camera system as shown in Figure 3, and its specific implementation method can be understood by referring to the corresponding description in the above embodiment, and the embodiment of the present application will not be repeated here.

[0122] As shown in Figure 7, an embodiment of the present application provides a computing device 700, and Figure 7 is a schematic diagram of a possible logical structure of the computing device 700. The computing device 700 includes: a processor 701, a communication interface 702, a memory 703, and a bus 704. The processor 701 may include a CPU, or a CPU and at least one of a GPU, an NPU, and other types of processors. The processor 701, the communication interface 702, and the memory 703 are interconnected via a bus 704. In an embodiment of the present application, the processor 701 is used to control and manage the actions of the computing device 700. For example, the processor 701 is used to execute steps 601 to 604 in Figure 6 and / or other processes for the technology described herein. The communication interface 702 is used to support the computing device 700 to communicate. The memory 703 is used to store program code and data of the computing device 700.

[0123] The processor 701 may be a central processing unit (CPU), a general-purpose processor (GPOR), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device (PLD), a transistor logic device (TLD), a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The bus 704 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG. 7 shows only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0124] Exemplarily, the computing device 700 is the camera system shown in FIG. 2 , and may also be a control unit in the camera system.

[0125] In another embodiment of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When at least one processor of a device executes the computer-executable instructions, the device executes the image acquisition method described in the above embodiment.

[0126] In another embodiment of the present application, a computer program product is also provided, which includes computer execution instructions, which are stored in a computer-readable storage medium; at least one processor of the device can read the computer execution instructions from the computer-readable storage medium, and at least one processor executes the computer execution instructions so that the device executes the image acquisition method described in the above embodiment.

[0127] In another embodiment of the present application, a chip system is provided, which includes at least one processor and an interface, wherein the interface is used to receive data and / or signals, and the at least one processor is used to support the image acquisition method described in the above embodiment. In one possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the computer device. The chip system can be composed of a chip or may include a chip and other discrete devices.

[0128] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0129] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0132] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A camera system, characterized in that, The camera system includes a supplementary light unit, a sensor, and a filter unit; The supplementary light unit is configured to emit non-visible light, and the filter unit is configured to switch to a first mode or a second mode. The first mode is for transmitting the non-visible light, and the second mode is for transmitting visible light; The sensor is configured to generate first image data based on the time of flight of the non-visible light when the filter unit is in the first mode, and generate second image data based on the visible light when the filter unit is in the second mode. The time of flight of the non-visible light is the time from when the non-visible light is emitted from the supplementary light unit until it is reflected back to the sensor.

2. The camera system according to claim 1, characterized in that, The camera system further includes a control unit, which is configured to control the supplementary light unit, the sensor, and the filter unit based on a preset working timing.

3. The camera system according to claim 2, characterized in that, The working timing includes consecutive and cyclic first, second, third, and fourth time periods; During the first time period, the control unit is configured to control the filter unit to switch to the first mode, control the supplementary light unit to emit the non-visible light, and control the sensor to generate first image data based on the time of flight of the non-visible light; During the second time period, the control unit is configured to control the filter unit to switch to the second mode and control the sensor to output the first image data; During the third time period, the control unit is configured to control the sensor to generate second image data based on the visible light; During the fourth time period, the control unit is configured to control the filter unit to switch to the first mode and control the sensor to output the second image data.

4. The camera system according to claim 3, characterized in that, The starting time period of the working timing is the first time period or the third time period.

5. The camera system according to any one of claims 1-4, characterized in that, The filter unit is a rotating filter, which includes a first region and a second region. The first region is for transmitting the non-visible light, and the second region is for transmitting the visible light.

6. The camera system according to claim 5, wherein The rotating filter further includes a third region that is light-impermeable.

7. The camera system according to any one of claims 1-4, characterized in that, The filter unit includes a first filter and a second filter. The first filter is for transmitting the non-visible light, and the second filter is for transmitting the visible light.

8. The camera system according to any one of claims 1 to 4, characterized in that, The filter unit is a liquid crystal filter.

9. The camera system according to any one of claims 1-8, characterized in that, The first image data is a depth image, and the second image data is an original raw image.

10. An image acquisition method, characterized in that, The method is applied to a camera system, which includes a supplementary light unit, a sensor, a filter unit, and a control unit. The method includes: During the first time period, the control unit controls the filter unit to switch to the first mode, controls the supplementary light unit to emit non-visible light, and controls the sensor to generate first image data based on the time of flight of the non-visible light. The first mode is for transmitting the non-visible light, and the time of flight of the non-visible light is the time from when the non-visible light is emitted from the supplementary light unit until it is reflected back to the sensor; During the second time period, the control unit controls the filter unit to switch to the second mode and controls the sensor to output the first image data. The second mode is for transmitting visible light; During a third time period, the control unit controls the sensor to generate second image data based on visible light; During a fourth time period, the control unit controls the filter unit to switch to the first mode and controls the sensor to output the second image data; Wherein, the first time period, the second time period, the third time period, and the fourth time period are consecutive and cyclic.

11. The method according to claim 10, wherein The first image data is a depth image, and the second image data is a raw image.

12. A computing device, characterized in that, Comprising a processor, computer-executable instructions are run on the processor, and when the computer-executable instructions are executed by the processor, the processor executes the method according to claim 10 or 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to claim 10 or 11 is implemented.

Images