Camera system, image acquisition method and related equipment

Abstract

The invention discloses a camera system, an image acquisition method and related equipment. The camera system comprises a light supplementing unit, a sensor and a light filtering unit. Wherein the light supplementing unit is used for emitting invisible light, and the light filtering unit is used for being switched into a first mode used for transmitting the invisible light or a second mode used for transmitting the visible light. The sensor is used for generating first image data based on the time when the invisible light is emitted from the light supplementing unit to reflected back to the sensor when the light filtering unit is in the first mode, and generating second image data based on the visible light when the light filtering unit is in the second mode. Therefore, the camera system can generate a depth image and an original image on one sensor at an extremely short time interval through rapid switching of the light filtering unit and based on direct measurement of the flight time, operation of time alignment and space registration is avoided, and the complexity of the camera system is also reduced.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of cameras, and in particular to a camera system, an image acquisition method, and related devices. Background Art

[0002] With the rapid development of camera technology, the application of camera systems has become increasingly widespread.

[0003] For example, in a traffic camera capture system, the camera system needs to simultaneously acquire a real-time image of a scene and distance information (i.e., depth image) of different objects in the scene. Currently, cameras are usually used separately to acquire real-time images, and radars are used to acquire depth images. Data fusion is used to detect the object content in the scene in real time and analyze information such as its speed.

[0004] However, fusing data from independent cameras and radars requires strict time alignment and spatial registration. Otherwise, obvious misalignment phenomena will occur in the fusion effect, affecting the detection and analysis effects. In addition, using two sets of independent devices simultaneously will also increase the overall complexity of the camera system. Summary of the Invention

[0005] Embodiments of the present application provide a camera system for simultaneously acquiring an original image and a depth image of a shooting scene and reducing the complexity of the camera system. Embodiments of the present application also provide corresponding image acquisition methods, computing devices, computer-readable storage media, etc.

[0006] In a first aspect of the present application, a camera system is provided. The camera system includes a fill light unit, a sensor, and a filter unit. The fill light unit is used to emit non-visible light, and the filter unit is used to switch to a first mode or a second mode. The first mode is used to transmit non-visible light, and the second mode is used to transmit visible light. The sensor is used to generate first image data based on the flight time of non-visible light when the filter unit is in the first mode, and generate second image data based on visible light when the filter unit is in the second mode. The flight time of non-visible light is the time from when the non-visible light is emitted by the fill light unit to when it is reflected back to the sensor.

[0007] In the present application, the camera system is used to shoot a shooting scene and acquire first image data and second image data of the shooting scene within a very short time interval. The filter unit can quickly switch between two states: transmitting visible light and transmitting narrow-band non-visible light. The fill light unit is a fill light source in the non-visible light band and emits non-visible light in the form of nanosecond-level pulses to irradiate the shooting scene. For example, the fill light unit is a non-visible stroboscopic lamp. The visible light is the ambient light in the shooting scene, that is, a light source with naturally existing visible light. The sensor is used to convert the received visible light signal or non-visible light signal into a digital signal and output it.

[0008] In this application, when there is a scene with insufficient ambient light, the camera system can also be additionally provided with a supplementary light source in the visible light band to act as ambient light. The control unit is used to control the supplementary light source in the visible light band to operate in the third time period. The supplementary light source can also be coupled with the supplementary light unit into one device.

[0009] In this application, the camera system further includes a lens. When the filter unit switches to the first mode and the supplementary light unit emits non-visible light, the non-visible light travels from the supplementary light unit to the shooting scene. After being reflected by an object in the shooting scene, it passes through the lens and the filter unit in sequence and enters the sensor. At this time, the sensor uses the method of directly measuring the time of flight dToF to generate first image data based on the time from when the non-visible light is emitted from the supplementary light unit to when it is reflected back to the sensor.

[0010] In this application, after the filter unit switches to the second mode, the ambient light also irradiates the shooting scene. The ambient light is visible light, which passes through the lens and the filter unit in sequence and enters the sensor. At this time, the sensor generates second image data based on the integration of the number of photons of the visible light. It should be understood that the second image data can be a single-frame image or a video image.

[0011] In the first aspect, the supplementary light unit is used to emit non-visible light, and the filter unit is used to switch to the first mode for transmitting non-visible light or the second mode for transmitting visible light. The sensor is used to generate first image data based on the time from when the non-visible light is emitted from the supplementary light unit to when it is reflected back to the sensor when the filter unit is in the first mode, and generate second image data based on visible light when the filter unit is in the second mode. Thus, the camera system can, through the rapid switching of the filter unit and based on the direct measurement of the time of flight, generate a depth image and a raw image at a very short time interval on one sensor, avoiding the operations of time alignment and spatial registration, and also reducing the complexity of the camera system.

[0012] In a possible implementation manner of the first aspect, the camera system further includes a control unit, and the control unit is used to control the supplementary light unit, the sensor, and the filter unit based on a preset working timing.

[0013] In this possible implementation manner, a control unit can also be set in the camera system to control the working timing of each unit in the camera system, so as to obtain a raw image and a depth image at a short time interval, improving the feasibility of the solution.

[0014] In a possible implementation of the first aspect, the working timing sequence includes consecutive and cyclic first, second, third, and fourth time periods. During the first time period, the control unit is used to control the filter unit to switch to the first mode, control the supplementary light unit to emit non-visible light, and control the sensor to generate first image data based on the time of flight of the non-visible light. During the second time period, the control unit is used to control the filter unit to switch to the second mode and control the sensor to output the first image data. During the third time period, the control unit is used to control the sensor to generate second image data based on visible light. During the fourth time period, the control unit is used to control the filter unit to switch to the first mode and control the sensor to output the second image data.

[0015] In this possible implementation, the working timing sequence can be specifically divided into four time periods, improving the feasibility of the solution.

[0016] In a possible implementation of the first aspect, the starting time period of the working timing sequence is the first time period or the third time period.

[0017] In this possible implementation, the working timing sequence can start from the first time period or the third time period, improving the feasibility of the solution.

[0018] In a possible implementation of the first aspect, the filter unit is a rotating filter, and the rotating filter includes a first area and a second area. The first area is used to transmit non-visible light, and the second area is used to transmit visible light.

[0019] In this possible implementation, the filter unit is a rotating filter that can rotate quickly. The rotating filter includes a first area for transmitting visible light and a second area for transmitting non-visible light, thereby enabling the filter unit to switch between the first mode and the second mode, improving the feasibility of the solution.

[0020] In a possible implementation of the first aspect, the rotating filter further includes a third area that is opaque.

[0021] In this possible implementation, the rotating filter can also be provided with an opaque third area, improving the feasibility of the solution.

[0022] In a possible implementation of the first aspect, the filter unit includes a first filter and a second filter. The first filter is used to transmit non-visible light, and the second filter is used to transmit visible light.

[0023] In this possible implementation, the filter unit can separate the first area and the second area into the first filter and the second filter, improving the feasibility of the solution.

[0024] In a possible implementation of the first aspect, the light filtering unit is a liquid crystal light filter.

[0025] In this possible implementation, the liquid crystal light filter can adjust the arrangement of liquid crystal molecules through voltage, generate polarization modulation for light, control the transmission of light with specific wavelengths, and improve the feasibility of the solution.

[0026] In a possible implementation of the first aspect, the first image data is a depth image, and the second image data is an original raw image.

[0027] In this possible implementation, the raw image can be a single-frame image or a video image, and can be output after image processing, which improves the feasibility of the solution.

[0028] The second aspect of the present application provides an image acquisition method. This method is applied to a camera system, which includes a supplementary light unit, a sensor, a light filtering unit, and a control unit. The method includes: within a first time period, the control unit controls the light filtering unit to switch to a first mode, controls the supplementary light unit to emit non-visible light, and controls the sensor to generate first image data based on the time of flight of the non-visible light. The first mode is used to transmit non-visible light, and the time of flight of the non-visible light is the time from when the non-visible light is emitted by the supplementary light unit to when it is reflected back to the sensor; within a second time period, the control unit controls the light filtering unit to switch to a second mode and controls the sensor to output the first image data. The second mode is used to transmit visible light; within a third time period, the control unit controls the sensor to generate second image data based on visible light; within a fourth time period, the control unit controls the light filtering unit to switch to the first mode and controls the sensor to output the second image data; where the first time period, the second time period, the third time period, and the fourth time period are continuous and cyclic.

[0029] In a possible implementation of the second aspect, the first image data is a depth image, and the second image data is an original raw image.

[0030] The third aspect of the present application provides a computing device, which includes a processor, a memory, and a computer-readable storage medium storing a computer program; the processor is coupled to the computer-readable storage medium, and computer execution instructions run on the processor. When the computer execution instructions are executed by the processor, the processor executes the method as described in the above second aspect or any possible implementation of the second aspect. Optionally, the computing device may further include an input / output (I / O) interface, and the computer-readable storage medium storing the computer program may be a memory.

[0031] A fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method according to the second aspect or any possible implementation manner of the second aspect as described above.

[0032] A fifth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method according to the second aspect or any possible implementation manner of the second aspect as described above.

[0033] A sixth aspect of the present application provides a chip system. The chip system includes at least one processor and an interface. The interface is used to receive data and / or signals, and the at least one processor is used to support a computer device to implement the functions involved in the second aspect or any possible implementation manner of the second aspect as described above. In a possible design, the chip system may further include a memory for storing necessary program instructions and data of the computer device. The chip system may be composed of chips or may include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the scenario of the traffic camera capture system;

[0035] Figure 2 It is a schematic diagram of an embodiment of the camera system provided by an embodiment of the present application;

[0036] Figure 3 It is a schematic diagram of the working timing of the camera system provided by an embodiment of the present application;

[0037] Figure 4 It is a schematic diagram of the structure of the sensor provided by an embodiment of the present application;

[0038] Figure 5A and Figure 5B It is a schematic diagram of another embodiment of the camera system provided by an embodiment of the present application;

[0039] Figure 6 It is a schematic diagram of an embodiment of the image acquisition method provided by an embodiment of the present application;

[0040] Figure 7 It is a schematic diagram of an embodiment of the computing device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of them. Those of ordinary skill in the art will understand that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0042] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0043] The special term "exemplary" here means "serving as an example, embodiment or illustration". Any embodiment described as "exemplary" here does not have to be construed as superior to or better than other embodiments.

[0044] In addition, for better illustration of the present application, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present application can be implemented without some specific details. In some instances, methods, means, elements and circuits well-known to those skilled in the art are not described in detail so as to highlight the gist of the present application.

[0045] Some term concepts related to the embodiments of the present application will be explained below.

[0046] (1) Time of flight (ToF):

[0047] A technology that measures the distance by measuring the time it takes for an object, particle or wave to travel a certain distance in a fixed medium.

[0048] (2) Single photon avalanche diode (SPAD):

[0049] An avalanche photodetector diode with single photon detection ability.

[0050] (3) Time to digital converter (TDC):

[0051] A device commonly used to measure time intervals and convert them into digital (binary) outputs.

[0052] (4) Direct Time-of-Flight (dToF):

[0053] Refers to a method or device for measuring distance by directly measuring the time taken for light to travel through a medium. The core components of this device generally include a vertical-cavity surface-emitting laser (VCSEL) as the light source, a SPAD, and a TDC.

[0054] (5) Indirect Time-of-Flight (iToF): The working principle of an iToF camera is to illuminate the scene with modulated light of a specific waveform and infer the depth of each pixel by collecting multiple images with different phases.

[0055] (6) Depth image:

[0056] Also known as a range image, it refers to an image in which the distance (depth) from the image collector to each point in the scene is used as the pixel value.

[0057] (7) Raw image:

[0058] The "raw" data collected by the camera image sensor during exposure needs to be further processed through image processing to be converted into an image or video suitable for human eye viewing habits.

[0059] The following is an example of the application scenarios involved in the embodiments of this application.

[0060] With the rapid development of camera technology, the application of camera systems is becoming more and more widespread. For scenarios such as autonomous driving or traffic capture, the camera system needs to simultaneously obtain the actual captured image of the scene and the distance information (i.e., depth image) of different objects in the scene.

[0061] As Figure 1 shown, for example, in the camera system 100 for traffic camera capture, a camera 101 is usually used to obtain the actual captured image, and a lidar 102 is used to obtain the depth image. Through data fusion, the object content in the scene is detected in real time, and information such as its speed is analyzed.

[0062] However, when fusing the data from the independent camera 101 and radar 102, strict time alignment and spatial registration are required. Otherwise, obvious misalignment will occur in the fusion effect, affecting the detection and analysis results. In addition, when using two sets of independent devices simultaneously, the overall complexity of the camera system 100 will also increase. Based on this, the embodiments of the present application provide a camera system for simultaneously acquiring the original image and depth image of a shooting scene and reducing the complexity of the camera system. The embodiments of the present application also provide corresponding image acquisition methods, computing devices, computer-readable storage media, etc. The following will be described in detail respectively.

[0063] The camera system provided by the embodiments of the present application will be described below in combination with the above-mentioned term concepts and application scenarios.

[0064] As Figure 2 shown, the embodiments of the present application provide a camera system. An embodiment of the camera system includes a supplementary light unit 110, a sensor 120, and a filter unit 130.

[0065] Among them, the supplementary light unit 110 is used to emit non-visible light, and the filter unit 130 is used to switch to a first mode or a second mode. The first mode is used to transmit non-visible light, and the second mode is used to transmit visible light. The sensor 120 is used to generate first image data based on the flight time of non-visible light when the filter unit 130 is in the first mode, and generate second image data based on visible light when the filter unit 130 is in the second mode. The flight time of non-visible light is the time from when the non-visible light is emitted by the supplementary light unit 110 to when it is reflected back to the sensor 120.

[0066] Specifically, the camera system is used to shoot a shooting scene and acquire the first image data and the second image data of the shooting scene within a very short time interval. The first image data is a depth image, and the second image data is a raw image.

[0067] The filter unit 130 can switch between a first mode for transmitting non-visible light and a second mode for transmitting visible light, that is, quickly switch between two states of transmitting visible light and transmitting narrow-band non-visible light. The supplementary light unit 110 is a supplementary light source in the non-visible light band and emits non-visible light in the form of nanosecond-level pulses to irradiate the shooting scene. For example, the supplementary light unit 110 is a non-visible flash lamp. The visible light is the ambient light in the shooting scene, that is, a light source where visible light naturally exists. The sensor 120 is used to convert the received visible light signal or non-visible light signal into a digital signal for output.

[0068] Optionally, the camera system further includes a lens 140. When the filter unit 130 switches to the first mode and the supplementary light unit 110 emits non-visible light, the non-visible light travels from the supplementary light unit 110 to the shooting scene. After being reflected by an object in the shooting scene, it passes through the lens 140 and the filter unit 130 in sequence and enters the sensor 120. At this time, the sensor 120 uses the dToF method to generate first image data based on the time from when the non-visible light is emitted from the supplementary light unit 110 to when it is reflected back to the sensor 120.

[0069] After the filter unit 130 switches to the second mode, the ambient light also illuminates the shooting scene. The ambient light is visible light, which passes through the lens 140 and the filter unit 130 in sequence and enters the sensor 120. At this time, the sensor 120 generates second image data based on the photon number integration of the visible light. It should be understood that the second image data can be the raw image of a single-frame image or the raw image of a video image.

[0070] Optionally, the camera system further includes a control unit 150, which is used to control the supplementary light unit 110, the sensor 120, and the filter unit 130 based on a preset working timing. That is, the control unit 150 has high time accuracy and can accurately control the supplementary light unit 110, the filter unit 130, and the sensor 120 to work at specific moments.

[0071] Exemplarily, as Figure 3 shown, the working timing includes consecutive and cyclic first, second, third, and fourth time periods.

[0072] During the first time period, the control unit is used to control the filter unit to switch to the first mode, control the supplementary light unit to emit non-visible light, and control the sensor to generate first image data based on the flight time of the non-visible light. That is, at the start of the first time period, the control unit simultaneously controls the filter unit, the supplementary light unit, and the sensor to work.

[0073] Specifically, the control unit controls the filter unit to switch to the transmission mode of narrow-band non-visible light, that is, the first mode. At the same time, the control unit controls the supplementary light unit to perform pulsed flash on the shooting scene. The pulse full width at half maximum time is in the nanosecond level. Therefore, the supplementary light unit finishes working first. At the same time, the control unit controls the sensor to start counting the flight time of the non-visible light photons reflected by the shooting scene from the flash to the reflection back to the sensor, calculates the peak of the photon distribution histogram, obtains the photon flight time, and simultaneously generates the depth image of the shooting scene, that is, the first image data.

[0074] It should be understood that the first image data is the depth image of the shooting scene during the first time period and does not involve other time periods.

[0075] During the second time period, the control unit is configured to control the filter unit to switch to the second mode and control the sensor to output the first image data. That is, at the beginning of the second time period, the control unit controls the filter unit and the sensor to work simultaneously.

[0076] Specifically, the control unit controls the filter unit to quickly switch from the narrow-band non-visible light transmission mode to the visible light transmission mode, that is, to switch the filter unit from the first mode to the second mode. At the same time, the control unit controls the sensor to complete the calculation of the depth image, that is, the first image data, to prepare for the next imaging.

[0077] During the third time period, the control unit is configured to control the sensor to generate the second image data based on visible light. That is, at the beginning of the third time period, the control unit only controls the sensor to work. At this time, the filter unit can already receive visible light.

[0078] Specifically, the control unit controls the sensor to start collecting visible light photons reflected by the shooting scene, and obtains the raw image of the shooting scene, that is, the second image data, by the number of photons within the internal integration exposure time. It should be understood that this raw image needs to be further processed through image processing to obtain a single-frame image or video image of the shooting scene.

[0079] It should be understood that the second image data is the raw image of the shooting scene during the third time period and does not involve other time periods.

[0080] During the fourth time period, the control unit is configured to control the filter unit to switch to the first mode and control the sensor to output the second image data. That is, at the beginning of the fourth time period, the control unit controls the filter unit and the sensor to work simultaneously.

[0081] Specifically, the control unit controls the filter unit to switch back from the visible light transmission mode to the narrow-band non-visible light transmission mode, that is, to switch the filter unit from the second mode to the first mode. At the same time, the control unit controls the sensor to complete the output of the raw image of the shooting scene, that is, the second image data, to prepare for the next imaging.

[0082] By alternately performing the above operations, the depth image and the raw image of the shooting scene can be continuously obtained, and the interval between the depth image and the raw image is very short. That is, the interval between the first time period and the third time period is the second time period. At this time, the control unit only needs to switch the filter unit to the second mode, and the second time period can be controlled very short, so the image content is basically aligned.

[0083] It should be understood that during the second cycle of operation, if the filter unit has been switched to the first mode in the fourth time period of the previous cycle, then in the first time period of the next cycle, there is no need to switch the filter unit to the first mode. Only for the first time is it necessary to switch the filter unit to the first mode in the first time period.

[0084] Optionally, the starting time period of the working timing sequence is the first time period or the third time period. That is, the control unit can control the working timing sequence of the camera system to cycle in the order of the first time period, the second time period, the third time period, and the fourth time period. At this time, the camera system first acquires the first image data and then acquires the second image data. It can also cycle in the order of the third time period, the fourth time period, the first time period, and the second time period. At this time, the camera system first acquires the second image data and then acquires the first image data.

[0085] Exemplarily, the duration of the first time period is from several hundred microseconds to several milliseconds, the duration of the second time period is from several hundred microseconds to several milliseconds, the duration of the third time period is several milliseconds, and the duration of the fourth time period is several tens of milliseconds. The specific duration of each time period can be determined according to the actual situation and user requirements. The embodiments of the present application do not limit the specific duration of each time period.

[0086] It should be understood that the camera system may not be provided with a control unit, and the control function may be integrated inside the light supplement unit, the sensor, and the filter unit, so that the light supplement unit, the sensor, and the filter unit automatically work according to the above working timing sequence, or the light supplement unit, the sensor, and the filter unit work according to the above working timing sequence in other ways. For example, after the control unit controls the sensor to start counting the flight time of non-visible light, the sensor starts to output the first image data by itself without the secondary control of the control unit. The embodiments of the present application do not limit this.

[0087] Optionally, when there is a scene with insufficient ambient light, the camera system can also be additionally provided with a light source for supplementing visible light to act as ambient light. The control unit is used to control the light source for supplementing visible light to work in the third time period. The light source for supplementing visible light can also be coupled with the light supplement unit into one device.

[0088] Exemplarily, as Figure 4 shown, the sensor in the embodiments of the present application is a single-photon avalanche diode (SPAD) area array sensor. The SPAD area array sensor includes a pixel array, a readout circuit, a time-to-digital converter (TDC), a histogram generator, a clock generator, etc. A specific filter is added above the pixel array. The filter above each pixel array has a high transmittance in the non-visible light band and transmits the red, green, and blue bands or all of them in the visible light band.

[0089] During the first time period, the sensor collects non-visible light photons reflected by the shooting scene. The non-visible light photons generate photoelectric conversion in the pixel array of the SPAD area array sensor to obtain electrons, and the electrons are amplified by avalanche multiplication. The TDC receives the electrons output by the pixel array and measures the number of electrons and the reception time. The TDC measures the flight time of each photon reflected back to the sensor after the flash, and obtains the flight time distribution histogram of all photons of each pixel through the histogram generator, that is, the TDC receives the high-time-resolution clock signal generated by the clock generator, quantifies the number of received electrons within each clock cycle, and stores them through the histogram generator. Finally, the electron number histogram of each time interval is obtained at different time intervals. Finally, according to the time period where the peak of the number of reflected photons in the histogram is located, the depth information of each pixel position can be calculated, and the depth image of the shooting scene can be obtained.

[0090] During the third time period, the sensor starts to collect visible light photons reflected by the shooting scene at the set starting exposure time point. Each pixel filters through different filters to obtain photons in the red, green, blue, or full visible light band respectively, and the number of photons in each time period is counted and stored in the histogram generator. By integrating the histogram within a specific exposure duration range, a raw image of the shooting scene can be obtained. Further, a single-frame image or a video image of the scene can be obtained through image processing.

[0091] In the embodiments of the present application, there are various possible implementation manners for the filter unit, which will be described separately below.

[0092] I. The filter unit is a filter

[0093] (1) The rotating filter includes a first region and a second region

[0094] The filter unit is a rotating filter, which can rotate at high speed. The rotating filter includes a first region and a second region. The first region is used to transmit non-visible light, and the second region is used to transmit visible light.

[0095] Specifically, the area ratio of the first region and the second region can be adjusted according to requirements. The shape of the rotating filter can be circular, fan-shaped, or annular, etc. The rotating filter can be placed between the lens and the camera or in front of the lens.

[0096] (2) The rotating filter further includes a third region

[0097] Compared with the above case (1), the difference of the filter unit is only that the rotating filter further includes a third region, and the third region is opaque. The filter unit can be switched to the third region during the fourth time period or the non-working time period.

[0098] Exemplarily, such as Figure 5AAs shown in the figure, the shooting scene is a vehicle, the supplementary light unit is a non-visible strobe light, the control unit is a controller, specifically a central processing unit (CPU), the filtering unit is a rotating filter, and the rotating filter includes a first area, a second area, and a third area. The first area is a narrow-band non-spatial light transmission area, the second area is a visible light transmission area, and the third area is an opaque area.

[0099] (3) The first filter and the second filter

[0100] The filtering unit includes a first filter and a second filter. The first filter is used to transmit non-visible light, and the second filter is used to transmit visible light.

[0101] Specifically, the filtering unit can be understood as separating the first area and the second area in the above-mentioned first case. The first area serves as a separate first filter, and the second area serves as a separate second filter. It should be understood that the filtering unit may also include a third filter, which is opaque.

[0102] II. The filtering unit is a liquid crystal filter

[0103] The liquid crystal filter can adjust the arrangement of liquid crystal molecules through voltage, generate polarization modulation for light, and control the transmission of specific wavelength light. For example, it can control the transmission of visible light or non-visible light, so as to realize the switching of the filtering unit between the first mode and the second mode.

[0104] Exemplarily, as Figure 5B shown in the figure, the shooting scene is a vehicle, the supplementary light unit is a non-visible strobe light, the control unit is a controller, specifically a CPU, and the filtering unit is a liquid crystal filter, which can also be understood as a liquid crystal filtering device.

[0105] In the embodiments of the present application, the camera system can be applied to a traffic scene to capture vehicle conditions and vehicle speeds in the captured image. When used in a traffic scene, the camera system transmits the acquired raw image (such as a video image) and depth image to the main control chip, where video image processing and depth image processing are respectively completed. Then, vehicle information is identified on the video image through an intelligent algorithm, and information such as vehicle distance and vehicle speed is calculated and transmitted to the backend platform for big data analysis and detection, etc.

[0106] In addition, the camera system can also be used in automotive vehicle detectors, serving as both a vehicle-mounted camera and a lidar, to accurately provide information about the surrounding road conditions and vehicles all-weather for scenarios such as assisted driving and autonomous driving. When used in vehicle-mounted devices, the camera system transmits the acquired raw images and depth images to the main control chip, where video image processing and depth image processing are respectively completed. The intelligent algorithm module analyzes the depth information in real time and analyzes the video images for auxiliary judgment to determine whether to perform operations such as braking, and prompts the driver through a display screen, voice, etc.

[0107] It should be understood that the camera system can also be used in various scenarios that require simultaneous acquisition of raw images and depth images, such as game entertainment, Augmented Reality (AR), and 3D reconstruction. The embodiments of this application do not limit this.

[0108] In the embodiments of this application, the camera system uses a fast-switching filter unit combined with a wide-spectrum SPAD sensor, and uses one lens and one sensor to simultaneously acquire raw images and depth images, avoiding problems such as installation and registration when currently using two independent devices to separately acquire raw images and depth images, and at the same time reducing the complexity and equipment cost of the camera system.

[0109] In addition, a control unit with high time accuracy controls the light supplement unit, the filter unit, and the sensor to work according to a preset working timing sequence, so as to obtain raw images and depth images within an ultra-short time interval, which is more conducive to image registration and synthesis. Among them, the depth image is acquired using dToF, and the depth image can be obtained with only one shot. Compared with the multiple shots of the iToF scheme, the interval between the raw image and the depth image is shorter, and the misalignment caused by the movement of objects in the shooting scene is smaller, and alignment is not required.

[0110] The above introduces the camera system provided by the embodiments of this application. Next, the image acquisition method and related devices provided by the embodiments of this application will be introduced with reference to the accompanying drawings.

[0111] As Figure 6 shown, the embodiments of this application also provide an image acquisition method, which is applied to the camera system as Figure 2 shown. The camera system includes a light supplement unit, a sensor, a filter unit, and a control unit. The method includes:

[0112] 601. In a first time period, the control unit controls the filter unit to switch to a first mode, controls the light supplement unit to emit non-visible light, and controls the sensor to generate first image data based on the flight time of the non-visible light.

[0113] Among them, the first mode is used to transmit non-visible light, and the flight time of the non-visible light is the time from when the non-visible light is emitted by the supplementary light unit to when it is reflected back to the sensor.

[0114] 602. During the second time period, the control unit controls the filter unit to switch to the second mode and controls the sensor to output the first image data.

[0115] Among them, the second mode is used to transmit visible light.

[0116] 603. During the third time period, the control unit controls the sensor to generate the second image data based on the visible light.

[0117] 604. During the fourth time period, the control unit controls the filter unit to switch to the first mode and controls the sensor to output the second image data.

[0118] Among them, the first time period, the second time period, the third time period, and the fourth time period are continuous and cyclic, that is, step 604 can be followed by step 601, and the execution order of this method can be step 601, step 602, step 603, and step 604, or step 603, step 604, step 601, and step 602, and step 602 can be followed by step 603.

[0119] In the embodiments of the present application, the image acquisition method can refer to the Figure 3 working timing of the camera system shown. Its specific implementation can be understood with reference to the corresponding descriptions in the above embodiments, and the embodiments of the present application will not be elaborated here.

[0120] As Figure 7 shown, the embodiments of the present application provide a computing device 700, Figure 7 which is a possible logical structure diagram of the computing device 700. The computing device 700 includes: a processor 701, a communication interface 702, a memory 703, and a bus 704. The processor 701 may include a CPU, or at least one of a CPU, a GPU, an NPU, and other types of processors. The processor 701, the communication interface 702, and the memory 703 are interconnected through the bus 704. In the embodiments of the present application, the processor 701 is used to control and manage the actions of the computing device 700. For example, the processor 701 is used to execute Figure 6 steps 601 to 604 in and / or other processes for the technologies described herein. The communication interface 702 is used to support the computing device 700 to communicate. The memory 703 is used to store the program code and data of the computing device 700.

[0121] Among them, the processor 701 may be a central processing unit, a general-purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. The bus 704 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 only a thick line is shown in Figure 7 , but it does not mean that there is only one bus or one type of bus.

[0122] Exemplarily, the computing device 700 is Figure 2 the camera system shown, or may also be a control unit in the camera system.

[0123] In another embodiment of the present application, there is also provided a computer-readable storage medium storing computer-executable instructions. When at least one processor of the device executes the computer-executable instructions, the device executes the image acquisition method described in the above embodiments.

[0124] In another embodiment of the present application, there is also provided a computer program product including computer-executable instructions stored in a computer-readable storage medium; at least one processor of the device can read the computer-executable instructions from the computer-readable storage medium, and the execution of the computer-executable instructions by at least one processor causes the device to execute the image acquisition method described in the above embodiments.

[0125] In another embodiment of the present application, there is also provided a chip system including at least one processor and an interface for receiving data and / or signals, and at least one processor for supporting the implementation of the image acquisition method described in the above embodiments. In a possible design, the chip system may further include a memory for storing necessary program instructions and data of the computer device. The chip system may be composed of chips or may include chips and other discrete devices.

[0126] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the embodiments of this application.

[0127] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, the functional units in the various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0131] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, read-only memory), random access memories (RAM, random access memory), magnetic disks, or optical discs.

Claims

1. A camera system, characterized in that, The camera system includes a supplementary light unit, a sensor, and a filter unit; The supplementary light unit is used to emit non-visible light, and the filter unit is used to switch to a first mode or a second mode. The first mode is used to transmit the non-visible light, and the second mode is used to transmit visible light; The sensor is used to generate first image data based on the time of flight of the non-visible light when the filter unit is in the first mode, and generate second image data based on the visible light when the filter unit is in the second mode. The time of flight of the non-visible light is the time from when the non-visible light is emitted by the supplementary light unit to when it is reflected back to the sensor.

2. The camera system according to claim 1, characterized in that, The camera system further includes a control unit, which is used to control the supplementary light unit, the sensor, and the filter unit based on a preset working timing.

3. The camera system according to claim 2, characterized in that, The working timing includes consecutive and cyclic first, second, third, and fourth time periods; During the first time period, the control unit is used to control the filter unit to switch to the first mode, control the supplementary light unit to emit the non-visible light, and control the sensor to generate first image data based on the time of flight of the non-visible light; During the second time period, the control unit is used to control the filter unit to switch to the second mode and control the sensor to output the first image data; During the third time period, the control unit is used to control the sensor to generate second image data based on the visible light; During the fourth time period, the control unit is used to control the filter unit to switch to the first mode and control the sensor to output the second image data.

4. The camera system according to claim 3, wherein The starting time period of the working timing is the first time period or the third time period.

5. The camera system according to any one of claims 1-4, characterized in that, The filter unit is a rotating filter, which includes a first area and a second area. The first area is used to transmit the non-visible light, and the second area is used to transmit the visible light.

6. The camera system according to claim 5, wherein The rotating filter further includes a third area, which is light-impermeable.

7. The camera system according to any one of claims 1-4, characterized in that, The filter unit includes a first filter and a second filter. The first filter is used to transmit the non-visible light, and the second filter is used to transmit the visible light.

8. The camera system according to any one of claims 1-4, characterized in that, The filter unit is a liquid crystal filter.

9. The camera system according to any one of claims 1-8, characterized in that, The first image data is a depth image, and the second image data is an original raw image.

10. An image acquisition method, characterized in that, The method is applied to a camera system, which includes a supplementary light unit, a sensor, a filter unit, and a control unit. The method includes: During the first time period, the control unit controls the filter unit to switch to the first mode, controls the supplementary light unit to emit non-visible light, and controls the sensor to generate first image data based on the time of flight of the non-visible light. The first mode is used to transmit the non-visible light. The time of flight of the non-visible light is the time from when the non-visible light is emitted by the supplementary light unit to when it is reflected back to the sensor; During the second time period, the control unit controls the filter unit to switch to the second mode and controls the sensor to output the first image data. The second mode is used to transmit visible light; During a third time period, the control unit controls the sensor to generate second image data based on visible light; During a fourth time period, the control unit controls the filter unit to switch to the first mode and controls the sensor to output the second image data; Wherein, the first time period, the second time period, the third time period and the fourth time period are consecutive and cyclic.

11. The method according to claim 10, characterized in that, The first image data is a depth image, and the second image data is a raw image.

12. A computing device, characterized in that, Comprising a processor, computer-executable instructions are run on the processor, and when the computer-executable instructions are executed by the processor, the processor executes the method according to claim 10 or 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to claim 10 or 11 is implemented.

Images