Camera devices with integrated heterogeneous video
By introducing time-division and frequency-division mode light source switching and filtering technology into the camera, the problem that existing cameras cannot simultaneously capture two-dimensional and three-dimensional images has been solved, achieving efficient image imaging in different environments and reducing equipment complexity and cost.
Patent Information
- Application Number
- CN202210285928.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-03-25
- Filing Date
- 2022-03-22
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-03-22
AI Technical Summary
Existing cameras struggle to capture high-quality 2D and 3D images simultaneously in both bright and dark environments with a single camera, necessitating the use of multiple cameras or complex lighting configurations.
Camera devices employing time-division and frequency-division modes generate two-dimensional and three-dimensional image signals by switching light sources at different times or wavelengths using a single camera. By utilizing the alternating operation of 2D and 3D light sources, combined with filters and the filter unit of the image sensor, time-division or frequency-division imaging of images can be achieved.
It enables the use of a single camera to capture high-quality 2D and 3D images in both bright and dark environments, reducing camera installation space and cost.
Smart Images

Figure CN115134518B_ABST
Abstract
Description
[0001] Cross-reference to related applications
[0002] This application claims the benefit of priority to Korean Patent Application No. 10-2021-0038925, filed on March 25, 2021, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field
[0003] The following description pertains to camera devices with integrated heterogeneous video. Background Technology
[0004] Typically, interior cameras for vehicles use both IR (infrared) and color (e.g., RGB) images to identify objects inside the vehicle, even in dark or low-light environments. Therefore, even in very low light conditions, people or objects inside the vehicle can be accurately identified in complete darkness using LED light sources that emit infrared (IR) light invisible to the human eye or vertical-cavity surface-emitting laser (VCSEL) light sources, without the need for visible light illumination.
[0005] In addition, (3D) cameras can be used to more accurately identify people or objects.
[0006] For example, stereo cameras, structured light cameras, and time-of-flight (TOF) cameras are 3D cameras that measure the pixel depth of each image sensor to form a 3D image.
[0007] Typically, two or more cameras are needed to simultaneously recognize two-dimensional (2D) color images and 3D images.
[0008] Meanwhile, recently, vehicle cameras need to be able to identify people or objects indoors, need to be small in size, and also need to be able to acquire as many 2D and 3D images as possible simultaneously.
[0009] For example, an RGB / IR image sensor can capture two types of images, including 2D color images and IR images. An RGB / IR image sensor consists of four pixels, three of which are RGB pixels, while the remaining pixels are IR pixels.
[0010] Cameras with RGB / IR image sensors can include a dual-pass filter that simultaneously transmits light in both the RGB and IR bands. Therefore, the camera can capture bright RGB color images indoors and can also activate an invisible IR light source to capture IR images indoors even without lights, thus enabling the identification of people or objects in dark environments.
[0011] However, since two cameras are needed to simultaneously recognize 2D and 3D images, the problem is that it may be difficult to use a single camera to simultaneously image 2D and 3D images.
[0012] For example, stereoscopic 3D cameras consist of two cameras. A TOF 3D camera requires a dedicated TOF sensor capable of measuring the arrival time of reflected light. Currently, because TOF sensors struggle to recognize all RGB color images, a separate camera is needed to capture 2D images. Furthermore, structured light 3D cameras can acquire 3D images by using an IR light source (VCSEL) with a specific pattern.
[0013] However, if an IR light source with a specific pattern is used, there is a drawback that the 2D IR image does not have a normal appearance due to the pattern.
[0014] As mentioned above, the problem with existing cameras is that it is impossible to capture both 2D and 3D images using a single camera. Summary of the Invention
[0015] The summary portion of this invention is intended to provide a brief overview of the chosen inventive concepts, which will be further described in the detailed description portion below. This summary portion is not intended to identify key or essential features of the claimed subject matter, nor to help determine the scope of the claimed subject matter.
[0016] A camera device with integrated heterogeneous video was disclosed, which is capable of imaging both two-dimensional (2D) and three-dimensional (3D) images not only in bright environments but also in dark environments using a single camera.
[0017] In one general aspect, a camera device with integrated heterogeneous video includes: a two-dimensional (2D) light source configured to generate a first light for 2D imaging and illuminate an object; a three-dimensional (3D) light source configured to generate a second light for 3D imaging and illuminate an object; a camera sensor configured to perform time-division mode operation, the time-division mode operation being divided into a first time and a second time, to image an object illuminated by the first light at the first time to generate a first image signal, and to image an object illuminated by the second light at the second time to generate a second image signal; and a controller configured to control the synchronization of the 2D light source and the camera sensor during the time-division mode operation, and to control the synchronization of the 3D light source and the camera sensor.
[0018] The controller can be configured to control the 2D light source to work at the first moment to generate the first light.
[0019] The controller can be configured to control the 3D light source to work at a second time to generate a second light.
[0020] The camera sensor may include: a filter configured to allow a first or third light (which is visible light) and a second light to pass through it; and an image sensor configured to sense the first or third light and the second light incident through the filter and to generate a first image signal and a second image signal.
[0021] The filter may include: a first filter unit configured to allow a first light or a third light to pass through; and a second filter unit configured to allow a second light to pass through.
[0022] The camera sensor may include an infrared (IR) sensor for time-division mode operation, and the image sensor may include: a first pixel unit configured to sense first light incident through a first filter unit of a filter; and a second pixel unit configured to sense second light incident through a second filter unit of a filter.
[0023] The controller can be configured to disable 2D light sources and enable 3D light sources in frequency division mode operation.
[0024] The camera sensor may include an RGB / IR sensor for frequency division mode operation, and the image sensor may include: a first pixel unit configured to sense third light incident through a first filter unit of a filter; and a second pixel unit configured to sense second light incident through a second filter unit of a filter.
[0025] In another general aspect, a camera device with integrated heterogeneous video includes: a 3D light source configured to generate a second light different from visible light for 3D imaging and to illuminate an object using the second light; a camera sensor configured to perform frequency division mode operation, imaging an object illuminated with visible light to generate a first image signal, and imaging an object illuminated with the second light to generate a second image signal; and a controller configured to control the camera sensor to generate the first and second image signals during frequency division mode operation.
[0026] The camera device may include a 2D light source configured to generate first light for 2D imaging and to illuminate an object, and a controller configured to disable the 2D light source during frequency division mode operation.
[0027] The controller can be configured to enable 3D light sources and generate a second light during frequency division mode operation.
[0028] The camera sensor may include: a filter configured to allow visible light and second light to pass through; and an image sensor configured to sense the visible light and second light incident through the filter and generate a first image signal and a second image signal.
[0029] The filter may include: a first filter unit configured to allow the visible light to pass through; and a second filter unit configured to allow a second light to pass through.
[0030] The camera sensor may include an RGB / IR sensor for frequency division mode operation, and the RGB / IR sensor may be configured to photograph an object illuminated by visible light on it to generate a first image signal in frequency division mode operation, and to photograph an object illuminated by a second light on it to generate a second image signal.
[0031] The camera sensor may include an RGB / IR sensor for frequency division mode operation, and the image sensor may include: a first pixel unit configured to sense visible light incident through a first filter unit of a filter; and a second pixel unit configured to sense second light incident through a second filter unit of a filter.
[0032] The second filter unit can be configured to allow the passage of a second light, which is an IR light of 940 nm, and the first filter unit can be configured to allow the passage of visible light with a wavelength shorter than that of the second light.
[0033] In another general aspect, the camera device includes: a two-dimensional (2D) light source configured to generate first light for 2D imaging and illuminate an object; a three-dimensional (3D) light source configured to generate second light for 3D imaging and illuminate an object; a camera sensor configured to selectively perform one of time-division mode operation and frequency-division mode operation, in which the first light illuminates the object at a first time to generate a first image signal and the second light illuminates the object at a second time to generate a second image signal, and in the frequency-division mode operation, visible light illuminates the object to generate the first image signal and the second light illuminates the object to generate the second image signal; and a controller configured to selectively enable and disable the 2D light source and the 3D light source based on whether the camera sensor performs time-division mode operation or frequency-division mode operation.
[0034] Other features and aspects will become apparent from the appended claims, the accompanying drawings, and the detailed description below. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the camera device used in the example.
[0036] Figure 2 This is a schematic diagram of the camera device used in the example.
[0037] Figure 3 This is a schematic diagram of the camera device used in the example.
[0038] Figure 4 This is a schematic diagram of the time-division mode of the camera device.
[0039] Figure 5 This is a schematic diagram of a camera sensor.
[0040] Figure 6 This is a schematic diagram of the frequency division mode (or wavelength division mode) of a camera device.
[0041] Figure 7 yes Figure 5 A schematic diagram of the camera sensor.
[0042] Throughout the accompanying drawings and detailed embodiments, the same reference numerals refer to the same elements. For purposes of clarity, illustration, and convenience, the drawings may not be drawn to scale, and the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation
[0043] The following detailed embodiments are provided to help readers gain a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will be readily apparent to those skilled in the art. The order of operations described herein is merely illustrative and is not limited to the order set forth herein, except for operations that must occur in a specific order, and may be varied, as will be apparent to those skilled in the art. Furthermore, for clarity and conciseness, descriptions of functions and structures well-known to those skilled in the art may be omitted.
[0044] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0045] It should be noted that in this document, the use of the term "may" (e.g., regarding what an example or implementation may include or implement) with respect to examples or implementations means that there exists at least one example or implementation that includes or implements such features, and not all examples and implementations are limited thereto.
[0046] Throughout this specification, when an element such as a layer, region, or substrate is described as being "on," "connected to," or "attached to" another element, the element may be directly "on," directly "connected to," or directly "attached to" the other element, or there may be one or more other elements between the element and the other element. Conversely, when an element is described as being "directly on," "directly connected to," or "directly attached to" another element, there are no other elements between the element and the other element.
[0047] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more items.
[0048] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited by these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, first part, first region, first layer, or first section mentioned in these examples may also be referred to as a second component, second part, second region, second layer, or second section.
[0049] Spatial relative terms such as “above,” “above,” “below,” and “below” may be used herein for descriptive convenience to describe the relationship of one element relative to another, as shown in the accompanying drawings. In addition to covering the orientation depicted in the drawings, these spatial relative terms are intended to also cover different orientations of the device in use or operation. For example, if the device in the drawings is flipped, an element described as being “above” or “above” another element would be located “below” or “below” that other element. Thus, depending on the spatial orientation of the device, the term “above” covers both orientations of “above” and “below”. The device may also be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatial relative terms used herein should be interpreted accordingly.
[0050] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the articles “a,” “an,” and “the” are intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the stated features, numbers, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, components, elements, and / or combinations thereof.
[0051] Variations in the shapes shown in the figures may occur due to manufacturing techniques and / or tolerances. Therefore, the examples described herein are not limited to the specific shapes shown in the figures, but include shape variations that may occur during manufacturing.
[0052] The features of the examples described herein can be combined in various ways that will become apparent after understanding the disclosure of this application. Furthermore, although the examples described herein have multiple configurations, other configurations that will become apparent after understanding the disclosure of this application are also possible.
[0053] Figure 1 This is a schematic diagram of the camera device used in the example.
[0054] Reference Figure 1 The camera device 10 may include a 2D light source 100, a 3D light source 200, a camera sensor 300, and a controller 400.
[0055] The 2D light source 100 can generate a first light LT1 for 2D imaging in response to the control signal SC1 of the controller 400, and illuminate the object 1 with the first light LT1.
[0056] The 3D light source 200 can generate a second light LT2 for 3D imaging in response to the control signal SC2 of the controller 400, and illuminate the object 1 with the second light LT2.
[0057] For example, the 3D light source 200 may be an infrared vertical-cavity surface-emitting laser (IR VCSEL) that illuminates structured light, which is a patterned light used for 3D imaging.
[0058] For example, during indoor low-light imaging, the method of first imaging a 2D image of one frame and then imaging a 3D image of the next frame is repeated, and at this time, in order to capture a 2D IR image during one frame, a general infrared light-emitting diode (IR LED) light source can be illuminated using a 2D light source 100, and structured light (an IR light source with a pattern) can be illuminated to capture a 3D image during the next frame.
[0059] In response to the control of the controller 400, the camera sensor 300 can determine one of the frequency division mode or the time division mode, execute the time division mode or the frequency division mode, and capture an image of an object 1 illuminated by a first light LT1 or visible light to generate a first image signal SV1, and capture an image of an object 1 illuminated by a second light LT2 to generate a second image signal SV2.
[0060] The controller 400 can execute frequency division mode or time division mode based on an externally input operation mode signal to control the 2D light source 100, the 3D light source 200, and the camera sensor 300, so as to suit each mode. For example, the operation mode signal can be based on an illuminance signal or a user selection signal.
[0061] During time-division mode operation, the controller 400 may, for example, control the synchronization of the 2D light source 100 and the camera sensor 300 to operate at a determined first time, and control the synchronization of the 3D light source 200 and the camera sensor 300 to operate at a determined second time.
[0062] Controller 400 can, for example, disable 2D light source 100 and enable 3D light source 200 during frequency division mode operation.
[0063] For example, during time-division mode operation, the first light LT1 generated by the 2D light source 100 may be IR-illuminated to correspond to 30 frames per second (fps), and the second light LT2 generated by the 3D light source 200 may be structured light IR-illuminated to correspond to 30 fps.
[0064] In this case, the camera sensor 300 may include a first image signal SV1 with 30fps for 2D and a second image signal SV2 with 30fps for 3D, to output an image signal SV1+SV2 with 60fps.
[0065] For example, camera sensor 300 can generate image signals at 60fps and can include a global shutter.
[0066] On the other hand, if the room is sufficiently bright and the 2D image (RGB color image without IR) is adequate, imaging in time-division mode is unnecessary. Typically, IR images use wavelength bands of 850nm or 940nm, which are less susceptible to interference from RGB images using wavelength bands less than or equal to 700nm. In this case, frequency-division imaging is possible. Specifically, 2D RGB color images and 3D IR images can be captured simultaneously in each frame because the 940nm structured light (a patterned IR light source) has less impact on the RGB color image.
[0067] For example, using the same image sensor at 60fps, 2D images can be obtained at 30fps and 3D images at 30fps in a dark environment, while 2D images and 3D images can be obtained at 60fps and 3D images at 60fps in a bright environment. This is because structured light is used to illuminate all frames; 2D color images are obtained via RGB pixels using visible light as natural light, while 3D images are obtained via IR pixels using structured IR rays.
[0068] For example, to completely prevent RGB images from being affected by structured IR rays, 2D and 3D images can be captured separately at 30fps and through the aforementioned time-division imaging.
[0069] Furthermore, if acquiring high-speed 2D and 3D images simultaneously, even in low light, results in 30fps being insufficient and 60fps being required, then methods exist for correcting the 2D image. In this case, conventional IR light sources are not used, and only structured light IR light sources are used for all frames. In this scenario, for the 2D IR image, an image tinted with an IR pattern can be obtained, and the conventional 2D image can be recovered and used by image processing using the tinted image.
[0070] For each figure discussed herein, unnecessary redundant descriptions of parts with the same reference numerals and the same function may be omitted, and possible differences may be described for each figure.
[0071] Figure 2 This is a schematic diagram of the camera device used in the example.
[0072] exist Figure 2 In the camera device 10 shown, the camera sensor 300 may include an IR sensor 301.
[0073] The camera sensor 300 can perform a time-division mode, which is divided into a first time T10 and a second time T20. In the first time T10, it captures an image of an object 1 illuminated by a first light LT1 to generate a first image signal SV1, and in the second time T20, it captures an image of an object 1 illuminated by a second light LT2 to obtain a second image signal SV2.
[0074] For example, during time-division mode operation, the first light LT1 generated by the 2D light source 100 may be IR-illuminated to correspond to 30fps, and the second light LT2 generated by the 3D light source 200 may be structured light IR-illuminated to correspond to 30fps.
[0075] Figure 3 This is a schematic diagram of the camera device used in the example.
[0076] exist Figure 3 In the camera device 10 shown, the camera sensor 300 may include an RGB / IR sensor 302.
[0077] The camera sensor 300 can perform a frequency division mode to capture an object 1 illuminated by visible light LT3 to generate a first image signal SV1, and to capture an object 1 illuminated by a second light LT2 to generate a second image signal SV2.
[0078] For example, during frequency division mode operation, the 2D light source 100 is in the off state, and a 2D image (e.g., 60fps) of an object 1 illuminated by visible light LT3 is incident on the camera sensor 300, and the second light LT2 generated by the 3D light source 200 may be structured light IR illumination to correspond to 60fps.
[0079] Figure 4 This is a schematic diagram of the time-division mode of the camera device.
[0080] Reference Figure 1 , Figure 2 and Figure 4 In the time-division mode, the first time T10 and the second time T20 can include times that are alternately allocated to each other.
[0081] For example, the first time T10 may include allocated times T11, T12 and T13, and the second time T20 may include allocated times T21, T22 and T23 so as not to overlap with the first time T10.
[0082] The controller 400 can control the 2D light source 100 to work at the first time T10 and generate the first light LT1, and the controller 400 can control the 3D light source 200 to work at the second time T20 and generate the second light LT2.
[0083] The 2D light source 100 can be activated under the control of the controller 400 at times T11, T12 and T13, which are included in the first time T10, to generate the first light LT1. The 3D light source 200 can be activated under the control of the controller 400 at times T21, T22 and T23, which are included in the second time T20, to generate the second light LT2.
[0084] The first image signal SV1 may include multiple frames generated during the times T11, T12 and T13 included in the first time T10: frame 1, frame 3 and frame 5, and the second image signal SV2 may include multiple frames generated during the times T21, T22 and T23 included in the second time T20: frame 2, frame 4 and frame 6.
[0085] Figure 5 This is a schematic diagram of a camera sensor.
[0086] refer to Figure 5 For example, camera sensor 300 may include IR sensor 301 for time-division mode operation.
[0087] The IR sensor 301 may include a filter 310 and an image sensor 320.
[0088] Filter 310 can allow first light LT1 and second light LT2 to pass through and block light other than first light LT1 and second light LT2.
[0089] Image sensor 320 can sense first light LT1 and second light LT2 incident through filter 310 to generate first image signal SV1 and second image signal SV2.
[0090] For example, filter 310 may include a first filter unit 311 and a second filter unit 312. The first filter unit 311 may allow a first light LT1 to pass through. The second filter unit 312 may allow a second light LT2 to pass through.
[0091] For example, the second filter unit 312 can allow the second light LT2, which is infrared (IR) at 940 nm, to pass through, and the first filter unit 311 can allow visible light LT3, which has a shorter wavelength than the second light LT2, to pass through. Alternatively, the first filter unit 311 can allow a signal of light with a higher frequency than the signal of the second light LT2.
[0092] For example, image sensor 320 may include a first pixel unit 321 and a second pixel unit 322. The first pixel unit 321 may sense first light LT1 incident through the first filter unit 311 of filter 310. The second pixel unit 322 may sense second light LT2 incident through the second filter unit 312 of filter 310.
[0093] Figure 6 This is a schematic diagram of the frequency division mode (or wavelength division mode) of a camera device.
[0094] Reference Figure 3 and Figure 6 The controller 400 can disable the 2D light source 100 and enable the 3D light source 200 during frequency division mode operation.
[0095] In this case, an image generated by each of visible light LT3(λ1) (e.g., 600 nm) and IR(λ2) (e.g., 940 nm) with different wavelengths can be incident on camera sensor 300. Typically, the wavelength range of visible light is 400 nm to 700 nm, and for example, the visible light wavelength used in this example may be about 600 nm, but is not limited thereto.
[0096] Thus, when using light of different wavelengths, the camera sensor 300 can separate and process the incident image by using a filter.
[0097] Figure 7 yes Figure 5 A schematic diagram of the camera sensor.
[0098] Reference Figure 3 , Figure 5 and Figure 7 The controller 400 can control the 2D light source 100 to be disabled during frequency division mode operation, and the controller 400 can control the 3D light source 200 to always be enabled during frequency division mode operation to generate the second light LT2.
[0099] For example, camera sensor 300 may include RGB / IR sensor 302 for frequency division mode operation.
[0100] The RGB / IR sensor 302 can capture images of an object 1 illuminated by visible light LT3 during frequency division mode operation to generate a first image signal SV1, and capture images of an object 1 illuminated by second light LT2 to generate a second image signal SV2.
[0101] The RGB / IR sensor 302 may include a filter 310 and an image sensor 320. The filter 310 may allow visible light LT3 or a third light and a second light LT2 to pass through. The image sensor 320 may sense the visible light LT3 and the second light LT2 incident through the filter 310 to generate a first image signal SV1 and a second image signal SV2.
[0102] For example, filter 310 may include a first filter unit 311 and a second filter unit 312. The first filter unit 311, as an RGB filter, can allow visible light LT3 to pass through and block light other than visible light LT3. The second filter unit 312, as an IR filter, can allow second light LT2 to pass through and block light other than second light LT2.
[0103] For example, image sensor 320 may include a first pixel unit 321 and a second pixel unit 322. The first pixel unit 321 can sense a third light, which is visible light LT3 incident through the first filter unit 311 of filter 310. The second pixel unit 322 can sense a second light LT2 incident through the second filter unit 312 of filter 310.
[0104] <Time-division mode operation process>
[0105] Reference Figure 1 , Figure 2 , Figure 4 , Figure 5 and Figure 7 As an example, IR sensor 301 ( Figure 2 ) or RGB / IR sensor 302 ( Figure 3 It can be used as a camera sensor 300.
[0106] As an example, IR sensor 301 ( Figure 2 It only includes IR pixels. IR pixels can be used as an IR sensor 301 in dark environments. Figure 2 Time-division imaging is used with a first light source (conventional IR rays) and a second light source (structured light IR rays).
[0107] Alternatively, RGB / IR sensor 302 ( Figure 7 This includes RGB pixels and IR pixels. Because dark areas may not be visible through the RGB / IR sensor 302 ( Figure 7 RGB images are obtained, and time-division imaging can be used when only IR pixels are used. As an example, the RGB / IR sensor 302 ( Figure 7 The IR sensor 301 can be a sensor with an RGB IR pixel ratio of 1:1:1:1, and the IR sensor 301 can be a sensor with only IR pixels.
[0108] Based on the above description and reference Figure 2 The process of operating various example camera devices in time-division mode is outlined below.
[0109] (1) A 2D light source 100 (e.g., an IR LED or a VCSEL) serving as the first light source illuminates the first light LT1 (IR ray) at 30 fps according to the sensor exposure time of an odd number of frames (1, 3, 5...).
[0110] (2) The first light LT1 (IR ray) reflected by object 1 at 30fps can sensitize the IR pixels of camera sensor 300 to obtain the first image signal SV1, which is the IR image signal at 30fps.
[0111] (3) The 3D light source 200 (e.g., structured light IR VCSEL) as the second light source irradiates the second light LT2 (IR ray) at 30fps according to the sensor exposure time of an even number of frames (2, 4, 6, 8).
[0112] (4) The second light LT2 is a 30fps structured light IR ray reflected from object 1, which can sensitize the IR pixels of camera sensor 300 to obtain the second image signal SV2, which is a 30fps structured light IR image signal.
[0113] (5) The first image signal SV1 generated by the first light LT1 (conventional IR ray) obtained by the IR pixels of the camera sensor 300 includes images of odd-numbered frames and is transmitted from the camera sensor 300 to the ECU 20 (see [link]). Figure 1Furthermore, the second image signal SV2, generated by the second light LT2 (structured light IR ray) obtained by the IR pixels of the camera sensor 300, comprises an even number of image frames and is transmitted from the camera sensor 300 to the ECU 20. Therefore, when the first image signal SV1 and the second image signal SV2 are combined, a 60fps image signal is transmitted from the camera device 10 to the ECU 20.
[0114] (6) ECU20 can separate the first image at odd 30fps and the structured light IR image at even 30fps.
[0115] (7) The first image generated by the first light LT1 (conventional IR ray) at odd 30fps can be used as a 2D image as needed.
[0116] (8) The second image generated by the second light LT2 (structured light IR ray) at an even number of 30 fps can be used to calculate the 3D depth using an algorithm and can be used as a 3D image including depth information as needed.
[0117] <Frequency Division Mode Operation Process>
[0118] Reference Figure 3 , Figure 5 and Figure 7 For example, when due to the RGB / IR sensor 302 as a camera sensor 300 ( Figure 3 When the light is bright enough to obtain an RGB image, frequency division imaging can be used. As an example, the RGB / IR sensor 302 ( Figure 3 The mutual interference between visible light and IR can be minimized by using a dual-pass filter that allows only the IR region (e.g., 940 nm wavelength 1) and the visible light region less than or equal to 700 nm (e.g., 600 nm wavelength 2) to pass through.
[0119] Based on the above description and reference Figure 3 The process of operating various example camera devices in frequency division mode is outlined below.
[0120] (1) When the 2D light source 100, which is the first light source, is turned off, visible light LT3, which is natural light, illuminates the object 1, and the reflected visible light LT3 is sensitized by the RGB pixels of the RGB / IR sensor 302 at 60fps, thereby obtaining the first image signal SV1 of the RGB image at 60fps.
[0121] (2) According to the sensor exposure time of 60fps, the second light LT2 (IR ray) irradiated by the 3D light source 200 (e.g., structured light IRVCSEL) as the second light source can be reflected from the object 1 and sensitized by the IR pixel of the RGB / IR sensor 302, thereby obtaining the second image signal SV2 of the structured light IR image at 60fps.
[0122] (3) A 2D image as a regular image can be obtained from the RGB pixels (3 out of 4 pixels) of the RGB / IR sensor 302, and a 3D image as a structured light IR image can be obtained from the IR pixel (1 out of 4 pixels) of the RGB / IR sensor 302. The 2D image and the 3D image are combined, and the combined image (RGB image + IR image) at 60fps can be output to the ECU20.
[0123] (4) The first image, which is a regular image obtained at 60fps by the RGB pixels of the RGB / IR sensor 302, can be transmitted to the ECU20 and used as a 2D image as needed.
[0124] (5) A second image, which is a 60fps structured light IR image obtained by the IR pixels of the RGB / IR sensor 302, can be transmitted to the ECU 20 and used as a 3D image including depth information as needed by calculating 3D depth using an algorithm.
[0125] As described above, the camera devices according to the various examples described herein can be used in both dark and bright environments to capture both 2D and 3D images.
[0126] The controller of a camera device, according to various examples (e.g., controller 400), can be implemented as a computing environment in which processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc.), memory (e.g., volatile memory (e.g., RAM, etc.), non-volatile memory (e.g., ROM, flash memory, etc.)), input devices (e.g., keyboard, mouse, pen, voice input device, touch input device, infrared camera, video input device, etc.), output devices (e.g., display, speaker, printer, etc.) and communication connection devices (e.g., modem, network interface card (NIC), integrated network interface, RF transmitter / receiver, infrared port, USB connector, etc.) are interconnected with each other (e.g., peripheral component interconnect (PCI), USB, firmware (IEEE 1394), optical bus structure, network, etc.).
[0127] The computing environment can be implemented as a distributed computing environment, including, but not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (mobile phones, PDAs, media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and any of the aforementioned systems or devices.
[0128] As mentioned above, based on various examples, camera devices with integrated heterogeneous video can capture 2D and 3D images not only in bright environments but also in dark environments using a single camera.
[0129] As mentioned above, based on various examples, in order to obtain 2D and 3D images, one camera is used instead of two cameras, and thus the camera installation space can be reduced, and the camera cost can be lowered.
[0130] While this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be understood in a descriptive sense only and not for purposes of limitation. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may still be achieved if the described techniques are performed with components in a different order, and / or if they are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be understood to be included in this disclosure.
Claims
1. Camera devices with integrated heterogeneous video capabilities, including: A two-dimensional light source is configured to generate a first light for two-dimensional imaging and to illuminate an object with the first light; A three-dimensional light source is configured to generate a second light for three-dimensional imaging and to illuminate the object with the second light; The camera sensor is configured to selectively perform either time-division mode operation or frequency-division mode operation; as well as A controller is configured to determine whether to perform the frequency-division mode operation or the time-division mode operation based on an illuminance signal, and is configured to control the synchronization of the two-dimensional light source and the camera sensor during the time-division mode operation, and to control the synchronization of the three-dimensional light source and the camera sensor, and is configured to disable the two-dimensional light source and always enable the three-dimensional light source during the frequency-division mode operation. The time-division mode operation is divided into alternating first and second time periods to alternately generate a first image signal and a second image signal. The camera sensor images the object illuminated by the first light at the first time period to generate the first image signal, and images the object illuminated by the second light at the second time period to generate the second image signal. During the frequency division mode operation, the first image signal and the second image signal are generated simultaneously. The camera sensor images the object illuminated by visible light to generate the first image signal, and images the object illuminated by the second light to generate the second image signal. Specifically, in a dark environment, the controller determines to perform the time-division mode operation; in a bright environment, the controller determines to perform the frequency-division mode operation.
2. The camera device according to claim 1, wherein, The controller is configured to control the two-dimensional light source to operate at the first time to generate the first light.
3. The camera device according to claim 1, wherein, The controller is configured to control the three-dimensional light source to operate at the second time to generate the second light.
4. The camera device according to claim 1, wherein, The camera sensor includes: A filter configured to allow the first and second light to pass through during time-division mode operation, and to allow a third light and the second light to pass through during frequency-division mode operation, wherein the third light is visible light; and An image sensor is configured to sense light incident through the filter to generate the first image signal and the second image signal.
5. The camera device according to claim 4, wherein, The filter includes: A first filter unit is configured to allow the first light to pass through during the time-division mode operation and to allow the third light to pass through during the frequency-division mode operation; and The second filter unit is configured to allow the second light to pass through.
6. The camera device according to claim 5, in, The image sensor includes: A first pixel unit is configured to sense the first light incident on the first filter unit of the filter during the time-division mode operation, and to sense the third light incident on the first filter unit of the filter during the frequency-division mode operation; and The second pixel unit is configured to sense the second light incident on the second filter unit of the filter.
7. The camera device according to claim 5, wherein, The camera sensor is an RGB / IR sensor.
8. The camera device according to claim 5, in, The second light is 940nm infrared light.
9. Camera equipment, including: A two-dimensional light source is configured to generate a first light for two-dimensional imaging and to illuminate an object with the first light; A three-dimensional light source is configured to generate a second light for three-dimensional imaging and to illuminate the object with the second light; A camera sensor is configured to selectively perform one of time-division mode operation and frequency-division mode operation. The time-division mode operation is divided into alternating first and second times to alternately generate a first image signal and a second image signal. In the time-division mode operation, first light illuminates the object at the first time to generate the first image signal, and second light illuminates the object at the second time to generate the second image signal. During the frequency-division mode operation, the first image signal and the second image signal are generated simultaneously. In the frequency-division mode operation, visible light illuminates the object to generate the first image signal, and second light illuminates the object to generate the second image signal. as well as A controller is configured to determine whether to perform frequency-division mode operation or time-division mode operation based on an illuminance signal, and to alternately enable the two-dimensional light source and the three-dimensional light source during the time-division mode operation, and to disable the two-dimensional light source and always enable the three-dimensional light source during the frequency-division mode operation. Specifically, in a dark environment, the controller determines to perform the time-division mode operation; in a bright environment, the controller determines to perform the frequency-division mode operation.
Citation Information
Patent Citations
Production of fuel products from waste rubber materials
KR1020210038925A
Hybrid imaging sensor for structured light object capture
CN109564290A
Image acquisition device and application system thereof
CN110312079A
KR1017267760000B1
