3D camera and application

Abstract

The application discloses a 3D camera and application, which comprises a camera lens group for capturing ambient light, an imaging unit and a processor for analyzing and processing a real image picture formed by the imaging unit. The imaging unit comprises photosensitive chips, and the photosensitive chips form conjugate pixel groups among a plurality of pixels. The proportion of pixels of any photosensitive chip participating in the conjugate grouping is greater than or equal to 60%. After the image plane light at different distances in the ambient environment is optically converted by the camera lens group, different brightness pictures are formed on the photosensitive chips. The processor analyzes and processes the recorded pictures, and distance measurement of the ambient environment is realized.

Description

Technical Field

[0001] This invention relates to the field of optical ranging technology, specifically to a 3D camera and its application. Background Technology

[0002] With the rapid development of AI technology, many application scenarios require accurate 3D perception capabilities. Currently, there are various 3D perception solutions available: 1) LiDAR is one type, but it suffers from high cost and low resolution; 2) Image-based visual recognition methods can also obtain good 3D environmental information, but they require a lot of learning in the early stages. When faced with unfamiliar scenes, the recognition accuracy cannot be guaranteed, and they may even fail, causing major accidents. 3) Binocular ranging can also directly obtain 3D information of the environment through the principle of geometric optics, but the calculation process requires accurate pixel matching. If the texture features of the image are not obvious, pixel matching cannot be performed effectively, resulting in the inability to complete the ranging calculation.

[0003] Therefore, each of the above methods has some insurmountable shortcomings. In response to these problems, the present invention proposes a completely new technical solution. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a 3D camera, comprising: A camera lens assembly used to capture ambient light. An imaging unit, comprising a photosensitive chip with multiple pixels forming conjugate pixel groups, wherein the proportion of pixels in any photosensitive chip participating in conjugate grouping is ≥60%; and A processor for analyzing and processing the real image formed by the imaging unit; Light rays from different distances in the external environment are optically converted by the imaging lens group and form images of different brightness on the image sensor. The processor then analyzes and processes the recorded images to achieve distance measurement of the external environment.

[0005] Furthermore, the imaging unit also includes an optical path integration mirror group whose positional relationship with the multiple photosensitive chips satisfies the optical imaging principle, and is used to optically convert image planes of different depths of field into real images. The optical path integration mirror group is a cubic prism formed by splicing multiple sub-prisms, and each of the photosensitive chips corresponds to one side of the cubic prism. The multiple photosensitive chips are integrated and fused together through the cubic prism.

[0006] Furthermore, each of the photosensitive chips is any one of a color photosensitive chip, a monochrome photosensitive chip, an infrared photosensitive chip, and a black and white photosensitive chip.

[0007] Furthermore, the 3D camera operates in a visual ranging mode, an optical ranging mode, or a hybrid mode of the two.

[0008] Furthermore, in optical ranging mode, the processor analyzes and processes the real image using optical ranging principles.

[0009] Furthermore, in visual ranging mode, the processor analyzes and processes the real-image image using machine learning methods for ranging.

[0010] Furthermore, in the hybrid mode of optical and visual ranging, the processor analyzes and processes the real image by using machine learning methods for ranging and optical ranging for auxiliary verification.

[0011] Furthermore, the pixels of the 3D camera are adjustable. During the analysis and processing of the real image formed by the imaging unit, the processor adjusts the pixels through pixel fusion, where 1 < fusion ratio < 100.

[0012] The present invention also provides an application of the above-mentioned 3D camera, which is applied to a mobile device and operates in a hybrid mode. The resolution and scanning frequency of the optical ranging mode are adjusted according to the magnitude of the verification error between the optical ranging result and the visual recognition result.

[0013] Furthermore, in hybrid mode, the processor sends control signals to the mobile device based on the magnitude of the error between the optical ranging result and the visual recognition result, thereby controlling the mobile device to adjust its speed.

[0014] Compared with the prior art, the advantages of the present invention are as follows: LiDAR is expensive, requires light source calibration, and suffers from problems such as temperature drift. It also has high energy consumption and interference issues in practical applications. In contrast, this invention adopts a passive imaging working mode, so ambient light will not cause interference. All components of this invention can be mature and existing components, and the commercialization cost is also low. This invention features a pixel fusion mode, which allows for switching the number of pixels for different scenarios, making it more flexible and versatile.

[0015] This invention achieves high-precision 3D imaging through optical imaging, and can work simultaneously in optical ranging, visual recognition, and hybrid modes, making its application more flexible and practical. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of the present invention. Figure 2 This is a schematic diagram of the structure of the present invention, in which a cubic prism is used as an integrated optical path mirror assembly 22. Figure 3 This is a schematic diagram of the structure of an imaging unit 2, which integrates a cubic prism as an optical path integration mirror group 22 with five photosensitive chips 21. Figure 4 This is a schematic diagram showing the intersection of the equivalent image planes of the three photosensitive chips 21 with the incident light. The attached figures are labeled as follows: The imaging lens group 1, the imaging unit 2, the photosensitive chip 21, and the optical path integrated lens group 22 are all included. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0019] Reference Figures 1 to 4 This invention provides a 3D camera for measuring distances in the external 3D environment, comprising a camera lens group 1, an imaging unit 2, and a processor, wherein: The shooting lens group 1 is used to capture the light of the external environment. The light from the image planes of the environmental scene at different depths of field is optically converted by the shooting lens group 1 and transmitted to the imaging unit 2. The shooting lens group 1 can be a single optical lens or a combination of multiple optical lenses. Imaging unit 2 includes photosensitive chips 21 that form conjugate pixel groups among multiple pixels. When a beam of light is incident, it intersects with the equivalent image planes of the multiple photosensitive chips 21 (see invention patent publication number CN111190325A). The multiple pixels at the intersection points of the incident light and the multiple equivalent image planes form a conjugate pixel group, such as... Figure 4 As shown, taking a device containing three photosensitive chips 21 as an example; Furthermore, the proportion of pixels of any photosensitive chip 21 participating in conjugation grouping is ≥60%. Specifically, multiple photosensitive chips 21 are respectively set at different positions at different distances from the back focal length of the lens group. In this way, when external object light enters, the area illuminated on the three photosensitive chips 21 is different. At this time, if the area of ​​the photosensitive chips 21 is the same, some pixels will not be able to form a group. If the proportion of pixels that cannot form a group is too large, it will limit the actual space that can be used for 3D imaging, resulting in a waste of hardware resources. Conversely, by reserving a portion of ungrouped pixels, the field of view can be expanded beyond 3D imaging to obtain more environmental information. These pixels can provide additional 3D information using visual recognition methods with low computational precision, thereby maximizing the overall benefits of accuracy and resources.

[0020] In order to ensure that the multiple photosensitive chips 21 do not block each other and that light can be transmitted smoothly, each photosensitive chip 21 has a certain light transmittance, preferably 50% light transmittance; More preferably, to avoid obstruction, an optical path integration lens group 22 can be introduced to optically convert the optical path. The positional relationship between this optical path integration lens group 22 and the multiple photosensitive chips 21 must satisfy the optical imaging principle, enabling the optical conversion of image planes at different depths of field into real images. Specifically, for example... Figure 3 The optical path integration mirror group 22 is a cubic prism formed by splicing multiple sub-prisms. Each photosensitive chip 21 corresponds to one side of the cubic prism. Multiple photosensitive chips 21 are integrated and fused together through the cubic prism. The optical path integrated mirror group 22 can also adopt other structural forms, such as a row of semi-transparent and semi-reflective mirrors. The technical solution of the optical path integrated mirror group 22 is described in detail in the invention patent with publication number CN111190325A, and will not be repeated here. The photosensitive chip 21 of the present invention can be any one of a color photosensitive chip, a monochrome photosensitive chip, an infrared photosensitive chip, and a black and white photosensitive chip. All of the above-mentioned photosensitive chips are mature existing technologies. Depending on the photosensitive chip 21, when the imaging unit 2 is set up, multiple photosensitive chips can be selected from the same type of photosensitive chip, or any combination of the above types can be used. The processor records and analyzes the real image formed by the imaging unit 2. Light rays from different distances in the external environment are optically converted by the imaging lens group 1, forming images of varying brightness on the image sensor 21. The processor then analyzes and processes the recorded images to achieve distance measurement of the external environment. Figure 4 .

[0021] The 3D camera of this invention operates in an optical ranging mode, a visual ranging mode, or a hybrid mode of the two, wherein: refer to Figure 4 With point O as the origin, and the three image planes located at positions X1, X2, and X3 respectively, when light from an object point is received by the mirror group, it is converged into a light cone that intersects the three image planes at positions X1, X2, and X3 respectively. The area of ​​the light spot formed at different positions is different, and the attenuation of light during propagation is negligible. Therefore, the size of the light spot is inversely proportional to the light intensity. Consequently, the light intensity values ​​p1, p2, and p3 collected by the pixels at the intersection of the light cone's center line and the image planes are also inversely proportional to the light spot size. Based on geometric relationships, the following equation can be obtained: (X-X2) / p2=(p2-p1) / (x2-x1); X=p2*(p2-p1) / (x2-x1) / x2; Where X is the x-coordinate of the vertex of the light cone. With the X value, the distance to the object point can be calculated in reverse using the parameters of the mirror group. Furthermore, to ensure the reliability of the calculation results, the following verification formula can also be used for verification: Verification formula: (X-X3) / p3=(p2+p3) / (x3-x2); In practical applications, X is the accurate position of the light cone vertex, which can be guaranteed to hold both of the above equations simultaneously (or with the smallest error). The distance to the object point can then be calculated in reverse using the parameters of the mirror group. In visual ranging mode, the processor analyzes and processes the real-world image using machine learning methods, which is a mature existing technology. Specifically, a dataset can be obtained through actual measurement. This dataset contains a large number of object points with known distances and their corresponding binocular / multi-view images. A machine learning algorithm is trained using a neural network model or other machine learning algorithms to form a machine learning data model. Distance is then measured using this data model to perceive the 3D information of the environment. For example, Tesla's camera-based autopilot technology utilizes the principle of visual recognition. In hybrid mode, the processor analyzes and processes real-world images by using machine learning for distance measurement and optical ranging for auxiliary verification. Specifically, distance measurement is performed through a data model, distance calculation is performed using geometric optics principles, and then cross-verification is performed. If errors exist, the optical ranging data can be used as learning data and input into the visual recognition model to help it learn and optimize. In hybrid mode, mature visual recognition algorithms can be utilized to greatly improve measurement speed. At the same time, because a small number of local optical ranging points can be flexibly introduced for verification, the reliability of visual recognition can be greatly improved with very little additional hardware burden.

[0022] The 3D camera of this invention features adjustable pixels. During the analysis and processing of the recorded real-world image, the processor adjusts pixels through pixel fusion, where 1 < fusion ratio < 100. Specifically, the processor can process multiple adjacent pixels as a single pixel. In practical applications, some 3D imaging scenarios require extremely high resolution, while others do not require high resolution but demand extremely high computational speed. Pixel fusion allows multiple pixels to be processed as a single pixel in scenarios with low resolution requirements (similar to digital zoom in a mobile phone camera), significantly saving hardware computing resources. In scenarios with high resolution requirements (such as high-speed driving), non-fusion or a smaller number of pixels can be fused to ensure the required resolution. This enables adaptive resolution adjustment and enhances adaptability to various application conditions. However, a higher fusion ratio is not always better; too high a fusion ratio leads to a significant reduction in resolution. Theoretical analysis shows that a fusion ratio between 1 and 100 provides good adaptability to practical applications.

[0023] The present invention also provides an application of the above-mentioned 3D camera, which is applied to a mobile device and operates in a hybrid mode. The resolution and scanning frequency of the optical ranging mode are adjusted according to the magnitude of the verification error between the optical ranging result and the visual recognition result. It can also adjust the speed of mobile devices. The processor sends control signals to the mobile device based on the magnitude of the error between the optical ranging result and the visual recognition result.

[0024] In hybrid mode, visual recognition mode is the primary mode under normal circumstances, while optical ranging mode is the secondary mode. This can maximize the saving of computing hardware resources. When it is found that the two modes have large errors during actual work, it means that the visual recognition mode has encountered difficulties and cannot accurately identify the distance. At this time, in order to ensure high reliability, the density of verification points and the measurement frequency of optical ranging can be increased to avoid the errors caused by visual recognition. It can also send speed control requests to mobile devices to reduce the driving speed. Reducing the driving speed can also improve the accuracy of the measurement and reduce the severity of the accident consequences caused by inaccurate measurement. In extreme cases, when the visual recognition mode has a large error, it can be converted to optical ranging as the primary mode and visual recognition as the secondary mode, or visual recognition can be abandoned and only optical ranging mode can be used.

[0025] The software control components involved in this invention can all be developed independently and are not limited thereto.

[0026] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A 3D camera, characterized in that, include: A camera lens assembly used to capture ambient light (1); Imaging unit (2), the imaging unit (2) includes a photosensitive chip (21) in which multiple pixels form conjugate pixel groups, and the proportion of pixels in any photosensitive chip (21) participating in conjugate grouping is ≥60%; and A processor for analyzing and processing the real image formed by the imaging unit (2); After the light rays from the image plane at different distances in the external environment are optically converted by the imaging lens group (1), they form images of different brightness on the photosensitive chip (21). The processor then analyzes and processes the recorded images to achieve distance measurement of the external environment.

2. The 3D camera according to claim 1, characterized in that, The imaging unit (2) also includes an optical path integration mirror group (22) whose positional relationship with multiple photosensitive chips (21) satisfies the optical imaging principle and is used to optically convert image planes of different depths of field into real images. The optical path integration mirror group (22) is a cubic prism formed by splicing multiple sub-prisms. Each photosensitive chip (21) corresponds to one side of the cubic prism, and multiple photosensitive chips (21) are integrated and fused together through the cubic prism.

3. A 3D camera according to claim 1, characterized in that, Each of the photosensitive chips (21) is any one of a color photosensitive chip, a monochrome photosensitive chip, an infrared photosensitive chip, and a black and white photosensitive chip.

4. A 3D camera according to claim 1, characterized in that, The 3D camera operates in visual ranging mode, optical ranging mode, or a hybrid mode of the two.

5. A 3D camera according to claim 4, characterized in that, In optical ranging mode, the processor analyzes and processes the real image using optical ranging principles.

6. A 3D camera according to claim 4, characterized in that, In visual ranging mode, the processor analyzes and processes the real-image image using machine learning methods for ranging.

7. A 3D camera according to claim 4, characterized in that, In the hybrid mode of optical ranging and visual ranging, the processor analyzes and processes the real image by using machine learning methods for ranging and optical ranging for auxiliary verification.

8. A 3D camera according to claim 1, characterized in that, The pixels of the 3D camera are adjustable. During the analysis and processing of the real image formed by the imaging unit (2), the processor adjusts the pixels by pixel fusion, where 1 < fusion ratio < 100.

9. The application of a 3D camera on a mobile device according to claim 4, characterized in that: When operating in hybrid mode, the resolution and scanning frequency of the optical ranging mode are adjusted according to the magnitude of the verification error between the optical ranging result and the visual ranging result.

10. The application of a 3D camera on a mobile device according to claim 4, characterized in that: When operating in hybrid mode, the processor sends control signals to the mobile device based on the magnitude of the error between the optical and visual ranging results, thereby controlling the mobile device to adjust its speed.

Images