Multi-faceted optical path-based multi-directional field of view acquisition and back-view field of view reconstruction system

Abstract

The present invention relates to a system for reconstructing a rear view by fusing data acquired from a first optical group (conventional wide-angle, ultra-wide-angle, telephoto, macro lenses, etc.) that captures a front view and a second optical group (multi-faceted prism or mirror module) that acquires rear information, by independently spaced apart from each other. In particular, the second optical group (multi-faceted prism) is positioned at the geometric center of four lenses to maximize spatial efficiency and protection of the front optical axis, and a mirror structure is provided as an alternative to increase design freedom. The first optical group integrates front and side information by utilizing all four lenses simultaneously, while the second optical group acquires the upper, lower, left, and right regions of the rear area separately. The field of view generation unit applies weight-based matching and boundary correction to the acquired rear area information and generates an integrated rear view through 3D point cloud reconstruction and reprojection. Such image processing and AI-based rear view generation are implemented as software (embedded software or application) executed on an application processor (AP) and a neural network processing unit (NPU), enabling continuous performance improvement without hardware changes. According to the present invention, the entire field of view including the front and back can be secured with a single shot without obstructing the frontal field of view, and thus can be utilized in various fields such as smartphones, robot vision, autonomous driving, security, and medical applications.

Description

Technology Field

[0001] The present invention relates to image acquisition and processing technology, and to a system that acquires a multi-directional field of view of a subject using a first optical group including a plurality of camera lenses and a second optical group including a prism or a mirror, and generates the entire field of view (front, back, top, bottom, left, right) of a subject by aligning and fusing the acquired images to reconstruct a rear field of view. Background Technology

[0002] Conventional camera systems provide a field of view in only a specific direction, so multiple shots or rotation of the subject are required to capture the entire shape of the subject.

[0003] In particular, it is difficult to simultaneously capture the front and back views of items such as clothing, products, and the human body, and alignment errors and time delays occur during the shooting process.

[0004] Although a back-view acquisition technique using a prism has been proposed, placing the prism on the front optical axis causes problems such as obstructing the frontal field of view or generating distortion.

[0005] In addition, existing multi-camera systems are all positioned facing forward, which limits their ability to directly acquire information from the rear. The problem to be solved

[0006] The present invention aims to solve the following problems.

[0007] Acquiring rear information without obstructing the frontal view

[0008] Creating a full field of view by simultaneously acquiring the front and rear views

[0009] Implementing a scalable system while maintaining existing camera structures (wide-angle, ultra-wide-angle, telephoto, macro lenses, etc.)

[0010] Improving accuracy by combining optical structures and image processing algorithms

[0011] Implementing a spatial layout optimized for small devices (smartphones, drones, etc.) means of solving the problem

[0012] The present invention includes a field of view acquisition unit, a field of view generation unit, and a field of view integration unit.

[0013] The field of view acquisition unit is composed of a first optical group and a second optical group.

[0014] The first optical group includes a plurality of lenses (wide-angle, ultra-wide-angle, telephoto, macro / depth, etc.) that acquire field of view information in the front, top, bottom, left, and right directions of the subject.

[0015] The second optical group includes a light path changing means (a multifaceted prism or a plurality of mirrors) for guiding a light ray incident from the back of the subject, and is configured to be physically spaced apart from the first optical group and not positioned on the front optical axis.

[0016] The field of view generation unit generates a back field of view using data acquired from the first optical group and the second optical group. To this end, it includes a weight-based matching algorithm, a boundary correction algorithm between back regions, and a back field of view generation algorithm through 3D shape restoration and reprojection.

[0017] The field of view integration unit aligns and integrates the generated rear field of view with the front / side images of the first optical group to generate the entire field of view. Effects of the invention

[0018] According to the present invention, the following effects are achieved.

[0019] Rear information can be acquired independently without obstructing the frontal view.

[0020] By simultaneously utilizing multiple camera lenses mounted on existing smartphones, robots, and autonomous vehicles to acquire frontal, top, bottom, left, and right information, it is possible to realize a full field of view while minimizing the burden of additional hardware.

[0021] Space efficiency and design freedom can be increased by placing a multifaceted prism at the center of four lenses or by applying a mirror structure.

[0022] Boundary discontinuities between back regions can be effectively corrected through a weight-based and hierarchical matching structure.

[0023] Since all camera modules operate simultaneously with a global shutter, a time-synchronized full field of view can be obtained even when shooting video.

[0024] By implementing image processing algorithms and AI models as software running on the AP and NPU, performance can be improved solely through firmware / app updates without hardware changes, and it offers excellent portability across various platforms. Brief explanation of the drawing

[0025] FIG. 1 is a diagram showing the arrangement structure of the first optical group (110) and the second optical group (120). Figure 2 is a drawing showing a multifaceted prism structure. Figure 3 is a drawing showing a plurality of mirror structures. Specific details for implementing the invention

[0026] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0027] The image processing steps described in the following embodiments are implemented in the form of software (embedded software or application) executed on an application processor (AP) and a neural network processing unit (NPU).

[0028] Although smartphones are used as an example for the sake of convenience, this can be applied in the same way to various systems such as robot vision, autonomous vehicles, security surveillance, and medical imaging.

[0029] 1. Overall System Configuration (Fig. 1)

[0030] As illustrated in FIG. 1, the present invention includes a first optical group (110) and a second optical group (120) with respect to a subject (100). The first optical group (110) includes a plurality of lenses (wide-angle, ultra-wide-angle, telephoto, macro / depth) arranged to face the front direction of the subject. The second optical group (120) is physically spaced apart from the first optical group (110) and is not positioned on the front optical axis, and is preferably located at the geometric center of the plurality of lenses.

[0031] 2. First optical group (110)

[0032] The first optical group (110) simultaneously drives a wide-angle lens, an ultra-wide-angle lens, a telephoto lens, and a macro / depth sensor to acquire field of view information in the front, top, bottom, left, and right directions of the subject.

[0033] Wide-angle lens: Captures the shape and texture of the entire front of the subject in high resolution.

[0034] Ultra-wide angle lens: Utilizes a wide field of view (approx. 120°) to capture information on the upper, lower, left, and right areas of a subject at once, and provides a geometric standard for restoring the side shape.

[0035] Telephoto lens: Separately acquires details of specific areas (e.g., faces, logos) from the front and uses them as precise reference points during alignment.

[0036] Macro / Depth sensor: Provides close-up or depth information to improve the accuracy of 3D point cloud reconstruction.

[0037] Image data acquired from these lenses is transmitted to the field of view generation unit and fused into a single integrated frontal and lateral field of view information through the weight-based matching and hierarchical matching processes described later.

[0038] 3. Second optical group (120)

[0039] The second optical group (120) is a means for directly acquiring optical information from the back of the subject and includes a multifaceted prism structure or a plurality of mirror structures.

[0040] 3.1 Multifaceted prism structure (Fig. 2)

[0041] The multifaceted prism (120) includes a first reflective surface (121), a second reflective surface (122), a third reflective surface (123), and a fourth reflective surface (124).

[0042] The first reflective surface (121) reflects light rays incident from the upper rear area (201).

[0043] The second reflective surface (122) reflects light rays incident from the lower rear area (202).

[0044] The third reflective surface (123) reflects light rays incident from the back left region (203).

[0045] The fourth reflective surface (124) reflects light rays incident from the back right region (204).

[0046] Each reflective surface has a different angle of inclination, and as a result, the four regions of the back surface (top, bottom, left, and right) are each divided and formed on the image sensor. The prism is installed so as to be positioned at the geometric center of the region where multiple lenses of the first optical group (110) are arranged, thereby maximizing space efficiency and not obstructing the front optical axis.

[0047] 3.2 Mirror structure (Fig. 3)

[0048] Multiple mirrors (121-1, 122-1, 123-1, 124-1) are arranged at different angles.

[0049] The upper mirror (121-1) reflects the light rays of the upper rear area (201).

[0050] The lower mirror (122-1) reflects the light rays of the lower rear area (202).

[0051] The left mirror (123-1) reflects the light from the rear left area (203).

[0052] The right mirror (124-1) reflects the light rays of the back right area (204).

[0053] The reflection angle of each mirror is optimized by considering the incident and reflection directions of the light rays, and the mirror structure acquires data by dividing the back surface into four regions, similar to a prism structure. The mirror structure is advantageous for acquiring high-precision back surface information because it is free from chromatic aberration and allows for the independent design of reflection angles for each region.

[0054] The back-side segmented image acquired through the second optical group (120) is formed in a peripheral or unused pixel area of ​​a separate image sensor or the image sensor of the first optical group.

[0055] 4. View Generation Unit

[0056] The field of view generation unit receives the front, bottom, left, and right side point groups fused by the first optical group (110) and the back segmented image (top / bottom / left / right) acquired by the second optical group (120) to generate a back field of view. The specific processing steps are as follows.

[0057] (1) Establishing optical correspondence relationships

[0058] The inclination angle and position of each reflective surface (or mirror) of the second optical group (120) are pre-calibrated so that the pixel coordinates of the back surface area correspond one-to-one with the spatial coordinates on the back surface of the actual subject, and transformation parameters (rotation, translation) are defined.

[0059] (2) Feature point extraction and initial registration

[0060] Feature points are extracted from each region (front, bottom, left, right, and back), and all pixel coordinates are converted into a common 3D spatial coordinate system (based on the center of the subject) using calibrated parameters. Through this, the front image is converted into a 3D point cloud of the front of the subject, and the back segmented image is converted into a 3D point cloud of the back upper / lower / left / right regions, respectively.

[0061] (3) Weighted matching

[0062] Weights are assigned based on the reliability of each area.

[0063] Back area (prism / mirror direct acquisition): Weight 1.0

[0064] Side area (extracted from ultra-wide angle): Weight 0.7

[0065] Frontal Area (Wide-angle + Telephoto Fusion): Weight 0.8

[0066] Interpolation area: Weight 0.3

[0067] Registration proceeds in a hierarchical structure, starting from the highest weighted region, in the order of back → side → front → interpolation.

[0068] (4) Boundary correction and back area integration

[0069] Geometric discrepancies and optical differences may occur at the boundaries of the segmented back top / bottom / left / right regions. The field of view generation unit corrects boundary discontinuities by applying feature point-based matching, optical flow-based warping, or multi-band blending, and integrates the four regions into a single continuous back image.

[0070] (5) 3D shape restoration and back view generation

[0071] After registration and correction are completed, 3D point clouds of the front and sides of the subject are generated through triangulation or depth estimation, and combined with the already integrated back point cloud to reconstruct the 3D shape of the entire subject. The reconstructed 3D shape is reprojected from a virtual back camera viewpoint to generate the final back view image. Areas with low point cloud density or empty areas (holes) are filled by interpolation of adjacent point clouds or AI-based inpainting.

[0072] (6) AI-based expansion

[0073] To replace or assist the above steps, the field of view generation unit may include an artificial neural network (U-Net, NeRF, etc.) trained to directly generate a back field of view by receiving back segmented images and front / side images acquired through a prism or mirror as input. In this case, the neural network can generate a back field of view in real time by integrally learning boundary correction between back regions and 3D reconstruction.

[0074] 5. Vision Integration Unit

[0075] The field of view integration unit combines frontal, lateral, and rear views to generate a complete field of view (in cubemap or panoramic format). During this process, geometric alignment, distortion correction, color correction, and blending are performed. The final result is output to smartphone displays, robot control screens, medical monitors, and the like.

[0076] 6. Specific Examples

[0077] Example 1: Arrangement of multifaceted prisms at the centers of four lenses (optimal structure for smartphones)

[0078] As shown in FIG. 1, a multi-faceted prism (120) is placed at the geometric center of the wide-angle, ultra-wide-angle, telephoto, and macro lenses on the back of the smartphone. The four reflective surfaces of the prism each guide light rays from the upper, lower, left, and right regions of the back surface and form an image on a separate image sensor. Subsequently, a field of view generation unit performs weighted matching, boundary correction, and 3D reconstruction according to the flow of FIG. 4 to generate a back field of view, and combines it with front and side images to output the entire field of view.

[0079] Example 2: Application of mirror structure (high-precision back view acquisition)

[0080] As shown in FIG. 3, a second optical group is formed by arranging multiple mirrors (121-1 to 124-1) at different angles instead of a multifaceted prism. Each mirror reflects light rays from the upper, lower, left, and right regions of the back surface. The subsequent processing steps are the same as those in Example 1. The mirror structure is free from chromatic aberration and allows for independent design of reflection angles, making it suitable for high-precision applications such as medical imaging and robot vision.

[0081] Example 3: Single image sensor sharing structure

[0082] Back surface area information induced from the prism reflective surface (or mirror) is formed onto a peripheral or unused pixel area of ​​the image sensor of the first optical group. In this case, miniaturization is possible without additional sensors.

[0083] Example 4: AI-based back surface generation

[0084] The vision generation unit can be implemented as an artificial intelligence model including a convolutional neural network, a generative model, or a 3D scene reconstruction model.

[0085] The input consists of a front image from a wide-angle lens, a side area extracted from an ultra-wide-angle lens, and images of the upper, lower, left, and right rear areas acquired in segments via a prism or mirror, and the output is an integrated rear image.

[0086] When training, the model can be trained using ground truth images captured by the actual rear camera, and the accuracy of rear view reconstruction can be improved through loss function optimization.

[0087] The trained model can be executed on a neural network processing unit (NPU), application processor (AP), graphics processing unit (GPU), or cloud-based server environment, and can generate back view in real-time or near-real-time.

[0088] 7. Applicable

[0089] The present invention can be applied to various devices, including not only smartphones but also robot vision systems, autonomous vehicles, drones, security surveillance devices, and medical imaging devices.

[0090] As such, while specific embodiments have been described in the detailed description of the present invention, it is understood that various modifications are possible within the scope of the invention.

[0091] Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0092] 100: Subject 110: First optical group 120: Second optical group 121: First reflective surface 122: Second reflective surface 123: Third reflective surface 124: 4th reflective surface 121-1: Upper mirror 122-1: Lower mirror 123-1: Left mirror 124-1: Right mirror 201: Rear upper area 202: Lower rear area 203: Left rear area 204: Rear right area

Claims

Claim 1 A system for acquiring a multi-directional field of view of a subject and reconstructing a rear field of view, comprising: a first optical group including one or more lenses or cameras for acquiring field of view information in the front or side direction of the subject; a second optical group formed as a light path changing means for guiding light rays incident by being reflected or scattered from the rear surface of the subject, wherein the second optical group is formed as a multi-faceted prism structure including a plurality of reflective surfaces or a mirror structure including a plurality of mirrors; and a field of view generating unit for generating a rear field of view using divided rear information of an upper rear area, a lower rear area, a left rear area, and a right rear area acquired through the second optical group. A system comprising: a field of view integration unit that creates an entire field of view of a subject by aligning and integrating the generated back field of view and a front or side image acquired from the first optical group; wherein the second optical group is physically spaced apart from the first optical group and is positioned in the periphery of the front optical axis or the geometric center area of ​​a plurality of lenses so as not to block the center area of ​​the front optical axis; and wherein the field of view generation unit converts the divided back information into a common 3D coordinate system and generates the back field of view by performing at least one of weight-based alignment, correction of boundary discontinuities between back areas, 3D shape restoration, or re-projection to a virtual back camera viewpoint. Claim 2 A system according to claim 1, wherein the first optical group comprises at least one of a wide-angle lens, an ultra-wide-angle lens, a telephoto lens, and a macro lens, and the lens or camera is configured to acquire at least one field of view information among the front, top, bottom, left, and right sides of a subject. Claim 3 A system according to claim 1, wherein the second optical group is formed as a multifaceted prism structure including a plurality of reflective surfaces or a mirror structure including a plurality of mirrors, wherein the multifaceted prism structure includes a first reflective surface that guides a ray of light incident from an upper region of the back surface of a subject, a second reflective surface that guides a ray of light incident from a lower region of the back surface of a subject, a third reflective surface that guides a ray of light incident from a left region of the back surface of a subject, and a fourth reflective surface that guides a ray of light incident from a right region of the back surface of a subject, and wherein the mirror structure includes a plurality of mirrors arranged at different angles to correspond to the upper region, lower region, left region, and right region of the back surface of a subject. Claim 4 A system according to claim 1, wherein the field of view generating unit comprises: a weight-based matching algorithm that defines information of a rear upper area, a rear lower area, a rear left area, and a rear right area acquired through the second optical group as rear area information and assigns weights according to area-specific reliability to the rear area information; a boundary correction algorithm that corrects boundary discontinuities between the rear upper area, the rear lower area, the rear left area, and the rear right area; and a reconstruction algorithm that generates a rear field of view through 3D shape restoration or re-projection to a virtual rear camera viewpoint using the matched and corrected rear area information. Claim 5 A method for providing an entire field of view of a subject using a system according to any one of claims 1 to 4, comprising: acquiring field of view information in the front or side direction of the subject through a first optical group; acquiring optical information for an upper area, a lower area, a left area, and a right area of ​​the rear surface of the subject in segments through a second optical group that is physically spaced apart from the first optical group and positioned so as not to block the central area of ​​the front optical axis; converting the information of the segmented acquired upper rear area, lower rear area, left rear area, and right rear area into a common three-dimensional coordinate system; aligning the converted rear area information by assigning weights according to the reliability of each area; correcting boundary discontinuities between the upper rear area, lower rear area, left rear area, and right rear area; and generating a rear field of view by performing three-dimensional shape restoration or re-projection to a virtual rear camera viewpoint using the aligned and corrected rear area information. A method characterized by including the step of aligning and integrating the rear view generated above with the front or side view information acquired through the first optical group to generate the entire view of the subject.

Images