Method and apparatus for generating a three-dimensional reconstruction of the environment surrounding a vehicle.

Abstract

To provide a method and a device for generating a three-dimensional restoration of a vehicle ambient environment.SOLUTION: There may be provided a computer implementation method for generating a three-dimensional restoration of a vehicle ambient environment. This method may include the steps of: capturing a plurality of wide-angle images around a vehicle using a plurality of fisheye lens cameras installed on the vehicle; creating a surround view image around the vehicle from a plurality of captured images by generating one or more feature maps; and posture and depth estimation from the generated feature map using at least one convolution neural network. This method may also include the steps of: detecting one or more objects in the plurality of captured images; mapping the one or more objects detected around a vehicle model into the created surround view image using the calculated posture and depth estimation; and configuring an environmental three-dimensional restoration using the surround view image and the mapped object.SELECTED DRAWING: Figure 2

Description

Technical Field

[0003] , , , , , , , , , , , , , ,

[0004] , ,

[0001] Various combinations of the features of the present disclosure relate to a method and an apparatus for generating a three-dimensional restoration of the environment around a vehicle. More specifically, this relates to detecting a scene around the vehicle. of

Background Art

[0002] A surround view system is an important function of a vehicle and can be useful in many situations. The ultrasonic sensors used in the surround view system help the driver to judge the possibility of obstacles, but in many cases, it does not work well due to poor detection. The surround view system is also not very sensitive due to low lighting and crowded spaces in places with little light or in parking lots. Existing deep learning models trained for vehicle light recognition in the surround view system also cannot generate reliable Time-To-Colision (TTC) values on completely dark images at night.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

[0005] In light of the above, the surroundings of the vehicle for the Around View System of There is a need for improved detection methods in the environment. [Means for solving the problem]

[0006] To meet the above needs, around the vehicle of A method is provided for generating a 3D reconstruction of the environment. This method may include the step of capturing multiple wide-angle images around a vehicle using multiple fisheye lens cameras mounted on the vehicle. This method may further include the step of creating a surround-view image of the vehicle from the multiple captured images by generating one or more feature maps, and the step of calculating pose and depth estimates from the generated feature maps using at least one neural network. This method may also include the step of detecting one or more objects in the multiple captured images. This further includes the step of mapping one or more objects detected around the vehicle model onto the created surround-view image using the calculated pose and depth estimates, and the step of constructing a 3D reconstruction image of the environment using the surround-view image and the mapped objects. This allows for the creation of a 3D reconstruction of the environment around the vehicle. of It may be possible to create a "bowl-shaped" image of the environment, and objects around the vehicle can be appropriately mapped onto the created image. The neural network may be a combination of a feature extraction network and a 3D convolutional network.

[0007] In alternative feature combinations that can be combined with the alternative embodiments described above, the step of capturing multiple wide-angle images includes capturing at least two wide-angle images having overlapping fields of view from multiple fisheye lens cameras. The method may include the steps of calculating the parallax between the two or more captured wide-angle images, and using the calculated parallax to detect the coordinates of objects on the two or more captured wide-angle images and the dimensions of the objects on the images. This may make it possible to obtain a clear field of view around the vehicle. This allows it to determine how far an object is from the vehicle and use the location of the object to safely maneuver the vehicle, even in a smaller parking space. The aforementioned “calculated parallax” may be calculated, for example, using a parallax map. The images may be converted to grayscale before calculating the parallax map. The parallax function may be used to calculate the parallax by comparing the difference of each block of pixels in the grayscale image. In alternative feature combinations, each pixel in one image is matched with its corresponding pixel in another image. The distance or difference of each pair of matching pixels may be calculated. Ultimately, the disparity map can be obtained by representing such distance values ​​as a luminance image. In other alternative feature combinations, other suitable processes may be used to calculate the disparity, for example, in the patent publication (Patent Document 1), more specifically see paragraphs

[0035] to

[0071] .

[0008] In the alternative combinations and possible combinations of features described above, the step of capturing multiple wide-angle images further includes the step of de-distorting the captured images and the step of correcting geometric alignment in the de-distorted images. This method further includes the step of generating a point cloud by providing a calculated difference between the captured left and right images as input to a neural network, and the step of using the generated point cloud to estimate the motion of an object and perform simultaneous positioning of the object on the left and right images. This allows for positioning of the area around the car. ofTo create the optimal environment, it may be possible to use corrected and adjusted wide-angle images, thereby obtaining error-free and accurate results for object detection and other processing on the image. This method also helps in estimating object motion, which in turn helps drivers operate vehicles safely.

[0009] In the alternative combinations and possible combinations of features described above, the method step is performed multiple times, and the steps of generating feature maps and calculating pose and depth estimates include processing each of the captured wide-angle images with a feature extraction network and calculating pose augmentation updates for each of the captured wide-angle images. Feature maps at different resolutions are generated for feature extraction, which can then be combined, flattened, and passed to a fully connected layer in the neural network. The extracted features can be transmitted to a 3D convolutional network for depth and pose estimation. This is done for pose augmentation updates between each adjacent pair. This can make it possible for a vehicle to avoid colliding with a moving object and can improve the safety of pedestrians around the vehicle. This is also useful for tracking the orientation of objects.

[0010] The feature extraction and feature detection process may include activating a YOLOv5 model trained on training data to form a model, which serves as the basis for object detection in the received camera data. Preferably, the YOLOv5 model is formed for object detection using a MobileNet-type backbone network.

[0011] The YOLO model family can consist of three main architectural blocks: i) backbone, ii) neck, and iii) head. YOLOv5 backbone: This utilizes CSPDarknet as a backbone for feature extraction from images, consisting of a cross-stage partial network. YOLOv5 Neck: This generates a feature pyramid network to perform aggregation on features and uses PANet to pass it on to the head for prediction. YOLOv5 Head: A layer that generates predictions from anchor boxes for object detection.

[0012] A convolutional neural network (CNN) is a multi-layer feedforward neural network constructed by stacking many hidden layers sequentially on top of each other. This sequential design allows convolutional neural networks to learn hierarchical features. The hidden layers are typically convolutional layers, followed by activation layers, some of which are pooling layers. A CNN can be configured to identify patterns in data. The convolutional layers may contain a convolutional kernel, which is used to search for patterns through the input data. The convolutional kernel returns a large positive value for parts of the input data that match the kernel's pattern, or a small value for other parts of the input data that do not match the kernel's pattern.

[0013] CNNs can extract informational features from training data, eliminating the need for manual processing of the training data. CNNs can produce accurate results even with large amounts of unstructured data, such as in image classification, speech recognition, and natural language processing. Furthermore, CNNs are computationally efficient because they can construct increasingly complex patterns using relatively small kernels in each hidden layer.

[0014] Trajectron++ is a machine learning algorithm that uses a conditional variational autoencoder, a long-term short-term memory network, and a convolutional neural network. It is used to predict the future dynamics of several entities in a scene. This algorithm may be based on the Trajectron algorithm described in detail in CoRR,abs / 1810.05993.2018,DOI http: / / arxiv.org / abs / 1810.05993 (Non-Patent Literature 1) and CoRR.abs / 2001.00735,2020,DOI https: / / arxiv.org / abs / 2001.03093 (Non-Patent Literature 2). It can be used, for example, to predict the trajectory of at least partially autonomous vehicles.

[0015] The entity environment is presented as a directed spatiotemporal graph. Nodes represent surrounding entities, connecting different entities and influencing each other within the same timestep, thus connected by spatial edges that cover interactions between different entities, and by temporal edges that connect the same node over time and therefore represent past dynamics. Interaction between two different entities occurs when their distance is smaller than an attention radius that can be independently and individually selected for each entity class of the two influencing entities. An implementation of the Trajectron++ algorithm can be found at https: / / github.com / StanfordASL / Trajectron-plus-plus. For training, for example, the ETH and UCY pedestrian datasets can be used.

[0016] MobileNet is an architectural model of a convolutional neural network for object detection that is primarily optimized for speed. The main building block of MobileNet is a depth-separable convolution that decomposes or separates a standard convolutional filter into two different operations: (i) a first operation ("depth-unit convolution") in which separate convolutional kernels, also called depth-unit convolutions, are applied to the input channels, and (ii) a second operation ("point-unit convolution") in which a point-unit (1x1) convolution is used to combine the information from the first operation. On the other hand, a standard convolutional filter performs channel-unit and spatial-direction calculations in one step. Separating or decomposing a standard convolution into two different operations results in fewer parameters and lower computational cost than a standard convolution because it has fewer multi-adds (multiplication and addition operations).

[0017] In some embodiments, the aforementioned CNN may include a first standard convolutional layer, followed by a number of depth-unit and point-unit convolutional layers, an average pooling layer, a fully connected layer, and a softmax classifier. Each layer of the CNN may be followed by batch normalization (BN) and normalized linear activation function (ReLU) nonlinearity, with the exception of the final fully connected layer, which is nonlinear and feeds into the softmax layer for classification.

[0018] In the alternative combinations and possible combinations of features described above, the vehicle model is placed at the center of the created surround view image. This allows the driver to interpret the distance of objects relative to the vehicle's position and helps the driver detect the vehicle's actions.

[0019] In the alternative combinations and possible combinations of features described above, the method is performed continuously until the vehicle's engine is shut off. This allows the driver to continuously detect the environment and operate the vehicle.

[0020] In the above-mentioned alternative combinations and combinations of possible alternative features, the surroundings of the vehicle ofAn apparatus for generating a three-dimensional restoration of an environment is provided, which includes a memory, a plurality of fisheye cameras mounted on a vehicle, one or more image processors connected to the plurality of fisheye cameras, one or more image processing units connected to the plurality of fisheye cameras, and one or more processing cores for executing the foregoing method.

[0021] In an alternative combination that can be combined with the above-mentioned alternative combination, a vehicle including the above-mentioned apparatus is provided.

[0022] In an alternative combination that can be combined with the above-mentioned alternative combination, a non-transitory computer-readable storage medium including instructions that, when executed by a processor, execute the foregoing method is provided.

[0023] In the figures, like reference numerals generally refer to like parts throughout the different figures. The drawings are not necessarily to scale, and instead, it is emphasized that generally the principles of the present invention are illustrated. In the following description, various embodiments are described with reference to the following drawings.

Brief Description of the Drawings

[0024] [Figure 1] Relates to an embodiment of a general-purpose computing machine for implementing the present disclosure. [Figure 2] Relates to a flowchart of an embodiment of the process disclosed in the present disclosure. [Figure 3] Relates to a flow diagram of an embodiment of the process disclosed in the present disclosure. [Figure 4] Relates to an example of the restoration of the environment around a vehicle. [Figure 5] Relates to an example of a system for executing the disclosed process.

Modes for Carrying Out the Invention

[0025] The combinations of features described below for the apparatus are equally valid for each method, and vice versa. Furthermore, the combinations of features described below can be considered together; for example, parts of one combination may be combined with parts of another.

[0026] Any characteristics described herein with respect to a particular apparatus should be understood to apply to any apparatus described herein. Any characteristics described herein with respect to a particular method should be understood to apply to any method described herein. Furthermore, it should be understood that not all components or steps described herein must necessarily be included in any apparatus or method described herein, and only some (but not all) components or steps may be included.

[0027] In the alternative combinations and possible combinations of alternative features described above, this disclosure provides information on the surroundings of the vehicle that can assist in monitoring, warning, braking, steering, and other such tasks. of This relates to representing the environment. This can be useful in dark, poorly lit, and crowded spaces where visual cues may be limited. The vehicle may be an automobile, a multi-axle vehicle with more than two axles, or a vehicle with a trailer attached.

[0028] Throughout this specification, the term "pose estimation" may mean predicting and tracking the position of an object within an image. This can predict the deformation of an object from a user-defined reference pose for a given image. The image data from which the object's pose is identified may be a single image, a pair of stereo images, or an image sequence.

[0029] Throughout this specification, the term "depth" may refer to the perpendicular distance between an object and the plane of the scene camera. Each pixel in an image has a depth value, which can indicate the distance of the pixel from the camera plane.

[0030] Figure 1 shows a generalized example of a suitable computing environment 100 that may be used in implementing the present disclosure. Computing environment 100 does not propose any limitations in terms of scope of use or functionality, as the technology may be implemented in a variety of general-purpose or application-specific computing environments.

[0031] Referring to Figure 1, the computing environment 100 includes at least one processing unit 110 connected to memory 120. In Figure 1, this basic configuration 130 is enclosed within a dashed line. The processing unit 110 executes computer execution instructions. In a multiprocessing system, multiple processing units execute computer execution instructions to increase processing capacity. Memory 120 may be non-temporary memory such as volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or a combination of the two. Memory 120 can store software 180 that implements any of the technologies described herein.

[0032] The computing environment may also have other features. For example, computing environment 100 includes storage 140, one or more input devices 150, one or more output devices 160, and one or more communication connections 170. Interconnection mechanisms (not shown), such as buses, controllers, or networks, interconnect the components of computing environment 100. Typically, operating system software (not shown) provides an operating environment for the execution of other software running within computing environment 100 and coordinates the activities of the components of computing environment 100.

[0033] The storage 140 may be removable or non-removable and may include magnetic disks, magnetic tapes, or any other non-temporary computer-readable media that can be used to store information and are accessible within the computing environment 100. The storage 140 may store software 180 containing instructions for any of the technologies described herein.

[0034] The input device 150 may be a keyboard, a touch input device such as a touchscreen, an audio input device, a scanning device, or other device that provides input to the computing environment 100. The output device 160 may be a display, a speaker, or other device that provides output from the computing environment 100. Some input / output devices, such as touchscreens, may include both input and output functions.

[0035] Communication connection 170 enables communication with other computing entities through a communication mechanism. The communication mechanism carries information, such as computer executable instructions, voice / video, or other information or data. For example, but not limited to, the communication mechanism includes wired or wireless technologies implemented by electrical, optical, RF, infrared, acoustic, or other carriers.

[0036] The technology of this application can generally be described in relation to computer executable instructions that run on a target real or virtual processor within a computing environment, such as those contained in program modules. Generally, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The functionality of program modules can be combined or separated among program modules in various combinations as desired. Computer executable instructions for program modules can be executed within a local or distributed computing environment.

[0037] Any of the storage operations described herein can be performed by storing information in one or more computer-readable media (e.g., non-temporary computer-readable storage media or other tangible media). Any of the information described as being "stored" can be stored in one or more computer-readable media (e.g., computer-readable storage media or other tangible media).

[0038] Any of the methods described herein can be executed by computer-executable instructions (e.g., encoded therein) in one or more computer-readable media (e.g., non-temporary computer-readable storage media or other tangible media). Such instructions can cause a computer to execute this method. The techniques described herein can be implemented in various programming languages.

[0039] Any of the methods described herein can be executed by computer-executable instructions stored in one or more non-temporary computer-readable storage devices (e.g., memory, CD-ROM, CD-RW, DVD, or others). Such instructions can cause a computer to execute this method.

[0040] In this combination of features, which may be combined in alternative ways, the computing environment (100) may be implemented within a vehicle appropriately configured with the aforementioned other components, including memory, a processor, input devices, and software instructions. In other combinations of features, the computing environment may be implemented as a system-on-a-chip (SoC) or other embedded system.

[0041] The process of this patent application will be explained with reference to Figure 2, with regard to the above-mentioned alternative feature combinations and possible alternative feature combinations. In one feature combination, the area around the vehicle ofMultiple wide-angle images of the environment can be captured (200). This can be done by one or more fisheye lens cameras that may be mounted on the vehicle. In one example, the surround view system may use four to six wide-angle or fisheye lens cameras mounted in front of, behind, and to the sides of the vehicle. This allows the wide-angle images captured by the fisheye lens cameras to capture the area around the vehicle. of It may become possible to detect and identify the environment. The use of wide-angle images, as will be discussed in more detail in the following paragraphs, helps to capture images with a larger field of view that may overlap with adjacent images, and can also assist in object detection and localization.

[0042] In the alternative combinations and possible feature combinations described above, the image captured by the fisheye lens camera may be distorted, misaligned, or require some other correction. The image can first be processed to remove its distortion. This can be done by correcting the geometric alignment. The image can then be further corrected for balancing and color correction. A distortion coefficient can be used to remove the distortion from the image. This can ensure that the further analysis and processing described in the following paragraphs are more accurate and error-free.

[0043] In a combination of features, at least two adjacent images with overlapping fields of view may be selected. The selected images may be superimposed to identify the parallax between them. The parallax may be a pixel difference or a difference due to motion. The identified parallax may help to calculate the alignment difference between the images to detect the actual position and motion of objects in both images. This may also help to calculate the size, dimensions, and coordinates of objects. This allows the vehicle to detect how far away and how large objects present in the vehicle's environment are. Based on this calculation, objects may be correctly mapped onto the surround-view images around the vehicle.

[0044] In the combination of features that can be combined with the alternative combinations described above, one or more feature maps may be generated using multiple wide-angle images (201). The feature maps may enable the calculation of pose and depth estimation (202). This may be done using one or more convolutional neural networks. The convolutional neural network may identify key points in the image for estimation. The depth estimation neural network may detect objects in the image, determine the size of objects in the image, estimate the actual size of objects by recognizing the type of object, and estimate the depth of objects based on the size of objects in the image and the estimated actual size. In a combination of features, the disparity identified between adjacent images may be provided as input to the neural network. This may be possible using a point cloud. A point cloud is a collection of data points useful for identifying the motion of objects detected around a vehicle. Simultaneous localization of objects on adjacent images may be performed by obtaining and estimating the motion of objects using a point cloud.

[0045] These can be used to generate surround-view images of the vehicle (203). In one combination of features, data from any other sensor can be fused with the fisheye lens camera data to reduce invalid depth estimates within the occluded region.

[0046] In one of the alternative combinations described above, along with a single feature that can be combined with it, the vehicle model can be placed at the center of the surround view image (204). In one example, an animated model of the vehicle may be used. This allows the driver to obtain a bird's-eye view of the environment. Other appropriate overlays may also be added to the image showing the vehicle's position relative to objects seen by the fisheye lens camera.

[0047] In the combination of the alternative combinations described above and one possible feature, all objects that may be around the vehicle can be detected (205). This can be done using any existing or appropriate detection process. Detected objects can be mapped to the surround view image (206). Pose and depth estimates, as well as the calculated size, dimensions, and coordinates of the objects, can be used to accurately map the objects to the surround view image, thereby enabling the driver to correctly interpret the surroundings. This may enable the driver to safely operate the vehicle while parked, while detecting blind spots, and in confined or dark places. This step can also ensure that objects such as pillars and pedestrians are not obstructed if they are in blind spots or around corners. This provides assistance when parking trailers and multi-axle vehicles with more than two axles.

[0048] In the alternative combinations and possible feature combinations described above, feature maps are generated at different levels, which can then be concatenated and flattened for depth and pose estimation. Pose and depth updates can be applied to the current depth and pose estimates through retraction and translation of a cubic special orthogonal group transformation. This is done for pose augmentation between each image pair.

[0049] In confined or limited spaces, the vehicle must be moved back and forth multiple times. The aforementioned steps may be necessary as long as the vehicle is moving, and the driver will always receive assistance when needed while the vehicle is parked. This is especially possible using the aforementioned neural network as "MobileNet," which operates around the vehicle. of This is because it allows for sufficiently rapid classification of the environment.

[0050] In the combination of the alternative combinations and one feature that can be combined as described above, a 3D reconstruction of the environment can be generated using a surround-view image in which objects are mapped. As previously mentioned, the position of the objects, their movement, and estimated movement are also identified. The combination allows for the generation of a 3D image of the vehicle's environment, which may appear as a bowl-shaped image around the vehicle.

[0051] In one combination of features, the already defined process may run continuously until the vehicle's engine is turned off. In an alternative combination of features, the process may run continuously until the driving control unit is turned off.

[0052] In the alternative combinations described above and in combinations of one feature that can be combined, a system-on-a-chip (SoC) may be used to implement the described process. The SoC may require capacities for multiple camera inputs, an image signal processor and hardware acceleration for image adjustment and adjustment, an image processing unit for creating a vehicle model, and an image overlay and processing core for algorithmic analysis of images.

[0053] Figure 3 illustrates an example of one feature combination of the process described above. In one feature combination that can be combined with the alternative combinations described above, a set of wide-angle images captured by a fisheye lens is shown (301). The images are processed and distortion removed as described above. Pose and depth measurements (301) are performed using the captured images and a feature map is generated (302).

[0054] Using the superimposed images, parallax can be calculated, and pose and depth estimation can be used for object detection and estimation of its motion. Simultaneous localization can be performed to detect and calculate further details such as the object's coordinates, size, and movement (303).

[0055] Figure 4 shows the area around the vehicle. of This shows an example of a 3D reconstruction of the environment. This 3D image is of the area around the vehicle. of This shows the environment, which can be created by surround view images, as described in the explanation of Figure 2. Objects detected around the vehicle are mapped to the image. As the vehicle moves in the desired direction, its surroundings are mapped. ofThe environment changes, and objects change and move. These movements can be detected as described above, and appropriate surround-view images are continuously generated for 3D reconstruction so that the driver can safely move the vehicle. By providing this 3D reconstruction, the driver may be able to operate the vehicle in dark places, confined spaces, and parking lots, and may also be able to detect objects in blind spots. The process described above can provide clear information about the distance and depth of the surroundings. This may also make it possible to generate a high-density depth map that provides a clear field of view of targets and objects even when the environment is very dark with no lighting. A high-density depth map may have fewer erroneous pixels, and therefore the depth of each pixel in the image can be identified.

[0056] Figure 5 shows an example of a system for performing the process as described above. An alternative combination of the features that can be combined with the alternative combinations described above may include a system-on-a-chip (SoC500). This system may include an integrated circuit in which many elements of functionality and components for realizing the disclosed features are combined onto a single chip.

[0057] In one combination of features, one or more fisheye lens cameras (501) can be indirectly integrated into the SoC. As described in conjunction with the explanation in Figure 2, the fisheye lens camera captures the area around the vehicle. of It can help capture wide-angle images of the environment. Fisheye lens cameras can be mounted on vehicles and can help capture images with overlapping fields of view.

[0058] In one combination of features, the SoC may have one or more image processors (502) directly or indirectly integrated therein. Images captured by a fisheye lens camera may be processed using the image processor for the necessary modifications as described in the process above. The image processor may be appropriately configured to produce the processed image as needed. In one combination of features, the image processor may be coupled with an accelerometer for other necessary adjustments and processing of the image.

[0059] In one combination of features, the SoC may have one or more image processing units (504) appropriately configured to create models of vehicles and objects mapped to surround view images, as described above.

[0060] In one combination of features, the SoC may have one or more processing cores configured to perform analysis and create a three-dimensional image of the vehicle's surroundings, as described in the previous paragraph.

[0061] In particular, the present invention has been shown and described with respect to specific embodiments of the present invention, but those skilled in the art will see that various modifications of form and detail can be made to the present invention without departing from the spirit and scope of the invention as defined by the accompanying claims. The scope of the present invention is therefore indicated by the accompanying claims and, as such, encompasses all modifications that fall within the meaning and scope of equivalence of the claims. Common numbers used in the relevant drawings should be understood to refer to components that are similar or serve the same purpose.

[0062] As those skilled in the art will see, the terminology used herein is intended to describe various embodiments and is not intended to limit the invention. As used herein, singular articles (a, an, the) are also plural unless the context makes it clear otherwise. The term "comprises and / or comprising" as used herein is to be understood as indicating the presence of the specified features, integers, steps, actions, elements, and / or components, and not as to preclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0063] The specific order or hierarchy of blocks in the disclosed process / flowchart should be understood as an example of an exemplary approach. The specific order or hierarchy of blocks in the process / flowchart may be modified based on design preferences. Furthermore, some blocks may be combined or omitted. The accompanying method claims illustrate the use of various blocks in an exemplary order and are not intended to limit the use to any specific order or hierarchy presented.

[0064] The foregoing description is provided to enable those skilled in the art to carry out the various embodiments described herein. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the claims are not intended to be limited to the embodiments shown herein, but to be consistent with the entire scope in which the claims are expressed, and references to singular elements are intended to mean "one or more" rather than "only" unless specifically stated so. In this specification, the word "exemplary" is used to mean "serving as an example, case or illustration." Any embodiment described herein as "exemplary" is not necessarily construed as being preferable or advantageous to any other embodiment. Unless specifically stated otherwise, the term "several" means one or more. The combinations such as "at least one of A, B, or C", "one or more of A, B, or C", "at least one of A, B, and C", "one or more of A, B, and C", and "A, B, C, or any combination thereof" include any combination of A, B, and / or C, and may include multiple A's, multiple B's, or multiple C's. Specifically, "at least one of A, B, or C", "one or more of A, B, or C", "at least one of A, B, and C", "one or more of A, B, and C", and "A, B, C, or any combination thereof" can be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and any such combination may include one or more members of A, B, or C. All structural and functional equivalents of elements of various aspects described throughout this disclosure, which are currently known or will be known to those skilled in the art, are expressly incorporated by reference into this application and are intended to be encompassed by the claims. While this application relates to the invention described in the claims, it also includes the following other aspects. 1. Around the vehicle of In a method for generating a 3D reconstruction of the environment, The steps include capturing multiple wide-angle images of the area around the vehicle using multiple fisheye lens cameras mounted on the vehicle, The steps include creating a surround-view image of the vehicle from the multiple captured images by generating one or more feature maps, The steps include: calculating pose and depth estimates from the generated feature map using at least one convolutional neural network; The steps include detecting one or more objects in the plurality of captured images, The steps include mapping one or more objects detected around the vehicle model onto the created surround view image using the calculated attitude and depth estimates, A step of constructing the three-dimensional reconstruction of the environment using the surround view image and the mapped object, A method that includes this. 2. The step of capturing the aforementioned multiple wide-angle images is: The steps include capturing at least two wide-angle images having overlapping fields of view from the aforementioned multiple fisheye lens cameras, The steps include: calculating the disparity between the two captured wide-angle images; Using the fully calculated parallax, the steps include detecting the coordinates of the objects on the at least two captured wide-angle images and the dimensions of the objects on the images. The method described in item 1 above, further comprising: 3. The step of capturing multiple wide-angle images is, The steps include removing distortion from the captured image and correcting the geometric alignment within the distorted image, The steps include generating a point cloud by providing the calculated difference between the two captured wide-angle images as input to the convolutional neural network, Using the generated point cloud, the steps include: simultaneously determining the position of the object on the captured wide-angle images; and estimating the movement of the object. The method described in 1 or 2 above, further comprising: 4. The method according to any one of 1 to 3 above, wherein the method step is performed multiple times, and the steps of generating the feature map and calculating the pose and depth estimates include processing each of the wide-angle images captured by the feature extraction network and calculating augmented pose updates for each of the captured wide-angle images. 5. The method according to any one of 1 to 4 above, wherein the vehicle model is placed at the center of the generated surround view image. 6. The method of any one of the above 1 to 4, further comprising the step of continuously performing the method of 1 to 5 until the drive control unit in the vehicle is turned off. 7. Around the vehicle of In a device that generates a 3D reconstruction of the environment, Memory and Multiple fisheye lens cameras mounted on the aforementioned vehicle, One or more image processors connected to the aforementioned multiple fisheye lens cameras, One or more image processing units connected to the aforementioned multiple fisheye lens cameras, One or more processing units for carrying out the methods described in 1 to 6 above, A device that includes this. 8. A non-temporary computer-readable storage medium containing instructions that, when executed by a processor, perform any one of the methods described in 1 to 5 above. [Explanation of Symbols]

[0065] 100 Computing Environments 110 Processing Units 120 memory 140 storage 150 Input Devices 160 Output device 170 Communication Connection 500 SoC 501 Fisheye lens camera 502 Image Processors 504 Image Processing Unit

Claims

1. In a method for generating a three-dimensional reconstruction of the environment surrounding a vehicle, The method involves the following steps: (a) A step (200) of capturing multiple wide-angle images of the surroundings of the vehicle using multiple fisheye lens cameras (501) mounted on the vehicle, wherein the capturing step (200) includes capturing at least two wide-angle images having overlapping fields of view from the multiple fisheye lens cameras (501), (b) A step (203) to create a surround view image of the vehicle from multiple captured wide-angle images by (201) generating one or more feature maps and (202) using at least one convolutional neural network to calculate the pose and depth estimation of one or more objects in multiple captured wide-angle images from the generated feature maps, (c) A step (205) in which one or more objects are detected in a plurality of captured wide-angle images, wherein this detection step (205) is (c-1) A step of calculating the disparity between at least two captured wide-angle images, (c-2) Using the calculated parallax, the steps of detecting the coordinates of at least two objects on the captured wide-angle images and the dimensions of the objects on the wide-angle images. Step (205), which includes, (d) Using the calculated pose and depth estimates of one or more objects in multiple captured wide-angle images, the step (206) of mapping one or more detected objects in multiple captured wide-angle images to a created surround-view image, (e) Step (207) of constructing a three-dimensional reconstruction of the environment using surround view images and mapped objects, A method that includes this.

2. The step of capturing multiple wide-angle images is, The steps include removing distortion from the captured wide-angle image and correcting the geometric alignment within the distorted wide-angle image, The steps include generating a point cloud by providing the calculated disparity between at least two captured wide-angle images as input to a convolutional neural network, The process involves using the generated point cloud to simultaneously determine the positions of objects on at least two captured wide-angle images, and estimating the motion of the objects. The method according to claim 1, further comprising:

3. The method according to claim 1, wherein the steps of generating a feature map and calculating pose and depth estimation include processing each of the wide-angle images captured by the feature extraction network and calculating augmented pose updates for each of the captured wide-angle images.

4. The method according to claim 1, wherein the model of the vehicle is placed at the center of the generated surround view image.

5. A method for generating a three-dimensional reconstruction of the environment surrounding a vehicle, The above method involves the following steps: (a) A step (200) of capturing multiple wide-angle images of the surroundings of the vehicle using multiple fisheye lens cameras (501) mounted on the vehicle, wherein the capturing step (200) includes capturing at least two wide-angle images having overlapping fields of view from the multiple fisheye lens cameras, (b) A step (203) to create a surround view image of the vehicle from a plurality of captured wide-angle images by (201) generating one or more feature maps and (202) using at least one convolutional neural network to calculate the pose and depth estimation of one or more objects in a plurality of captured wide-angle images from the generated feature maps, (c) A step (205) in which one or more objects are detected in a plurality of captured wide-angle images, wherein this detection step (205) is (c-1) A step of calculating the disparity between at least two captured wide-angle images, (c-2) Using the calculated parallax, the steps of detecting the coordinates of at least two objects on the captured wide-angle images and the dimensions of the objects on the wide-angle images. Step (205), which includes, (d) Using the calculated pose and depth estimates of one or more objects in multiple captured wide-angle images, the step (206) of mapping one or more detected objects in multiple captured wide-angle images to a created surround-view image, (e) Step (207) of constructing a three-dimensional reconstruction of the environment using surround view images and mapped objects, Includes, The step of capturing multiple wide-angle images is further done in the following steps: The steps include removing distortion from the captured wide-angle image and correcting the geometric alignment within the distorted wide-angle image, The steps include generating a point cloud by providing the calculated disparity between at least two captured wide-angle images as input to a convolutional neural network, The process involves using the generated point cloud to simultaneously determine the positions of objects on at least two captured wide-angle images, and estimating the motion of the objects. including, and The steps of generating a feature map and calculating pose and depth estimates are as follows: The steps include processing each of the wide-angle images captured by the feature extraction network, A step of calculating an increasing pose update for each of the captured wide-angle images, Includes, The vehicle model is placed at the center of the generated surround view image. In this method, The method is characterized in that the above steps are performed until the drive control unit in the vehicle is turned off.

6. In a device that generates a three-dimensional reconstruction of the environment surrounding a vehicle, Memory and Multiple fisheye lens cameras (501) mounted on the vehicle, One or more image processors (502) connected to the plurality of fisheye lens cameras (501), One or more image processing units (504) connected to the plurality of fisheye lens cameras (501), An apparatus comprising one or more processing units for performing the method according to any one of claims 1 to 5.

7. A non-temporary computer-readable storage medium that, when executed by a processor, includes instructions that perform the method according to any one of claims 1 to 5.

Images