Method and device for generating three-dimensional restoration of vehicle ambient environment

Abstract

To provide a method and a device for generating a three-dimensional restoration of a vehicle ambient environment.SOLUTION: There may be provided a computer implementation method for generating a three-dimensional restoration of a vehicle ambient environment. This method may include the steps of: capturing a plurality of wide-angle images around a vehicle using a plurality of fisheye lens cameras installed on the vehicle; creating a surround view image around the vehicle from a plurality of captured images by generating one or more feature maps; and posture and depth estimation from the generated feature map using at least one convolution neural network. This method may also include the steps of: detecting one or more objects in the plurality of captured images; mapping the one or more objects detected around a vehicle model into the created surround view image using the calculated posture and depth estimation; and configuring an environmental three-dimensional restoration using the surround view image and the mapped object.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] Various combinations of features of the present disclosure relate to methods and apparatus for generating a three-dimensional reconstruction of a vehicle's surroundings, and more particularly, to detecting a scene around a vehicle. [Background technology]

[0002] Surround view systems are an important feature of vehicles and can be useful in many situations. Ultrasonic sensors used in surround view systems help drivers judge possible obstacles, but often fail due to poor detection. Surround view systems are also less sensitive in poorly lit areas and parking lots due to low lighting and crowded spaces. Existing deep learning models trained on vehicle light recognition for surround view systems also fail to produce reliable Time-To-Collision (TTC) values ​​on pitch black images at night. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 20210142095A1 [Non-patent literature]

[0004] [Non-Patent Document 1] B. Ivanovic et al., “Modeling multimodal dynamic spatiotemporal graphs” [Non-Patent Document 2] T. Salzmann et al., “Trajectron++: Multi-agent generative trajectory forecasting with heterogeneous data for control” Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above, there is a need for an improved method of detecting the vehicle's surrounding environment for an around view system. [Means for solving the problem]

[0006] In order to meet the above-mentioned needs, a method for generating a 3D reconstruction of an environment surrounding a vehicle is provided. The method may include capturing a plurality of wide-angle images of the environment surrounding the vehicle using a plurality of fisheye lens cameras mounted on the vehicle. The method may further include creating a surround view image of the environment surrounding the vehicle from the plurality of captured images by generating one or more feature maps, and calculating a pose and depth estimate from the generated feature maps using at least one neural network. The method may also include detecting one or more objects in the plurality of captured images. This further includes mapping the one or more objects detected around the model of the vehicle to the created surround view image using the calculated pose and depth estimate, and constructing a 3D reconstruction of the environment using the surround view images and the mapped objects. This may enable the creation of a "cone-shaped" image of the environment surrounding the vehicle, with the objects around the vehicle being appropriately mapped onto the created image. The neural network may be a combination of a feature extraction network and a 3D convolutional network.

[0007] In an alternative feature combination that may be combined with the alternative embodiment described above, capturing a plurality of wide-angle images includes capturing at least two wide-angle images with overlapping fields of view from a plurality of fisheye lens cameras. The method may include calculating a disparity between the at least two or more captured wide-angle images, and detecting coordinates of an object on the at least two or more captured wide-angle images and dimensions of the object on the images using the calculated disparity. This may allow a clear view of the surroundings of the vehicle to be obtained. This allows finding how far an object is from the vehicle, and the location of the object helps to safely operate the car even in smaller parking lots. The aforementioned "calculated disparity" may be calculated, for example, using a disparity map. The image may be converted to grayscale before calculating the disparity map. A disparity function may be used to calculate the disparity by comparing the difference of each block of pixels in the grayscale image. In an alternative feature combination, each pixel in one image is matched with its corresponding pixel in the other image. The distance or difference of each pair of matching pixels may be calculated. Finally, a disparity map can be obtained by expressing such distance values ​​as a luminance image. In other alternative feature combinations, other suitable processes can be used for calculating disparity, see, for example, patent publication (Patent Document 1), more specifically, paragraphs

[0035] to

[0071] .

[0008] In an alternative feature combination that may be combined with the alternative combination described above, the step of capturing a plurality of wide-angle images further includes a step of dewarping the captured images and a step of correcting the geometric alignment in the dewarped images. The method further includes a step of generating a point cloud by providing the calculated difference between the captured left and right images as input to a neural network, and a step of using the generated point cloud to estimate the motion of the object and perform simultaneous localization of the object on the left and right images. This may enable the use of the corrected and adjusted wide-angle images to create an environment surrounding the vehicle, thereby providing error-free and accurate results for object detection and other processing on the images. The method also helps to estimate the motion of the object, which helps the driver to operate the vehicle in a safe manner.

[0009] In alternative feature combinations that may be combined with the alternative combinations described above, the method steps are performed multiple times, and the steps of generating feature maps and calculating pose and depth estimates include processing each captured wide-angle image through a feature extraction network and calculating a pose incremental update for each captured wide-angle image. Feature maps at different resolutions are generated for feature extraction, and then can be combined, flattened, and passed to a fully connected layer in the neural network. The extracted features can be sent to a 3D convolutional network for depth and pose estimation. This is done for pose incremental updates between each adjacent pair. This can help the vehicle avoid hitting moving objects, and increase the safety of pedestrians around the vehicle. This also helps in tracking the object's orientation.

[0010] The feature extraction and feature detection process may include activating a YOLOv5 model trained on the training data to form a model, which serves as a basis for object detection in the received camera data. Preferably, the YOLOv5 model is formed for object detection with a MobileNet-type backbone network.

[0011] The YOLO model family can be composed of three main architectural blocks: i) the backbone, ii) the neck, and iii) the head. YOLOv5 Backbone: This uses CSPDarknet as the backbone for feature extraction from images, consisting of a cross-stage partial network. YOLOv5 Neck: It uses PANet to generate a feature pyramid network to perform aggregation on features and hand it over to the head for prediction. YOLOv5 head: A layer that generates predictions from anchor boxes for object detection.

[0012] A convolutional neural network (CNN) is a multi-layer feed-forward neural network that is built by stacking many hidden layers on top of each other in sequence. This sequential design may enable convolutional neural networks to learn hierarchical features. The hidden layers are typically convolutional layers followed by activation layers, some of which are followed by pooling layers. CNNs may be configured to identify patterns in data. Convolutional layers may contain convolutional kernels, which are used to look for patterns through the input data. The convolutional kernel returns a large positive value for portions of the input data that match the kernel's pattern, or a smaller value for other portions of the input data that do not match the kernel's pattern.

[0013] CNNs can extract informative features from training data without the need for manual processing of the training data. CNNs can produce accurate results involving large unstructured data, such as image classification, speech recognition, and natural language processing. CNNs are also computationally efficient because they can assemble increasingly complex patterns using relatively small kernels in each hidden layer.

[0014] Trajectron++ is a machine learning algorithm that uses a conditional variational autoencoder, a long short-term memory network, and a convolutional neural network. It is used to predict the future dynamics of some entities in a scene. The algorithm may be based on the Trajectron algorithm, which is described in detail in CoRR, abs / 1810.05993.2018, DOI http: / / arxiv.org / abs / 1810.05993 (Non-Patent Document 1) and CoRR.abs / 2001.00735, 2020, DOI https: / / arxiv.org / abs / 2001.03093 (Non-Patent Document 2). It can be used, for example, to predict the trajectory of at least partially autonomous vehicles.

[0015] The environment of an entity is presented as a directed spatiotemporal graph. Nodes represent surrounding entities, connected through spatial edges that connect different entities and thus affect each other within the same timestep, covering interactions between different entities, and through temporal edges that connect the same nodes through time, thus representing past dynamics. An interaction between two different entities occurs if their distance is smaller than an attention radius that can be chosen independently and separately for each entity class of the two interacting entities. An implementation of the Trajectron++ algorithm can be found at https: / / github.com / StanfordASL / Trajectron-plus-plus. For training, for example the ETH and UCY pedestrian datasets can be used.

[0016] MobileNet is an architectural model of convolutional neural networks for object detection optimized primarily for speed. The main building block of MobileNet is depthwise separable convolution, which decomposes or separates standard convolution filters into two different operations: (i) a first operation ("depthwise convolution") in which separate convolution kernels, also called depthwise in convolution, are applied to the input channels, and (ii) a second operation ("pointwise convolution") in which a pointwise (1×1) convolution is used to combine the information of the first operation. On the other hand, standard convolution filters perform channel-wise and spatial direction calculations in one step. Decomposing or separating standard convolution into two different operations has fewer parameters and is less computationally expensive than standard convolution due to fewer mult-adds (multiplication and addition operations).

[0017] In some embodiments, the CNN may include a first standard convolutional layer, followed by multiple depth-wise and point-wise convolutional layers, an average pooling layer, a fully connected layer, and a softmax classifier. Each layer of the CNN may be followed by batch normalization (BN) and rectified linear activation function (ReLU) nonlinearities, except for the final fully connected layer, which has no nonlinearities and is fed into a softmax layer for classification.

[0018] In an alternative feature combination that can be combined with the alternative combinations described above, a model of the vehicle is placed at the center of the generated surround view image, which allows the driver to interpret the distance of objects relative to the position of the car and also helps the driver detect the operation of the car.

[0019] In an alternative feature combination, which may be combined with the alternative combinations described above, the method runs continuously until the vehicle engine is shut off, allowing the driver to continuously sense the environment and operate the vehicle.

[0020] In an alternative feature combination that may be combined with the alternative combinations described above, an apparatus for generating a three-dimensional reconstruction of a vehicle's surroundings is provided, comprising a memory, a plurality of fisheye lens cameras mounted on the vehicle, one or more image processors coupled to the plurality of fisheye lens cameras, one or more image processing units coupled to the plurality of fisheye lens cameras, and one or more processing cores for performing the aforementioned method.

[0021] In alternative feature combinations which may be combined with the alternative combinations described above, there is provided a vehicle including the apparatus described above.

[0022] In an alternative feature combination that may be combined with the alternative combinations described above, there is provided a non-transitory computer readable storage medium comprising instructions which, when executed by a processor, perform the aforementioned method.

[0023] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the drawings, as follows: [Brief description of the drawings]

[0024] [Figure 1] One embodiment of a general purpose computing machine for implementing the present disclosure is described below. [Diagram 2] 1 is a flow chart of an embodiment of a process disclosed in the present disclosure. [Diagram 3] 1 is a flow diagram of an embodiment of a process disclosed in the present disclosure. [Figure 4] This concerns an example of the reconstruction of the vehicle's surroundings. [Diagram 5] The present invention relates to an example of a system for carrying out the disclosed processes. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Combinations of features described below with respect to the apparatus are equally valid for the respective methods and vice versa. Furthermore, it is to be understood that combinations of features described below may be considered together, e.g., parts of one combination may be combined with parts of other combinations.

[0026] It should be understood that any feature described herein with respect to a particular device may apply to any device described herein. It should be understood that any feature described herein with respect to a particular method may apply to any method described herein. Furthermore, it should be understood that with respect to any device or method described herein, not all of the components or steps described must necessarily be included in the device or method, and only some (but not all) of the components or steps may be included.

[0027] In alternative feature combinations that may be combined with the alternative combinations described above, the present disclosure relates to representing the vehicle's surroundings, which may assist in monitoring, warning, braking, steering, and other such tasks. This may be useful in dark, poorly lit, congested spaces where visual cues may be limited. The vehicle may be an automobile, or any multi-axle vehicle with more than two axles, or a vehicle with a trailer attached.

[0028] Throughout this specification, the term "pose estimation" may mean predicting and tracking the position of an object in an image. This may involve predicting the deformation of an object from a user-specified reference pose for an image. The image data from which the pose of an object is determined may be a single image, a stereo image pair, or an image sequence.

[0029] Throughout this specification, the term "depth" may have the meaning of the perpendicular distance between an object and the plane of the scene camera. Each pixel in an image may have a depth value, indicating the pixel's distance from the camera plane.

[0030] 1 illustrates a generalized example of a suitable computing environment 100 that may be used in implementing the present disclosure. The computing environment 100 is not intended to suggest any limitation as to scope of use or functionality, as the technology may be implemented in a variety of general-purpose or special-purpose computing environments.

[0031] Referring to FIG. 1, a computing environment 100 includes at least one processing unit 110 coupled to a memory 120. In FIG. 1, this basic configuration 130 is included within a dashed line. The processing unit 110 executes computer-executable instructions. In a multiprocessing system, multiple processing units execute computer-executable instructions to increase processing power. The memory 120 can be non-transitory memory such as volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or any combination of the two. The memory 120 can store software 180 that implements any of the techniques described herein.

[0032] A computing environment may have other features as well. For example, computing environment 100 includes storage 140, one or more input devices 150, one or more output devices 160, and one or more communication connections 170. An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of computing environment 100. Typically, operating system software (not shown) provides an operating environment for the execution of other software executing within computing environment 100 and coordinates the activities of the components of computing environment 100.

[0033] Storage 140 may be removable or non-removable and may include a magnetic disk, magnetic tape, or any other non-transitory computer-readable medium that can be used to store information and that can be accessed within computing environment 100. Storage 140 may store software 180, including instructions for any of the techniques described herein.

[0034] Input device(s) 150 may be a keyboard, a touch input device such as a touch screen, a voice input device, a scanning device, or other device that provides input to computing environment 100. Output device(s) 160 may be a display, a speaker, or other device that provides output from computing environment 100. Some input / output devices, such as a touch screen, may include both input and output capabilities.

[0035] The communications connection(s) 170 enable communication with other computing entities through communications mechanisms that convey information, such as computer-executable instructions, audio / video, or other information, or other data. For example, and without limitation, communications mechanisms may include wired or wireless technologies implemented with electrical, optical, RF, infrared, acoustic, or other carriers.

[0036] The technology herein may be generally described in terms of computer-executable instructions, such as those included in program modules, executed on a target real or virtual processor in a computing environment. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided between program modules as desired in various combinations. Computer-executable instructions for program modules may be executed in a local or distributed computing environment.

[0037] Any of the storage operations described herein may be performed by storage on one or more computer-readable media (e.g., non-transitory computer-readable storage media or other tangible media). Anything described as being stored may be stored on one or more computer-readable media (e.g., computer-readable storage media or other tangible media).

[0038] Any of the methods described herein may be performed by computer-executable instructions in (e.g., encoded on) one or more computer-readable media (e.g., non-transitory computer-readable storage media or other tangible media). Such instructions can cause a computer to perform the method. The techniques described herein may be implemented in a variety of programming languages.

[0039] Any of the methods described herein may be performed by computer-executable instructions stored on one or more non-transitory computer-readable storage media devices (e.g., memory, CD-ROM, CD-RW, DVD, or the like). Such instructions may cause a computer to perform the method.

[0040] In this combination of features, which may be combined in alternative combinations, the computing environment (100) may be implemented within a vehicle suitably configured with memory, processor, input devices, and other components previously described, including software instructions. In other combinations of features, the computing environment may be implemented in a system-on-chip (SoC) or other embedded system.

[0041] In the alternative feature combinations described above and alternative feature combinations that may be combined, the process of this patent application is described with reference to FIG. 2. In one feature combination, multiple wide-angle images of the vehicle's surroundings may be captured (200). This may be done by one or more fisheye lens cameras that may be mounted on the vehicle. In one example, a surround view system may use four to six wide-angle or fisheye lens cameras mounted on the front, rear, and sides of the vehicle. This may allow the wide-angle images captured by the fisheye lens cameras to be used to detect and identify the vehicle's surroundings. The use of wide-angle images may help capture images with a larger field of view that may also overlap with adjacent images, which may also aid in object detection and location, as described in more detail in the following paragraphs.

[0042] In one combination of features that may be combined with the alternative combinations described above, an image captured by a fisheye lens camera may be distorted, misaligned, or in need of some other correction. The image may first be processed to remove the distortion. This may be done by geometric alignment correction. The image may be further corrected for balancing and color correction. The distortion coefficients may be used to remove the image distortion. This may ensure that the further analysis and processing described in the following paragraphs is more accurate and error-free.

[0043] At least two adjacent images with overlapping fields of view in one feature combination may be selected. The selected images may be overlaid to identify disparity between the images. The disparity may be pixel difference or motion difference. The identified disparity may help to calculate alignment difference between the images to detect actual position and movement of the object in both images. This may also help to calculate the size, dimensions, and coordinates of the object. This may allow the vehicle to detect how far away and how big the object is present in the vehicle's environment. Based on this calculation, the object may be correctly mapped onto the surround view image around the vehicle.

[0044] In one feature combination, which may be combined with the alternative combinations described above, one or more feature maps may be generated using multiple wide-angle images (201). The feature maps may enable the computation of pose and depth estimation (202). This may be done using one or more convolutional neural networks. The convolutional neural networks may identify key points in the images for estimation. The depth estimation neural network may detect objects in the images, identify the size of the objects in the images, infer the actual size of the objects by recognizing the type of the objects, and estimate the depth of the objects based on the size of the objects in the images and the estimated actual size. In one feature combination, the disparity identified between adjacent images may be provided as input to the neural network. This may be possible with a point cloud. The points are a collection of data points useful for identifying the motion of the detected objects around the vehicle. The simultaneous localization of objects on adjacent images may be performed by obtaining the object's movement and estimating its movement using the point cloud.

[0045] These can be used to generate a surround view image of the vehicle's surroundings 203. In one feature combination, apart from the fisheye camera data, any other sensor data can be fused to reduce invalid depth estimates in occluded regions.

[0046] In one feature combination that may be combined with the alternative combinations described above, a model of the vehicle may be centered in the surround view image (204). In one example, an animated model of the vehicle may be used. This may give the driver a bird's-eye view of the environment. Other suitable overlays may also be added to the image that indicate the position of the vehicle relative to objects seen by the fisheye lens camera.

[0047] In one feature combination, which may be combined with the alternative combinations described above, all possible objects around the vehicle may be detected (205). This may be done using any existing or suitable detection process. The detected objects may be mapped to a surround view image (206). The pose and depth estimates as well as the calculated size, dimensions, and coordinates of the objects may be used to accurately map the objects to the surround view image, allowing the driver to correctly interpret the surroundings. This may enable the driver to safely operate the vehicle while parking or during blind spot detection and in occluded or dark areas. This step may also ensure that objects such as pillars, pedestrians, etc. are not occluded if they are in the blind spot or around a corner. This provides assistance during parking of trailers and multi-axle vehicles with more than two axles.

[0048] In one feature combination that can be combined with the alternative combinations above, feature maps can be generated at different levels and then concatenated and flattened for depth and pose estimation. Pose and depth updates can be applied to the current depth and pose estimates through retraction and translation of a special orthogonal group transform of order 3. This is done for incremental pose updates between each image pair.

[0049] In closed or limited spaces, the vehicle must move back and forth many times. The steps mentioned above may be necessary as long as the vehicle is moving, and the driver always has assistance when necessary while parking the vehicle. This is especially possible with the neural network mentioned above as "MobileNet", because this neural network can classify the vehicle's surroundings quickly enough.

[0050] In one feature combination, which may be combined with the alternative combinations described above, a 3D reconstruction of the environment may be generated using surround view images onto which objects are mapped. As previously described, the location of the object, its motion, and estimated motion are also identified. In combination, this allows for the generation of a 3D image of the vehicle's environment, which may appear as a cone-shaped image of the vehicle's surroundings.

[0051] In one combination of features, the process defined above may be run continuously until the vehicle engine is turned off. In an alternative combination of features, the process may be run continuously until the driving control unit is turned off.

[0052] In one feature combination, which may be combined with the alternative combinations described above, a system on a chip (SoC) may be used to implement the described process. The SoC may require capacity for multiple camera inputs, an image signal processor and hardware acceleration for image conditioning and adjustment, an image processing unit for creating a vehicle model, and image overlay and processing cores for algorithmic analysis of the images.

[0053] Figure 3 relates to an example of one feature combination of the above process. In one feature combination that can be combined with the alternative combinations described above, a collection of wide-angle images captured with a fisheye lens is shown (301). The images are de-warped and processed as described above. Pose and depth measurement (301) is performed using the captured images and a feature map is generated (302).

[0054] Using the overlapped images, disparity can be calculated and pose and depth estimates can be used for object detection and estimation of its motion. Co-localization can be performed (303) to detect and calculate further details such as object coordinates, size, and movement.

[0055] FIG. 4 shows an example of a 3D reconstruction of the vehicle's surroundings. This 3D image shows the vehicle's surroundings, which may be created by a surround view image, as described with reference to FIG. 2. Objects detected around the vehicle are mapped onto the image. When the vehicle moves in the required direction, its surroundings change, and objects may also change and move. These movements may be detected as described above, and appropriate surround view images are continuously generated for the 3D reconstruction to allow the driver to safely navigate the vehicle. Providing this 3D reconstruction may enable the driver to operate the vehicle in dark areas, occluded areas, parking lots, and may also enable the driver to detect objects in blind spots. The process described above may provide clear information about the distance and depth of the surroundings. It may also be possible to generate a dense depth map that provides a clear view of targets and objects, even when the environment is unlit and very dark. A dense depth map may have fewer erroneous pixels, and thus the depth of each pixel in the image may be identified.

[0056] Figure 5 shows an example of a system for performing the process as described above. One alternative combination of features that may be combined with the alternative combinations described above may include a system on a chip (SoC 500). The system may include an integrated circuit in which many elements of functionality and components for implementing the disclosed features are combined into one chip.

[0057] In one combination of features, one or more fisheye lens cameras (501) may be indirectly integrated into the SoC. As discussed in conjunction with the description of FIG. 2, the fisheye lens cameras may aid in capturing wide-angle images of the vehicle's surroundings. The fisheye lens cameras may be mounted on the vehicle and aid in capturing images with overlapping fields of view.

[0058] In one feature combination, the SoC may have one or more image processors (502) directly or indirectly integrated therein. Images captured by the fisheye camera may be processed using the image processor for necessary modifications as described in the above process. The image processor may be appropriately configured to generate processed images as required. In one feature combination, the image processor may be coupled with an accelerometer for other necessary adjustments and processing of the images.

[0059] In one combination of features, the SoC may have one or more image processing units (504) suitably configured to create models of vehicles and objects that are mapped to the surround view image, as described above.

[0060] In one combination of features, the SoC may have one or more processing cores configured to perform analysis and create a three-dimensional image of the vehicle's surroundings, as described in the preceding paragraph.

[0061] Although the present invention has been particularly shown and described with respect to specific embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the present invention is therefore indicated by the appended claims and is therefore intended to embrace all changes that come within the meaning and range of equivalency of the claims. It should be understood that common numbers used in related drawings refer to components that are similar or serve the same purpose.

[0062] As will be appreciated by those skilled in the art, the terminology used herein is for the purpose of describing various embodiments and is not intended to limit the invention. As used herein, the singular articles (a, an, the) are intended to include the plural, unless the context clearly dictates otherwise. The terms "comprises" and / or "comprising", as used herein, are to be understood as specifying the presence of stated features, integers, steps, operations, elements, and / or components and not excluding the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0063] It should be understood that the particular order or hierarchy of blocks in the disclosed processes / flow charts is illustrative of example approaches. Based on design preferences, it should be understood that the particular order or hierarchy of blocks in the processes / flow charts may be modified. Additionally, some blocks may be combined or omitted. The accompanying method claims present the use of various blocks in an example order and are not intended to be limited to the particular order or hierarchy presented.

[0064] The foregoing description is provided to enable one skilled in the art to practice the various aspects described herein. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Thus, the claims are not intended to be limited to the aspects set forth herein but are to be accorded the full scope consistent with the language of the claims, and references to elements in the singular are intended to mean "one or more" and not "only one" unless specifically so stated. As used herein, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term "some" means one or more. Combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" include any combination of A, B, and / or C, and may include a plurality of A, a plurality of B, or a plurality of C. Specifically, "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and any such combination may include one or more members of A, B, or C. All structural and functional equivalents to the elements of the various embodiments described throughout this disclosure that are now known or that later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. [Explanation of symbols]

[0065] 100 Computing Environments 110 Processing Unit 120 Memory 140 Storage 150 Input Device 160 Output Device 170 Communication Connections 500 SoC 501 Fisheye Lens Camera 502 Image Processor 504 Image Processing Unit

Claims

1. 1. A method for generating a three-dimensional reconstruction of an environment surrounding a vehicle, comprising: capturing a plurality of wide angle images of a surrounding of the vehicle using a plurality of fisheye lens cameras mounted on the vehicle; generating one or more feature maps to create a surround view image of an area surrounding the vehicle from the plurality of captured images; computing pose and depth estimates from the generated feature maps using at least one convolutional neural network; detecting one or more objects in the plurality of captured images; using the calculated pose and depth estimates to map one or more objects detected around the vehicle model onto the created surround view image; constructing the 3D reconstruction of the environment using the surround view images and the mapped objects; The method includes:

2. The step of capturing a plurality of wide angle images includes: capturing at least two wide-angle images having overlapping fields of view from the plurality of fisheye lens cameras; calculating a disparity between the at least two captured wide-angle images; using all the calculated disparities to detect coordinates of an object on the at least two or more captured wide-angle images and dimensions of the object on the images; The method of claim 1 further comprising:

3. The step of capturing a plurality of wide angle images includes: dewarping the captured image and correcting geometric alignment in the dewarped image; generating a point cloud by providing the calculated difference between the at least two captured wide-angle images as inputs to the convolutional neural network; using the generated point cloud to perform simultaneous localization of the object on the at least two captured wide-angle images and to estimate the motion of the object; The method of claim 1 or 2, further comprising:

4. 4. The method of claim 1, wherein the method steps are performed multiple times, and wherein generating the feature map and computing the pose and depth estimates comprises: processing each of the captured wide-angle images with a feature extraction network; and computing a pose augmentation update for each of the captured wide-angle images.

5. A method according to any one of claims 1 to 4, wherein the vehicle model is centred in the generated surround view image.

6. A method according to any one of claims 1 to 4, comprising the step of continuously carrying out the method according to claims 1 to 5 until a drive control unit in the vehicle is switched off.

7. 1. An apparatus for generating a three-dimensional reconstruction of an environment surrounding a vehicle, comprising: Memory, A plurality of fisheye lens cameras mounted on the vehicle; one or more image processors coupled to the plurality of fisheye lens cameras; one or more image processing units coupled to the plurality of fisheye lens cameras; one or more processing units for carrying out the methods according to claims 1 to 6; An apparatus comprising:

8. A non-transitory computer readable storage medium comprising instructions which, when executed by a processor, perform the method of any one of claims 1 to 5.

Images