Methods, systems, articles of manufacture, and apparatus for reconstructing scenes using convolutional neural networks

Abstract

Methods, apparatus, systems, and articles of manufacture for reconstructing scenes using convolutional neural networks are disclosed. An example apparatus includes: a sensor data acquirer for acquiring ground truth data representing an environment; an environment detector for identifying environmental characteristics of the environment; a synthetic database builder for applying noise to the ground truth data to form a training set; a model builder for training a machine learning model using the training set and the ground truth data; and a model adjuster for modifying the machine learning model to include residual OR gate connections between corresponding layers of the machine learning model. The synthetic database builder is also used to store the machine learning model in association with the environmental characteristics of the environment.

Description

[0001] Related Applications

[0002] U.S. Provisional Patent Application No. 62 / 674,462 is hereby incorporated by reference in its entirety. Priority is hereby claimed to U.S. Provisional Patent Application No. 62 / 674,462. Technical Field

[0003] The present disclosure relates generally to scene reconstruction, and more particularly, to methods, systems, articles of manufacture, and apparatus for reconstructing scenes using convolutional neural networks. Background Art

[0004] Image and video data have become extremely available in recent years, especially with the advent of static cameras (e.g., surveillance cameras) and dynamically moving cameras (e.g., quad-rotor cameras, drones, etc.). Although the human brain includes the ability to infer objects from a single viewpoint based on previous experience, one or more gaps in the captured image and video data may cause problems for machine-based interpretation and / or analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a schematic illustration of an example reconstruction system constructed in accordance with the teachings of the present disclosure.

[0006] Figure 2 Is Figure 1 An example reconstruction system implements an OR-like architecture to reconstruct a scene using a convolutional neural network.

[0007] Figure 3 is a flow chart representing machine readable instructions that may be executed to implement creation of a synthetic database.

[0008] Figure 4 is a flow chart representing machine readable instructions that may be executed to implement scene reconstruction.

[0009] Figure 5 is constructed to execute Figure 3 and / or Figure 4 The instructions to achieve Figure 1 and Figure 2 Block diagram of an example processing platform of an example reconstruction system.

[0010] The drawings are not drawn to scale. Rather, the thickness of layers or regions may be exaggerated in the drawings. DETAILED DESCRIPTION

[0011] The acquired image data may include errors and / or gaps caused by one or more factors. In some examples, the errors and / or gaps in the acquired image data are caused by highly reflective surfaces that saturate the image acquisition device (e.g., a light detection and ranging (LiDAR) system). In some examples, the errors and / or gaps are caused by occlusions in the scene (e.g., trees, vehicles, buildings, etc.). In examples disclosed herein, machine learning is used to create one or more models that can be used to process input sensor data to correct errors in the input data (e.g., gaps caused by occlusions in the scene).

[0012] Machine learning (ML) and / or artificial intelligence (AI) enable machines (e.g., computers, logic circuits, etc.) to generate outputs based on given inputs. AI enables machines to use models to process inputs and generate outputs. ML enables models to be trained so that specific (multiple) inputs lead to specific (multiple) outputs. Typically, implementing an ML / AI system involves two phases, a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train a model to create a specific output given a specific input based on, for example, training data. Typically, a model contains internal parameters that guide how to convert input data into output data. In addition, hyperparameters are used as part of the training process to control the way learning is performed (e.g., learning rate, number of layers to be used in a machine learning model, etc.). Once trained, other data (e.g., real-time data) is input into the model in the inference phase to create outputs. The inference phase is when the AI ​​is "thinking" to generate outputs based on what it has learned from training (e.g., from a model). In some examples, the input data is preprocessed before being used as input to a machine learning model. Also, in some examples, after output data is generated by an AI model, the output data may be post-processed to convert the output into a useful result (e.g., instructions to be executed by a machine).

[0013] There are many different types of machine learning models and / or machine learning architectures. In the examples disclosed herein, a convolutional neural network (CNN) model is used. In the examples disclosed herein, an additional OR gate structure is added to the machine learning model to reduce false positives (e.g., reduce the possibility of introducing errors through the processing of the machine learning model). The use of a CNN model enables processing of incoming sensor data of any size. However, other model architectures, such as a deep neural network (DNN), a support vector machine (SVM), etc., may be used in addition or alternatively.

[0014] As noted above, ML / AI models are trained to produce specific outputs given specific inputs. Different types of training can be performed based on the type of ML / AI model and / or the expected outputs. For example, supervised training uses inputs and corresponding expected outputs to select parameters for the ML / AI model that reduce model error (e.g., by iteratively selecting combinations of parameters).

[0015] In the examples disclosed herein, the ML / AI model is trained using ground truth data (e.g., selected sensor data known not to include noise) and ground truth data with added noise. That is, the ground truth data with added noise is used as training input, and the ground truth data (without added noise) is used as the expected output. In the examples disclosed herein, the training is performed at a central facility (e.g., in a cloud computing environment), but can be performed additionally or alternatively at any other computing environment.

[0016] Once training is complete, the model is stored in a database. The database can be located at a server location (e.g., in a cloud computing environment). In some examples, the model can be provided to an edge device (e.g., an edge user computer, a drone, an Internet of Things (IoT) device, etc.). The model can then be executed by the server and / or edge device to generate a reconstructed scene. This approach is useful in scenarios where a drone is operating in an environment and attempting to perform scene reconstruction for navigation and / or collision avoidance purposes.

[0017] Figure 1 is a schematic illustration of an example reconstruction system 100. Figure 1 In the illustrated example, the reconstruction system 100 includes an example scan reconstruction manager 101, (multiple) sensors 102, and a sensor data storage device (103). The example scan reconstruction manager 101 includes an example sensor data acquisition device 104, an example environment detector 106, an example model selector 108, an example model builder 110, an example model adjuster 112, an example error characteristic evaluator 114, an example synthesis database builder 116, an example synthesis data storage device 124, and an example scene reconstructor 130. An example bus 118 communicatively connects the above structures of the example reconstruction manager 101. The example sensor data acquisition device 104 is also communicatively connected to the example sensor data source 103 and / or the example sensor (multiple) 102.

[0018] Figure 1The (multiple) example sensors 102 of the illustrated example are implemented by one or more data acquisition sensors, for example, one or more light detection and ranging (LiDAR) sensors, image sensors, distance sensors, ultrasonic sensors, etc. In some examples, multiple different types of sensors are used. In the examples disclosed herein, (multiple) sensors 102 provide data to the sensor data acquirer 104. In some examples, the data from (multiple) sensors 102 are stored in the sensor data storage device 103. In some examples, the sensor data is stored and / or otherwise provided as voxels (volume elements) indicating image information in three-dimensional space. In some other examples, the sensor data is provided as image data for generating voxel representations. In some examples, the sensor data is provided as a point cloud for generating voxel representations. In some examples, the voxels include binary values ​​indicating the occupancy of points and / or areas in three-dimensional space (e.g., whether a particular space is consumed by an object). In some examples, such voxel information includes additional information, for example, color (e.g., RGB (red, green, blue)) data, brightness data, etc.

[0019] Figure 1 The example sensor data storage device 103 of the illustrated example is implemented by any memory, storage device, and / or storage disk for storing data, such as flash memory, magnetic media, optical media, solid-state memory, (multiple) hard drives, (multiple) thumb drives, etc. In addition, the data stored in the example sensor data storage device 103 can be in any data format, such as binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, etc. Although in the illustrated example, the sensor data storage device 103 is shown as a single device, the example sensor data storage device 103 and / or any other data storage device described herein may be implemented by any number and / or type of memory. Figure 1 In the illustrated example, the example sensor data storage device 103 stores sensor data (e.g., cached sensor data collected by (multiple) sensors 102) and / or ground truth data (e.g., selected sensor data that is known to not include noise).

[0020] Figure 1The example sensor data acquirer 104 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may be used in addition or alternatively, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) application specific integrated circuits (ASICs), (multiple) programmable logic devices (PLDs), (multiple) field programmable logic devices (FPLDs), (multiple) digital signal processors (DSPs), etc. The example sensor data acquirer 104 acquires ground truth data and / or sensor data (e.g., images, videos, etc.). In some other examples, the sensor data and / or ground truth data is provided as image data, and the example sensor data acquirer 104 uses the image data to generate a voxel representation. In some examples, the sensor data and / or ground truth data is provided as a point cloud, and the example sensor data acquirer 104 uses the point cloud to generate a voxel representation.

[0021] Figure 1 The example environment detector 106 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may be used in addition or alternatively, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) ASICs, (multiple) PLDs, (multiple) FPLDs, (multiple) programmable controllers, (multiple) GPUs, (multiple) DSPs, etc. The example environment detector 106 identifies characteristics and / or (multiple) characteristic types of ground truth data and / or sensor data. In the examples disclosed herein, such characteristics include identifying whether the environment is an urban environment (e.g., a city landscape) or a rural environment. In some examples, additional characteristics of the environment may be identified, such as the types of buildings and / or buildings that are typically present in the environment. For example, modern urban landscapes / buildings (e.g., San Francisco, New York, etc.) may produce different models from other urban landscapes / buildings (e.g., Siena, Italy; Dublin, Ireland). In some examples, metadata (e.g., Exchangeable Image File Format (EXIF) data) included in the sensor data may be used to determine the geographic location of the environment.

[0022] Figure 1The example model selector 108 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may additionally or alternatively be used, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) ASICs, (multiple) PLDs, (multiple) FPLDs, (multiple) programmable controllers, (multiple) GPUs, (multiple) DSPs, etc. The example model selector 108 selects a synthetic database for use in scene reconstruction based on (multiple) environmental characteristics identified by the environmental detector 106. In the examples disclosed herein, the synthetic database represents one or more machine learning models to be used when performing scene reconstruction.

[0023] Figure 1 The example model builder 110 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may be used in addition or alternatively, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) ASICs, (multiple) PLDs, (multiple) FPLDs, (multiple) programmable controllers, (multiple) GPUs, (multiple) DSPs, etc. The example model builder 110 divides the ground truth data into training sets. In the example disclosed herein, three training sets are used. However, any other number of training sets may be used in addition or alternatively. In some examples, each training set includes the same data as other training sets. This approach allows various levels of noise to be applied to the training set by the synthetic database builder 116. However, in some other examples, a single training set is used, and a progressive amount of noise is applied to the training set. In the example disclosed herein, the training sets have the same size. However, any size of training data may be used in addition or alternatively.

[0024] The example model builder 110 uses the training set (including the noise applied by the synthetic database builder 116) to train the model. In the example disclosed herein, the model is implemented using a convolutional neural network (CNN). However, any other machine learning model architecture may be used in addition or alternatively. In the example disclosed herein, the example model builder 110 uses stochastic gradient descent to train the model. However, any other training algorithm may be used in addition or alternatively. In the example disclosed herein, for each training set, training is performed until an acceptable error level between the validation data set (e.g., a portion of the ground truth data and / or training data to which noise is applied) and the ground truth data is reached.

[0025] The example model builder 110 then determines whether there are any additional training sets, and if so, repeats the training until all training sets have been used. In the examples disclosed herein, variable noise amounts may be applied to various training sets (e.g., by the synthetic database builder 116). For example, in a first iteration, a first noise amount may be applied to a first training set (e.g., 10% noise), and in a second iteration, a second noise amount different from the first noise amount may be applied to a second training set, e.g., 20% noise. In the examples disclosed herein, three training sets are used, with noise values ​​of 10%, 20%, and 50%, respectively. Using increasing amounts of noise improves the convergence of the model because the model gradually improves itself in scenarios where the noise gradually increases.

[0026] Figure 1 The example model adjuster 112 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may be used in addition or alternatively, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) ASICs, (multiple) PLDs, (multiple) FPLDs, (multiple) programmable controllers, (multiple) GPUs, (multiple) DSPs, etc. Once the model builder 110 has used all training sets to generate a model, the example model adjuster 112 utilizes a residual OR gate connection to adapt and / or otherwise enhance the model. Therefore, in some examples, the model adjuster 112 may be referred to as an OR gate customizer. As noted above, the added OR gate improves system accuracy by eliminating false inferences. That is, the OR gate architecture ensures that the final result of the model (e.g., when used in conjunction with real-time data) does not result in the creation of erroneous voxel data. In other words, the performance of the system is guaranteed to be no worse than the input data.

[0027] Typical CNN techniques operate in a manner similar to regression that produces a certain degree of error. In some conventional approaches, CNN techniques employ a logical “AND” operation between image input data and one or more convolutional layers. Thus, if a noisy and / or otherwise imperfect (e.g., containing errors / gaps) input sensor does not contain information about a particular voxel indicating the current structure, and the CNN layer suggests a non-zero probability of indicating the current structure, the logical “AND” architecture produces a zero-valued output. Similarly, if the noisy input includes voxels indicating a structure (e.g., a binary value “1” in the voxel), but the CNN layer iteration does not indicate a structure, the logical “AND” architecture produces a zero-valued output. Thus, conventional CNN techniques may result in output image reconstructions that are worse than the initial input image (e.g., containing more errors when compared to the real data). This effect is exacerbated in environments where gaps in the input sensor data are common.

[0028] The examples disclosed herein improve scene reconstruction in part by employing an OR-like gate in the residual connection by the example model adjuster 112, thereby eliminating spurious inferences (e.g., false positive identifications). Figure 2 An example class "OR" architecture 200 is shown. Figure 2 In the illustrated example of , the example architecture 200 includes an encoder stage 202 that includes a first network layer 204 , a second network layer 206 , and a third network layer 208 . Figure 2 The illustrated example also includes a decoder stage 210, which includes corresponding fourth network layer 212, fifth network layer 214, and sixth network layer 216. To eliminate spurious inferences during the convolution activities of the example scan reconstruction manager 101, the example architecture 200 includes corresponding OR gates 218, 219, 220 that accept inputs associated with the image data and intermediate convolution outputs. In some examples, the architecture 200 generates one or more synthetic data sources / models adapted in conjunction with specific nuances of the source image data.

[0029] return Figure 1 , Figure 1 The example error characteristic evaluator 114 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may be used in addition or alternatively, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) ASICs, (multiple) PLDs, (multiple) FPLDs, (multiple) programmable controllers, (multiple) GPUs, (multiple) DSPs, etc. The example error characteristic evaluator 114 determines typical error characteristics associated with the environment of interest. For example, an urban environment (e.g., an environment with metal structures and / or glass windows) may encounter reflections in the sensor data. Such reflections may cause themselves to appear as unoccupied voxels (when the voxel should actually have an occupied value). The example synthetic database builder 116 uses the error characteristics to impose a specific type and / or amount of error (e.g., noise) on the ground truth data.

[0030] Figure 1The example synthetic database builder 116 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may be used in addition or alternatively, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) ASICs, (multiple) PLDs, (multiple) FPLDs, (multiple) programmable controllers, (multiple) GPUs, (multiple) DSPs, etc. The example synthetic database builder 116 injects environment-specific errors / noise into a first training set of ground truth data. In order to apply noise to the training set, the example synthetic database builder 116 switches one or more voxels from an occupied value (e.g., indicating that an object exists at a specific location) to an unoccupied value (e.g., indicating that an object does not exist at a specific location). In the examples disclosed herein, the percentage of voxels that appear in the training data is switched (e.g., 10%, 20%, etc.). However, any other method for identifying voxels to be switched may be used in addition or alternatively. Furthermore, in some examples, instead of switching from an occupied value to an unoccupied value, the synthetic database builder 116 may switch the voxel from an unoccupied value to an occupied value (e.g., representing noise indicating that an object exists at a particular location when there is no such object at that location). In the examples disclosed herein, the synthetic database builder 116 randomly applies noise to the voxels in the training dataset. However, noise may be applied to the voxels in the training dataset in any other manner.

[0031] Figure 1 The example synthetic data storage device 124 of the illustrated example is implemented by any memory, storage device, and / or storage disk for storing data, such as flash memory, magnetic media, optical media, solid-state memory, (multiple) hard drives, (multiple) thumb drives, etc. In addition, the data stored in the example synthetic data storage device 124 can have any data format, such as binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, etc. Although in the illustrated example, the synthetic data storage device 124 is shown as a single device, the example synthetic data storage device 124 and / or any other data storage device described herein may be implemented by any number and / or type of memory. Figure 1 In the illustrated example of , the example synthetic data storage device 124 stores environment-specific machine learning models (e.g., synthetic databases) that can be used for scene reconstruction.

[0032] Figure 1The example scene reconstructor 130 of the illustrated example is implemented by a logic circuit such as a hardware processor. However, any other type of circuit may be used in addition or alternatively, for example, one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) ASICs, (multiple) PLDs, (multiple) FPLDs, (multiple) programmable controllers, (multiple) GPUs, (multiple) DSPs, etc. During the inference and / or use phase of the (multiple) trained models, the example scene reconstructor 130 uses the selected model to process the sensor data (e.g., sensor data received via the sensor data acquirer 104) to generate a reconstructed scene. In this way, a scene is created using a machine learning model adapted to the specific environment represented by the sensor data (e.g., to take into account errors expected to occur in the environment being reconstructed). Such adaptation ensures that noise specific to the type of environment is taken into account when analyzing the sensor data. The generated scene can then be used, for example, to facilitate navigation of a drone.

[0033] In operation, the example sensor data acquirer 104 acquires sensor data, e.g., image data, from the example sensor data source 103 and / or (multiple) example sensors 102. The acquired image data may be associated with a particular type of environment that affects the image acquisition device / hardware in different ways. For example, when collecting spatial data in an urban environment, a LiDAR system may exhibit specific errors. More specifically, a postmodern urban environment (e.g., San Francisco) may include office buildings and / or skyscrapers that utilize highly reflective materials (e.g., highly reflective glass, highly reflective metal, etc.), such highly reflective materials causing (multiple) LiDAR sensors to become saturated. However, due to, for example, relatively large degrees of optical abrupt building materials, the LiDAR system may not exhibit such errors to the same extent in traditional cities (e.g., Siena-Italy, Bath-England, etc.). Instead, the LiDAR system may exhibit different types of errors caused by shadows in such environments.

[0034] The example environment detector 106 identifies an environmental target associated with the acquired sensor data, and the example model selector 108 selects a synthetic database associated with the identified environmental target (from the example synthetic data storage device 124). The environmental target may include, but is not limited to, situations and nuances caused by postmodern urban images, traditional urban images, postmodern residential images, rural images, etc. In general, the selected synthetic database includes training data related to the acquired image data to improve image reconstruction efforts when the acquired image includes gaps and / or errors. The example model selector 108 selects a synthetic database and / or data set associated with the identified environmental target, and the example model builder 110 builds a CNN.

[0035] The example error characteristic evaluator 114 determines typical error characteristics associated with the environment in which the ground truth data was acquired. As described above, in the case where the ground truth data is acquired in a postmodern urban environment, the example error characteristic evaluator 114 identifies brightness saturation as a characteristic of interest to the LiDAR-based acquisition system. The example synthetic database builder 116 injects environment-specific errors into the ground truth data and develops an environment-specific model through one or more neural network training operations. In this way, the model can be applied to other environments of similar circumstances without having to reacquire ground truth data for an alternative environment (e.g., another postmodern city that exhibits building materials with a relatively high degree of light reflectivity). In some examples, the environment may correspond to the type of area in which the sensor platform (e.g., the device operating the (multiple) sensor 102) to host the sensor(s) 102 is expected to operate. For example, the environment may correspond to the type of road on which the autonomous vehicle operates (e.g., city streets, suburban streets, rural roads, highways, etc.).

[0036] In some examples, the scan reconstruction manager 101 is implemented at a server. In such examples, additional server resources can then be used to perform training of (multiple) machine learning models. In addition, the scan reconstruction manager 101 implemented at the server can then receive sensor data from a sensor platform (e.g., a device operating (multiple) sensors 102) and provide a reconstructed scene to the sensor platform. In some examples, the scan reconstruction manager 101 is implemented as a component of the sensor platform and / or communicates with the sensor platform. This approach enables a sensor platform (e.g., a mobile device, an unmanned aerial vehicle (UAV), a drone, an industrial device, an autonomous vehicle, etc.) to locally process sensor data and perform environment-specific scene reconstruction.

[0037] Although in Figure 1 and Figure 2 The implementation is shown in Figure 1 An example method of reconstructing system 100, but Figure 1 and Figure 2 One or more of the elements, processes, and / or devices shown may be combined, split, rearranged, omitted, eliminated, and / or implemented in any other manner. Figure 1The example sensor data acquirer 104, the example environment detector 106, the example model selector 108, the example model builder 110, the example model adjuster 112, the example property evaluator 114, the example synthetic database builder 116, the example synthetic data store 124, the example scene reconstructor 130 and / or more generally the example scan reconstruction manager 101 may be implemented by hardware, software, firmware and / or any combination of hardware, software and / or firmware. Thus, for example, Figure 1 Any of the example sensor data acquirer 104, the example environment detector 106, the example model selector 108, the example model builder 110, the example model adjuster 112, the example property evaluator 114, the example synthetic database builder 116, the example synthetic data storage device 124, the example scene reconstructor 130, and / or more generally the example scan reconstruction manager 101 may be implemented by one or more analog or digital circuits, logic circuits, (multiple) programmable processors, (multiple) programmable controllers, (multiple) graphics processing units (GPUs), (multiple) digital signal processors (DSPs), (multiple) application specific integrated circuits (ASICs), (multiple) programmable logic devices (PLDs), and / or (multiple) field programmable logic devices (FPLDs). When any of the apparatus or system claims of this patent is read to cover pure software and / or firmware implementations, Figure 1 At least one of the example sensor data acquirer 104, the example environment detector 106, the example model selector 108, the example model builder 110, the example model adjuster 112, the example property evaluator 114, the example synthetic database builder 116, the example synthetic data storage device 124, the example scene reconstructor 130, and / or more generally the example scan reconstruction manager 101 is hereby expressly defined to include a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including software and / or firmware. Further, Figure 1 The example reconstruction system 100 may include in addition to or in place of Figure 1 and / or Figure 2 One or more of the elements, processes and / or devices shown, and / or may include more than one of any or all of the elements, processes and devices shown. As used herein, the phrase "communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but additionally includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals and / or one-time events.

[0038] exist Figure 3 and / or Figure 4 A flow chart is shown in FIG. 1 , which represents a method for implementing Figure 1 and / or Figure 2 The example hardware logic, machine readable instructions, hardware implemented state machine and / or any combination thereof of the reconstruction system 100. The machine readable instructions may be an executable program or a portion of an executable program for execution by a computer processor, such as the following in combination with Figure 5 The processor 512 shown in the example processor platform 500 discussed above. The program may be embodied as software stored in a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 512, but the entire program and / or portions thereof may alternatively be executed by a device other than the processor 512 and / or embodied as firmware or dedicated hardware. In addition, although reference is made to Figure 3 and Figure 4 The flowchart shown describes an example procedure, but many other methods of implementing the example reconstruction system 100 may be used instead. For example, the order of execution of the blocks may be changed, and / or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform corresponding operations without executing software or firmware.

[0039] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a packaged format, and the like. The machine-readable instructions described herein may be stored as data (e.g., instructions, code, portions of code representations, and the like) that may be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, redistribution, and the like so that they may be directly read and / or executed by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple portions that are individually compressed, encrypted, and stored on separate computing devices, wherein the portions, when decrypted, decompressed, and combined, form a set of executable instructions that implement a program such as the program described herein. In another example, the machine-readable instructions may be stored in a state in which they can be read by a computer, but require the addition of a library (e.g., a dynamic link library (DLL), a software development kit (SDK), an application programming interface (API), etc.) in order to execute the instructions on a particular computing device or other device. In another example, the machine-readable instructions and / or corresponding program(s) may need to be configured (e.g., storing settings, inputting data, recording network addresses, etc.) before the machine-readable instructions and / or corresponding program(s) can be executed in whole or in part. Therefore, the disclosed machine-readable instructions and / or corresponding program(s) are intended to cover such machine-readable instructions and / or program(s) regardless of the particular format or state of the machine-readable instructions and / or program(s) when stored or otherwise at rest or in motion.

[0040] As mentioned above, Figure 3-4 The example processes of may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium (e.g., a hard drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk in which information is stored for any duration (e.g., for an extended period of time, permanently, for a brief moment, temporarily buffered and / or cached information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and / or storage disk, and to exclude propagating signals and to exclude transmission media.

[0041] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim adopts any form of "include" or "comprising" (e.g., comprising, including, including, including, having, etc.) as a preamble or in any kind of claim statement, it should be understood that additional elements, terms, etc. may be present without exceeding the scope of the corresponding claim or statement. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is open-ended in the same manner as the terms "include" and "comprising" are open-ended. When used, for example, in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, and C, such as (1) only A, (2) only B, (3) only C, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein in the context of describing structures, components, articles, objects, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, articles, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing the execution or performance of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or implementation of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an implementation that includes any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0042] Figure 3 The example process 300, when executed, results in the creation of a machine learning model corresponding to one or more specific environments. Figure 3The process 300 of the example sensor data acquirer 104 begins when acquiring ground truth data (e.g., images, videos, etc.). (Block 310). In the examples disclosed herein, the ground truth data represents high-quality images, videos, sensor data, etc. of a specific environment (e.g., a city). In some examples, the ground truth data is manually curated and / or edited to ensure that the ground truth data does not include noise. In the examples disclosed herein, the ground truth data is stored in the sensor data storage device 103. However, the ground truth data can be stored in any other location.

[0043] The example environment detector 106 identifies characteristics of the ground truth data. (Block 315). In examples disclosed herein, such characteristics include whether the environment is an urban environment (e.g., a cityscape) or a rural environment. In some examples, additional characteristics of the environment may be identified, such as the types of buildings and / or architecture that are typically present in the environment. For example, modern cityscapes / architecture (e.g., San Francisco, New York, etc.) may produce different models than other cityscapes / architecture (e.g., Siena, Italy; Dublin, Ireland). In some examples, metadata (e.g., Exchangeable Image File Format (EXIF) data) included in the sensor data may be used to determine the geographic location of the environment.

[0044] The example error characteristic evaluator 114 determines typical error characteristics associated with the environment of interest. (Block 320). For example, an urban environment (e.g., an environment with metal structures and / or glass windows) may encounter reflections in the sensor data. Such reflections may make themselves appear as unoccupied voxels (when the voxel should actually have an occupied value). The example model builder 110 divides the ground truth data into training sets. (Block 330). In the examples disclosed herein, three training sets are used. However, any other number of training sets may be used in addition or alternatively. In some examples, each training set includes the same data as other training sets. This approach allows various levels of noise to be applied to the training sets. However, in some other examples, a single training set is used, and progressive noise amounts are applied to the training sets. In the examples disclosed herein, the training sets have the same size. However, any size of training data may be used in addition or alternatively.

[0045] The example synthetic database builder 116 injects environment-specific error / noise into a first training set of ground truth data. (Block 340). To apply noise to the training set, one or more voxels are switched from an occupied value (e.g., indicating that an object exists at a particular location) to an unoccupied value (e.g., indicating that an object does not exist at a particular location). In the examples disclosed herein, the percentage of voxels that appear in the training data is switched (e.g., 10%, 20%, etc.), however, any other method for identifying voxels to be switched may be used in addition or alternatively. In addition, in some examples, instead of switching from an occupied value to an unoccupied value, a voxel may be switched from an unoccupied value to an occupied value (e.g., representing noise that indicates that an object exists at a particular location when there is no such object at that location). In the examples disclosed herein, noise is randomly applied to voxels in the training data set. However, noise may be applied to voxels in the training data set in any other manner.

[0046] The example model builder 110 trains the model using the training set. (Block 350). In the examples disclosed herein, the example model builder 110 uses stochastic gradient descent to train the model. However, any other training algorithm may be used in addition or alternatively. In the examples disclosed herein, training is performed until an acceptable error level between a validation data set (e.g., a portion of the ground truth data and / or training data to which noise is applied) and the ground truth data is reached.

[0047] The example model builder 110 then determines whether there are any additional training sets. (Block 360). If there are additional training sets, control returns to block 340, where the example process of blocks 340 to 360 is repeated until all training sets have been used. In the examples disclosed herein, variable noise amounts may be applied to various training sets (e.g., at block 340). For example, in a first iteration, a first noise amount may be applied to a first training set (e.g., 10% noise), and in a second iteration, a second noise amount different from the first noise amount may be applied to a second training set, e.g., 20% noise. In the examples disclosed herein, three training sets are used, with noise values ​​of 10%, 20%, and 50%, respectively. Using increasing amounts of noise improves the convergence of the model because the model gradually improves itself in scenarios where the noise gradually increases.

[0048] Once all training sets have been used to generate the model (e.g., block 360 returns a negative result), the example model adjuster 112 utilizes the residual OR gate connection to adapt and / or otherwise enhance the model. (Block 370). As noted above, the added OR gate improves system accuracy by eliminating false inferences. That is, the OR gate architecture ensures that the final result of the model (e.g., when used in conjunction with real-time data) does not result in the creation of erroneous voxel data. In other words, the system is guaranteed to perform no worse than the input data.

[0049] The example synthetic database builder 116 then stores the model in the synthetic data store 124. (Block 380). The example model is stored in conjunction with the environmental characteristics identified in conjunction with block 315. Storing the model in conjunction with the environmental characteristics (identified at block 315) then allows various models to be selected for use based on the environment. The example synthetic database builder 116 then determines whether additional locations of ground truth data exist (e.g., alternative environments). (Block 390). If additional locations of ground truth data were accessed (e.g., block 390 returns a "yes" result), then Figure 3 The example process of is repeated until such other locations of ground truth data have been processed (e.g., a model is created). Once additional locations of ground truth data have been processed (e.g., block 390 returns a "no" result), Figure 3 The example process terminates. It may be repeated, for example, when additional ground truth data becomes available, upon instruction (e.g., from a user, from an administrator, etc.) to perform retraining, etc. Figure 3 Example process.

[0050] Figure 4 The example process 400, when executed, results in using a machine learning model corresponding to an environment. Figure 4 The process 400 of FIG. 4 begins at block 410, where the example sensor data acquirer 104 acquires sensor data. (Block 410). In the examples disclosed herein, the sensor data represents real-time sensor data (e.g., from one or more sensors). In some examples, the sensor data is obtained from a drone and / or other unmanned aerial vehicle (UAV) navigating an environment. In some examples, the scan reconstruction manager 101 can be implemented locally on the drone so that the sensor data is processed on the drone. In some other examples, the sensor data can be sent to a remote location (e.g., a server) for processing.

[0051] The example environment detector 106 identifies environmental characteristics and / or nuances that may be associated with a particular environment. (Block 420). In examples disclosed herein, environmental characteristics represent information (e.g., metadata) about the environment to be reconstructed. In examples disclosed herein, such characteristics include whether the environment is an urban environment (e.g., a cityscape) or a rural environment. In some examples, additional characteristics of the environment may be identified, such as the types of buildings and / or architecture that are typically present in the environment. For example, modern urban landscapes / architecture (e.g., San Francisco, New York, etc.) may produce different models than other urban landscapes / architecture (e.g., Siena, Italy; Dublin, Ireland). In some examples, global positioning data (GPS) may be used to determine the location and / or environment to which the sensor data corresponds.

[0052] Using the identified (multiple) environmental characteristics, the example model selector 108 selects a synthetic database. (Block 430). The example scene reconstructor 130 then processes the sensor data using the selected model to generate a scene. (Block 440). In this way, a scene can be created using a machine learning model adapted to the specific environment represented by the sensor data. Such adaptation ensures that noise specific to that type of environment is taken into account when analyzing the sensor data. The generated scene can then be used, for example, to facilitate navigation of a drone.

[0053] Figure 5 is constructed to execute Figure 3 and / or Figure 4 The instructions to achieve Figure 1 and Figure 2 The processor platform 500 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet computer (e.g., an iPad TM )), personal digital assistant (PDA), Internet appliance, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, set-top box, headphones or other wearable device, or any other type of computing device.

[0054] The processor platform 500 of the illustrated example includes a processor 512. The processor 512 of the illustrated example is hardware. For example, the processor 512 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements the example scan reconstruction manager 101 and the components / structures contained therein.

[0055] The processor 512 of the illustrated example includes a local memory 513 (e.g., a cache). The processor 512 of the illustrated example communicates with a main memory including a volatile memory 514 and a non-volatile memory 516 via a bus 518. The volatile memory 514 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The main memory 514, 516 may be implemented by a memory controller.

[0056] The processor platform 500 of the illustrated example also includes an interface circuit 520. The interface circuit 520 may be implemented by any type of interface standard, for example, an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0057] In the example shown, one or more input devices 522 are connected to the interface circuit 520. The input device(s) 522 allow a user to enter data and / or commands into the processor 512. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, isopoint, and / or a voice recognition system.

[0058] One or more output devices 524 are also connected to the interface circuit 520 of the illustrated example. The output device 524 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switch (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 520 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0059] The interface circuitry 520 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate data exchange with external machines (e.g., any kind of computing device) via a network 526. Communications may occur via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, and the like.

[0060] The processor platform 500 of the illustrated example also includes one or more mass storage devices 528 for storing software and / or data. Examples of such mass storage devices 528 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0061] Figure 3 and Figure 4 The machine-executable instructions 532 may be stored in the mass storage device 528, in the volatile memory 514, in the non-volatile memory 516, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0062] From the foregoing, it will be appreciated that example systems, methods, apparatus, and articles of manufacture have been disclosed for improving three-dimensional scan reconstruction of image and / or video data by utilizing an "OR"-like structure in the convolution process of a neural network. In addition, the examples disclosed herein avoid the time-consuming, processor-intensive, and memory-intensive requirements for obtaining real data for areas of interest that exhibit similar environmental characteristics. For example, because San Francisco and Chicago exhibit similar environmental characteristics due to their postmodern urban features (e.g., skyscrapers with reflective surfaces), the ground truth data acquisition effort only needs to be performed once at one of these two locations. The resulting model can then be used for other similarly situated locations.

[0063] Note that this patent claims priority to U.S. Provisional Patent Application No. 62 / 674,462, filed on May 21, 2018, and the entire contents of which are hereby incorporated by reference.

[0064] Disclosed herein are methods, devices, systems, and articles of manufacture for reconstructing scenes using convolutional neural networks. Additional examples and combinations thereof include the following:

[0065] Example 1 includes a device for generating a model for scene reconstruction, the device comprising: a sensor data acquirer for acquiring ground truth data representing an environment; an environment detector for identifying environmental characteristics of the environment; a synthetic database builder for applying noise to the ground truth data to form a training set; a model builder for training a machine learning model using the training set and the ground truth data; and a model adjuster for modifying the machine learning model to include residual "OR" gate connections between corresponding layers of the machine learning model, wherein the synthetic database builder is also used to store the machine learning model in association with the environmental characteristics of the environment.

[0066] Example 2 includes the apparatus of Example 1, further comprising an environmental characteristic evaluator for determining an error characteristic associated with the environmental characteristic, the synthetic database builder for applying noise to the ground truth data based on the error characteristic.

[0067] Example 3 includes the apparatus of any one of Examples 1 or 2, wherein a synthetic database builder is used to divide ground truth data into multiple training sets and apply different amounts of noise to corresponding training sets in the multiple training sets, and a model builder is used to train a machine learning model based on the multiple training sets.

[0068] Example 4 includes the apparatus of Example 3, wherein the model builder is used to train the machine learning model using a training set with increasing amounts of noise.

[0069] Example 5 includes the apparatus of any of Examples 1-4, wherein the environmental characteristic comprises an amount of light reflectivity in the environment.

[0070] Example 6 includes the apparatus of any of Examples 1-5, wherein the machine learning model is implemented using a convolutional neural network (CNN).

[0071] Example 7 includes the apparatus of any one of Examples 1 to 6, wherein the environmental characteristic is a first environmental characteristic, the sensor data acquirer is used to access sensor data, the environmental detector is used to identify a second environmental characteristic of the environment represented by the sensor data, and the apparatus further includes: a model selector for selecting a machine learning model based on a second environmental characteristic that matches the first environmental characteristic; and a scene reconstructor for processing the sensor data using the selected machine learning model to create a scene.

[0072] Example 8 includes the apparatus of Example 7, wherein the environment detector is to identify the second environmental characteristic based on metadata accessed in conjunction with the sensor data.

[0073] Example 9 includes the apparatus of any one of Examples 1 to 8, further comprising a synthetic data storage device for storing the machine learning model.

[0074] Example 10 includes at least one non-transitory computer-readable medium comprising instructions that, when executed, cause at least one processor to perform at least the following operations: obtain ground truth data representing an environment; identify environmental characteristics of the environment; apply noise to the ground truth data to form a training set; train a machine learning model using the training set and the ground truth data; modify the machine learning model to include residual "OR" gate connections between corresponding layers of the machine learning model; and store the machine learning model in association with the environmental characteristics of the environment.

[0075] Example 11 includes at least one non-transitory computer-readable medium of Example 10, wherein the instructions, when executed, further cause at least one processor to determine an error characteristic associated with the environmental characteristic, and wherein the noise applied to the ground truth data is based on the error characteristic.

[0076] Example 12 includes at least one non-transitory computer-readable medium of any of Examples 10 or 11, wherein the instructions, when executed, further cause at least one processor to divide the ground truth data into multiple training sets, apply different amounts of noise to corresponding training sets in the multiple training sets, and train a machine learning model based on the multiple training sets.

[0077] Example 13 includes at least one non-transitory computer-readable medium of Example 12, wherein the instructions, when executed, further cause at least one processor to train the machine learning model using a training set with increasing amounts of noise.

[0078] Example 14 includes the at least one non-transitory computer-readable medium of any of Examples 10 to 14, wherein the environmental characteristic comprises an amount of light reflectivity in the environment.

[0079] Example 15 includes at least one non-transitory computer-readable medium of any one of Examples 10 to 14, wherein the machine learning model is implemented using a convolutional neural network (CNN).

[0080] Example 16 includes at least one non-transitory computer-readable medium of any one of Examples 10 to 15, wherein the environmental characteristic is a first environmental characteristic and the instructions, when executed, further cause at least one processor to: access sensor data; identify a second environmental characteristic of the environment represented by the sensor data; select a machine learning model based on the second environmental characteristic matching the first environmental characteristic; and process the sensor data using the selected machine learning model to recreate a scene.

[0081] Example 17 includes the at least one non-transitory computer readable medium of Example 16, wherein the instructions, when executed, further cause the at least one processor to identify a second environmental characteristic based on metadata accessed in conjunction with the sensor data.

[0082] Example 18 includes a method for generating a model for scene reconstruction, the method comprising: obtaining ground truth data representing an environment; identifying environmental characteristics of the environment; applying noise to the ground truth data to form a training set; training a machine learning model using the training set and the ground truth data; modifying the machine learning model to include residual "OR" gate connections between corresponding layers of the machine learning model; and storing the machine learning model in association with the environmental characteristics of the environment.

[0083] Example 19 includes the method of Example 18, further comprising determining an error characteristic associated with the environmental characteristic, and wherein the noise applied to the ground truth data is based on the error characteristic.

[0084] Example 20 includes the method of any one of Examples 18 to 20, further comprising dividing the ground truth data into multiple training sets, wherein applying noise to the training sets includes applying different amounts of noise to corresponding training sets, and training the machine learning model is performed based on the multiple training sets.

[0085] Example 21 includes the method of Example 20, wherein training the machine learning model is performed using a training set with increasing amounts of noise.

[0086] Example 22 includes the method of any of Examples 18 to 22, wherein the environmental characteristic includes an amount of light reflectivity in the environment.

[0087] Example 23 includes the method of any one of Examples 18 to 23, wherein the machine learning model is implemented using a convolutional neural network (CNN).

[0088] Example 24 includes the method of any one of Examples 18 to 23, wherein the environmental characteristic is a first environmental characteristic, and the method further includes: accessing sensor data; identifying a second environmental characteristic of the environment represented by the sensor data; selecting a machine learning model based on the second environmental characteristic matching the first environmental characteristic; and processing the sensor data using the selected machine learning model to recreate the scene.

[0089] Example 25 includes the method of Example 24, wherein the identification of the second environmental characteristic is based on metadata accessed in conjunction with the sensor data.

[0090] Example 26 includes a device for generating a model for scene reconstruction, the device including: a module for acquiring ground truth data representing an environment; a module for identifying environmental characteristics of the environment; a module for applying noise to the ground truth data to form a training set; a module for training a machine learning model using the training set and the ground truth data; and a module for modifying the machine learning model to include a residual "OR" gate connection between corresponding layers of the machine learning model, the module for applying is used to store the machine learning model in association with the environmental characteristics of the environment. In some examples, the module for acquiring can be implemented by the example sensor data acquirer 104. In some examples, the module for identifying can be implemented by the example environment detector 106. In some examples, the example module for applying can be implemented by the example synthetic database builder 116. In some examples, the example module for training can be implemented by the example model builder 110. In some examples, the example module for modifying can be implemented by the example model adjuster 112.

[0091] Example 27 includes the apparatus of Example 26, further comprising a module for determining a typical error characteristic associated with an environmental characteristic, and wherein the module for applying noise is used to apply noise to the ground truth data based on the error characteristic. In some examples, the module for determining can be implemented by the environmental characteristic evaluator 114.

[0092] Example 28 includes the apparatus of any of Examples 26 or 27, wherein the module for applying is used to divide the ground truth data into multiple training sets, the module for applying is used to apply different amounts of noise to the corresponding training sets, and the module for training is used to perform training based on the multiple training sets.

[0093] Example 29 includes the apparatus of Example 28, wherein the module for training is used to train the machine learning model using a training set to which increasing amounts of noise are applied.

[0094] Example 30 includes the apparatus of any of Examples 26 to 29, wherein the environmental characteristic comprises an amount of light reflectivity expected in the environment.

[0095] Example 31 includes the apparatus of any one of Examples 26 to 30, wherein the machine learning model is implemented using a convolutional neural network (CNN).

[0096] Example 32 includes the apparatus of any one of Examples 26 to 31, wherein the environmental characteristic is a first environmental characteristic, and the apparatus further comprises: a module for accessing sensor data, wherein the module for identifying is used to identify a second environmental characteristic of the environment represented by the sensor data; a module for selecting a machine learning model based on the second environmental characteristic matching the first environmental characteristic; and a module for processing the sensor data using the selected machine learning model to recreate a scene.

[0097] Example 33 includes the apparatus of Example 32, wherein the module for identifying is used to identify a second environmental characteristic based on metadata accessed in conjunction with sensor data. Although certain example methods, apparatus, and articles have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles that fall fully within the scope of the claims of this patent.

Claims

1. An apparatus for generating an environment-specific model for scene reconstruction, the apparatus comprising: a sensor data acquirer for acquiring ground truth data representing an environment; an environment detector for identifying environmental characteristics of the environment; an environmental characteristic evaluator for determining an error characteristic associated with the environmental characteristic; a synthetic database builder for applying noise to the ground truth data based on the error characteristics to form a training set; a model builder for training a machine learning model using the training set and the ground truth data; as well as a model adjuster for modifying the machine learning model to include residual OR gate connections between corresponding layers of the machine learning model, wherein the synthetic database builder is further for storing the machine learning model in association with environmental characteristics of the environment, The environmental characteristic is a first environmental characteristic, the sensor data acquirer is used to access sensor data, the environmental detector is used to identify a second environmental characteristic of the environment represented by the sensor data, and the apparatus further includes: a model selector for selecting the machine learning model based on the second environmental characteristic matching the first environmental characteristic; and A scene reconstructor for processing the sensor data using the selected machine learning model to create a scene.

2. The device according to claim 1, wherein: The synthetic database builder is used to divide the ground truth data into multiple training sets and apply different noise amounts to the corresponding training sets, and the model builder is used to train the machine learning model based on the multiple training sets.

3. The device according to claim 2, wherein: The model builder is used to train the machine learning model using a training set with increasing amounts of noise.

4. The device according to any one of claims 1 to 3, wherein: The environmental characteristic includes an amount of light reflectivity in the environment.

5. The device according to any one of claims 1 to 3, wherein: The machine learning model is implemented using a convolutional neural network (CNN).

6. The device according to any one of claims 1 to 3, wherein: The environment detector is for identifying the second environmental characteristic based on metadata accessed in conjunction with the sensor data.

7. The apparatus according to any one of claims 1 to 3 further comprises a synthetic data storage device for storing the machine learning model.

8. At least one non-transitory computer-readable medium comprising instructions that, when executed, cause at least one processor to perform at least the following operations: Obtaining ground truth data representing the environment; identifying environmental characteristics of the environment; determining an error characteristic associated with the environmental characteristic; applying noise to the ground truth data based on the error characteristic to form a training set; training a machine learning model using the training set and the ground truth data; modifying the machine learning model to include residual OR gate connections between corresponding layers of the machine learning model; as well as storing the machine learning model in association with an environmental characteristic of the environment, The environmental characteristic is a first environmental characteristic, and when the instruction is executed, the at least one processor further causes the at least one processor to perform the following operations: Access sensor data; identifying a second environmental characteristic of the environment represented by the sensor data; selecting the machine learning model based on the second environmental characteristic matching the first environmental characteristic; as well as The sensor data is processed using the selected machine learning model to recreate the scene.

9. The at least one non-transitory computer-readable medium of claim 8, wherein: When executed, the instructions also cause the at least one processor to divide the ground truth data into multiple training sets, apply different amounts of noise to corresponding training sets in the multiple training sets, and train the machine learning model based on the multiple training sets.

10. The at least one non-transitory computer-readable medium of claim 9, wherein: The instructions, when executed, also cause the at least one processor to train the machine learning model using a training set with increasing amounts of noise.

11. At least one non-transitory computer-readable medium according to any one of claims 8 to 10, wherein: The environmental characteristic includes an amount of light reflectivity in the environment.

12. At least one non-transitory computer-readable medium according to any one of claims 8 to 10, wherein: The machine learning model is implemented using a convolutional neural network (CNN).

13. At least one non-transitory computer-readable medium according to any one of claims 8 to 10, wherein: The instructions, when executed, further cause the at least one processor to identify the second environmental characteristic based on metadata accessed in conjunction with the sensor data.

14. A method of generating an environment-specific model for scene reconstruction, the method comprising: Obtaining ground truth data representing the environment; identifying environmental characteristics of the environment; determining an error characteristic associated with the environmental characteristic; applying noise to the ground truth data based on the error characteristic to form a training set; training a machine learning model using the training set and the ground truth data; modifying the machine learning model to include residual OR gate connections between corresponding layers of the machine learning model; as well as storing the machine learning model in association with an environmental characteristic of the environment, Wherein, the environmental characteristic is a first environmental characteristic, and the method further comprises: Access sensor data; identifying a second environmental characteristic of the environment represented by the sensor data; selecting the machine learning model based on the second environmental characteristic matching the first environmental characteristic; and The sensor data is processed using the selected machine learning model to recreate the scene.

15. The method of claim 14, further comprising dividing the ground truth data into a plurality of training sets, wherein: The applying of the noise to the training sets includes applying different amounts of noise to corresponding training sets, and the training of the machine learning model is performed based on the multiple training sets.

16. The method according to claim 15, wherein: The training of the machine learning model is performed using a training set with increasing amounts of noise.

17. The method according to any one of claims 14 to 16, wherein: The environmental characteristic includes an amount of light reflectivity in the environment.

18. The method according to any one of claims 14 to 16, wherein: The machine learning model is implemented using a convolutional neural network (CNN).

19. An apparatus for generating an environment-specific model for scene reconstruction, the apparatus comprising: A module for acquiring ground truth data representing the environment; means for identifying environmental characteristics of said environment; means for determining an error characteristic associated with said environmental characteristic; means for applying noise to the ground truth data based on the error characteristics to form a training set; means for training a machine learning model using the training set and the ground truth data; as well as means for modifying the machine learning model to include residual OR gate connections between corresponding layers of the machine learning model, means for applying to store the machine learning model in association with environmental characteristics of the environment, Wherein, the environmental characteristic is a first environmental characteristic, and the apparatus further comprises: Modules for accessing sensor data; means for identifying a second environmental characteristic of the environment represented by the sensor data; means for selecting the machine learning model based on the second environmental characteristic matching the first environmental characteristic; and Module for processing the sensor data using a selected machine learning model to recreate a scene.

20. A computing device comprising: A memory for storing instructions; as well as A processor coupled to the memory, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 14-18.

21. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 14-18.

Images