Method for simulating images with aberrations based on ray tracing

Abstract

A method for simulating an image having aberrations, or a non-transitory computer-readable storage medium having instructions recorded thereon. The method includes ray tracing a plane of an object to produce a planar image, estimating a point spread function from ray tracing, and creating one or more perceived images. The method may include feeding one or more synthetic pulse images or scenes to the ray tracing simulator. The method may include feeding the perceived image into a perceived pipeline, and generating a composite image from the planar image and the perceived image.

Description

Technical Field

[0001] The present disclosure relates to a method for simulating an image with aberration based on ray tracing. This can be used for autonomous vehicles or other neural networks that require synthetic images to train vision. Currently, there is no technology to assist in simulating images with aberration based on ray tracing. Summary of the Invention

[0002] A method for simulating an image with aberration, or a non-transitory computer-readable storage medium having instructions recorded thereon. The method includes ray tracing a plane of an object to generate a planar image, estimating a point spread function from the ray tracing, and creating one or more perceptual images. The method may include feeding the perceptual images into a perception pipeline, and generating synthetic imagery from the planar image and the perceptual images.

[0003] The method may include implementing a blur effect and implementing a noise tradeoff. The method may further include providing an actual imaging sensor of a vehicle, and the imaging sensor of the vehicle is positioned within a cabin of the vehicle. The method may further include incorporating optical effects into the generated synthetic imagery, and due to the generated synthetic imagery, test-driving may not be required for data collection, and presenting one or more photo-realistic scenes on the generated synthetic imagery. The method may include feeding one or more synthetic pulse images or scenes into a ray tracing simulator.

[0004] The present disclosure provides the following embodiments.

[0005] 1. A method for simulating an image with aberration, comprising: Ray tracing a plane of an object to generate a planar image; Estimating a point spread function from the ray tracing; Creating one or more perceptual images; Feeding the perceptual images into a perception pipeline; and Generating synthetic imagery from the planar image and the perceptual images.

[0006] 2. The method according to embodiment 1, further comprising: Feeding one or more synthetic pulse images or scenes into a ray tracing simulator.

[0007] 3. The method according to embodiment 2, further comprising: Implementing a blur effect; and Implementing a noise tradeoff.

[0008] 4. The method according to embodiment 3 further includes: Providing an actual imaging sensor of the vehicle.

[0009] 5. The method according to embodiment 4, wherein the imaging sensor of the vehicle is positioned inside the cockpit of the vehicle.

[0010] 6. The method according to embodiment 5 further includes: Incorporating optical effects into the generated synthetic image.

[0011] 7. The method according to embodiment 6, wherein due to the generated synthetic image, test drives are not required for data collection.

[0012] 8. The method according to embodiment 7 further includes: Presenting one or more photo-realistic scenes on the generated synthetic image.

[0013] 9. The method according to embodiment 8 further includes: Generating a point spread function sampled at multiple positions in one or more frames.

[0014] 10. The method according to embodiment 1 further includes: Providing an actual imaging sensor of the vehicle, wherein the imaging sensor of the vehicle is positioned inside the cockpit of the vehicle.

[0015] 11. The method according to embodiment 1 further includes: Incorporating optical effects into the generated synthetic image, wherein due to the generated synthetic image, test drives are not required for data collection.

[0016] 12. The method according to embodiment 11 further includes: Presenting one or more photo-realistic scenes on the generated synthetic image.

[0017] 13. A non-transitory computer-readable storage medium having instructions recorded thereon, wherein execution of the instructions by a processor causes the processor to: Ray trace a plane of an object; Estimate a point spread function from the ray tracing; Create one or more perceptual images; Implement a blur effect; Implement a noise trade-off; Feed the perceptual image into a perceptual pipeline; Generate a synthetic image from the perceptual image; Incorporate optical effects into the generated synthetic image; Render one or more photo-realistic scenes on the generated synthetic image, wherein the imaging sensor is positioned within the cockpit of the vehicle; and Feed one or more synthetic pulse images or scenes to a ray tracing simulator.

[0018] 14. The non-transitory computer-readable storage medium having instructions recorded thereon according to embodiment 13, further comprising: Generate a point spread function sampled at multiple locations in one or more frames.

[0019] 15. A method for simulating an image with aberration, comprising: Ray trace a plane of an object to produce a planar image; Feed one or more synthetic pulse images or scenes to a ray tracing simulator; Estimate a point spread function from the ray tracing; Create one or more perceptual images; Feed the perceptual image into a perceptual pipeline; Generate a synthetic image from the planar image and the perceptual image; Provide an actual imaging sensor of the vehicle; and Incorporate optical effects into the generated synthetic image.

[0020] 16. The method according to embodiment 15, wherein the imaging sensor of the vehicle is positioned within the cockpit of the vehicle.

[0021] 17. The method according to embodiment 16, wherein due to the generated synthetic image, test drives are not required for data collection.

[0022] 18. The method according to embodiment 17, further comprising: Render one or more photo-realistic scenes on the generated synthetic image.

[0023] 19. The method according to embodiment 18, further comprising: Implement a blur effect; and Implement a noise trade-off.

[0024] When considered in conjunction with the accompanying drawings, the above and other features and advantages of the present disclosure become apparent from the following detailed description of the best mode for carrying out the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of a connectivity system for simulating an image with aberration based on ray tracing.

[0026] Figure 2Schematic flowcharts of one or more methods for simulating an image with aberration based on ray tracing. Detailed implementation

[0027] Referring to the accompanying drawings, where possible, the same reference numerals refer to similar components. Figure 1 Schematically illustrates a connectivity network or connectivity system 10. The connectivity system 10 includes many components, only some of which are listed and / or illustrated herein.

[0028] A remote or cellular communication system or cellular network 12, which may represent many types of communication protocols, including but not limited to: cellular, satellite, Wi-Fi, Bluetooth, ultra-wideband (UWB), or other communications recognizable to those skilled in the art. UWB is a radio-based communication technology for short-range use and fast and stable data transmission.

[0029] A centralized location 14 is highly schematically shown, but may represent many different structures, clouds, servers, or elements, as will be recognized by those skilled in the art. The centralized location 14 represents a system that communicates with some or all of the other systems and / or objects described herein. The centralized location 14 includes many controllers 20. Additionally, the centralized location 14 may be the back office (BO) of a vehicle manufacturer.

[0030] Schematically illustrates several transmission protocols or transmissions 16. These transmissions 16 may include but are not limited to: cellular, Wi-Fi, wired networks, over-the-air (OTA), other transmission protocols, including machine-to-machine (M2M) or other telematics devices, or other systems recognizable to those skilled in the art. M2M systems use peer-to-peer communication between machines, sensors, and hardware via cellular, Wi-Fi, or wired networks.

[0031] The drawings and figures presented herein are diagrams, not drawn to scale, and are provided for descriptive purposes only. Therefore, any specific or relative dimensions or alignments shown in the drawings should not be construed as limiting. Although the present disclosure may be illustrated with respect to a particular application or industry, those skilled in the art will recognize the broader applicability of the present disclosure. Those skilled in the art will recognize that terms such as "above," "below," "upward," "downward," etc. are used to describe the figures and do not represent a limitation on the scope of the present disclosure as defined by the appended claims. Any numerical designations (such as "first" or "second") are merely illustrative and are not intended to limit the scope of the present disclosure in any way.

[0032] The features shown in one figure may be combined with, substituted for, or modified by the features shown in any figure. Unless otherwise stated, no feature, element, or limitation is mutually exclusive of any other feature, element, or limitation. Additionally, there is no feature, element, or limitation that is absolutely required for an operation. Any particular configuration shown in the figures is illustrative only, and the particular configurations shown do not limit the claims or the description.

[0033] The term "vehicle" is broadly applicable to any mobile platform. Vehicles into which the present disclosure may be incorporated include, for example but not limited to: passenger or freight vehicles; autonomous vehicles; industrial, construction, and mining equipment; and various types of aircraft.

[0034] All numerical values of parameters (e.g., quantities or conditions) in this specification (including the appended claims) should be understood to be modified in all instances by the term "about", whether or not the term actually appears before the value. "About" indicates that the stated value allows some slight imprecision (having some approximation to exactness in the value; about or reasonably close to the value; nearly). If the imprecision provided by "about" cannot otherwise be understood in the ordinary sense in this field, then "about" as used herein at least indicates variations that may result from the ordinary methods of measuring and using such a parameter. Additionally, the disclosure of a range includes the disclosure of all values within the entire range and further divided ranges. Each value and range endpoint within the range is hereby disclosed as a separate embodiment.

[0035] When used herein, the term "substantially" generally refers to the following relationship: the relationship is ideally perfect or complete, but manufacturing realities prevent absolute perfection. Thus, "substantially" indicates a typical deviation from perfection. For example, if height A is substantially equal to height B, it may be preferred that the two heights are 100.0% equal, but manufacturing realities may result in variations from such perfection. Those skilled in the art will recognize the acceptable amount of deviation. For example but not limited to, coverage, area, or distance may be substantially within 10% of perfection for substantial equivalence. Similarly, relative alignment (such as parallel or perpendicular) is generally considered to be within 5%.

[0036] A general control system, computing system, or controller 20 communicates operably with the relevant components of all systems and is recognizable to those skilled in the art. Controller 20 includes, for example but not limited to, non - general - purpose electronic control devices having a pre - programmed digital computer or processor, a memory for storing data (such as control logic, instructions, look - up tables, etc.), a storage device or non - transitory computer - readable storage medium, and a plurality of input / output peripheral devices, ports, or communication protocols.

[0037] In addition, the controller 20 may include or communicate with a plurality of sensors. The controller 20 is configured to execute or implement all of the control logic or instructions described herein and may communicate with any sensors described herein or recognized by those skilled in the art. Any of the methods described herein may be executed by one or more controllers 20.

[0038] The connectivity system 10 may be used to perform methods for simulating aberration-based images based on ray tracing. In 3D or 2D computer graphics, ray tracing is a technique used to model light transport and is used in various rendering algorithms for generating digital images and photo-realistic scenes. Ray tracing can simulate various optical effects such as reflection, refraction, soft shadows, scattering, depth of field, motion blur, caustics, ambient occlusion, and dispersion phenomena such as chromatic aberration.

[0039] This technology is used to generate one or more base images for training a neural network (NN), saving the long process of collecting real data. Instead of actual images captured by the system and having to label the data to train the neural network, these systems / methods / techniques avoid the need for test drives of one or more mobile vehicles 22. By using synthetic images that have been modified by optical aberration information (i.e., point spread function (PSF)), the system can generate more realistic images, with the added benefit that since it is synthetic, no further annotation effort is required. By the nature of how the synthetic data is generated, the synthetic data contains all the information, i.e., the label or identification of the object and its position automatically in 3D space. This information is then used to train the neural network.

[0040] Ray tracing can also be used to trace the path of sound waves in a manner similar to light waves, making it a viable option for more immersive sound design in video games by presenting realistic reverberation and echo. In fact, any physical wave or particle phenomenon with approximately linear motion can be simulated with ray tracing. Aberration can be a deviation from the normal, usual, or expected situation, which is typically an undesirable deviation.

[0041] Figure 2is a schematic flowchart of method 100 or more methods for simulating an image with aberration based on ray tracing. One or more of the methods described herein may be performed by a controller 20, which includes a non-transitory computer-readable storage medium, or other structures or devices recognizable to those skilled in the art. All steps described herein are also optional, except for those steps explicitly stated as such, and all steps described may be reordered or removed. Any method described herein may store data in a centralized location 14 via a connectivity system 10.

[0042] Figure 1 A vehicle 22 is shown, but there may be other vehicles 22 not shown. Figure 1 An imaging sensor 24 is shown highly schematically, and it should be noted that the imaging sensor 24 may be located at any position of the vehicle 22. The imaging sensor 24 may be of any of the types listed, or as will be recognized by those skilled in the art, including but not limited to: an image sensor, a radar sensor, a lidar sensor, a GPS sensor, and / or an ultrasonic sensor. The imaging sensor 24 may include embedded MEMS technology or other technologies recognizable to those skilled in the art.

[0043] Figure 1 Sensor capture lines 26 are shown schematically. These may be used to capture a scene 28. It should be noted that the scene 28 may consist of many elements (both real and modified elements), as will be recognized by those skilled in the art. It should be noted that the scene 28 may include one or more humans 30.

[0044] Step 110: Start. At step 110, method 100 initializes or starts. Method 100 may start operating when called by one or more controllers 20, may run continuously, or may iterate in a loop.

[0045] Step 112: Feed some synthetic pulse images / scenes to a ray tracing simulator. At step 112, method 100 feeds some synthetic pulse images / scenes to a ray tracing simulator.

[0046] Those skilled in the art will recognize how to feed one or more synthetic images to a ray tracing simulator. This may include but is not limited to generating one or more planar images or source images, which may be used to generate one or more generated synthetic images. This includes but is not limited to generating synthetic pulse scene data that generally assumes perfect optical performance, and transforming a synthetic scene that mimics the real world where three-dimensional (3D) objects are randomly distributed in 3D space so as to be mapped to a two-dimensional (2D) image. Planar images may be stacked to create a three-dimensional space.

[0047] Step 114: Estimate the point spread function via ray tracing of the planar image. At step 114, method 100 estimates the point spread function (PSF) via ray tracing of the planar image. A PSF is generated, which can be sampled at multiple positions in the frame. Note that this process is repeated for each color component (R, G, B) and each depth - i.e., assuming the synthetic impulse images are placed at different distances from the imager. For example but not limited to, the synthetic impulse images can be between 10 meters and 100 meters in 10 - meter steps.

[0048] Ray tracing was described in detail above. The PSF describes the response of a focused optical imaging system to a point source or point object. A more general term for the PSF is the impulse response of the system, and the PSF is the impulse response or impulse response function (IRF) of a focused optical imaging system.

[0049] This includes but is not limited to generating 2D PSF maps for each depth on a 3D volume. These can be 2D slices XY taken at each Z - depth, which create 2D maps of all the optical aberrations generated by the optics. This is a PSF library of aberrations. Steps 112 and 114 can be combined by their mutual Z - positions to blend the associated XY aberrations with the mutual XY images.

[0050] Step 116: Generate a synthetic 3D image from the data set. At step 116, method 100 generates a synthetic 3D rendered image for the data set. Preferably, these can be photo - realistic images. Additionally, each image in the data set is associated with a per - pixel depth map. This can further include creating one or more perceptual images.

[0051] Step 118: Filter the images in the data set using a collection of PSF filters. At step 118, method 100 filters the images in the data set. First, the imaging sensor 24 or the actual imaging sensor 24 will need to know where the object is located relative to themselves. Each group of pixels is filtered using the PSF filters according to its depth extracted from the depth map. The filtered pixels (each corresponding to a different spatial position within the frame) are combined back into a unified image.

[0052] The PSF can be adjusted to register the picked - up object. A predictable perception pipeline for an autonomous vehicle system can be developed and can include but is not limited to predicting temporal changes based on intermediate results such as proposals and raw points. Additional elements of the perception pipeline will be recognized by those skilled in the art.

[0053] Optional step 122: Implement a blurring effect. At optional step 122, method 100 may implement a blurring effect. Blurring will be understood by those skilled in the art. Simply, blurring may be one aberration among many other aberrations - focusing or lack of focusing, which aberrations include but are not limited to: spherical aberration, coma aberration, astigmatic aberration, tilt aberration, chromatic aberration (longitudinal and lateral aberration), and irregularity aberration.

[0054] Optional step 124: Implement a noise effect. At optional step 124, method 100 may implement a noise trade-off or effect. Generally, as those skilled in the art will recognize, noise trade-off refers to increasing the information rate, signal-to-noise ratio, and the allocated bandwidth, which may be traded off against each other.

[0055] Step 126: Filter the data set and feed it into the perception pipeline. At step 126, method 100 filters the data set and feeds it into the perception pipeline. Note that this may include but is not limited to generating synthetic imagery from both planar images and perception images. Those skilled in the art will recognize different filters that may be applied to one or more data sets.

[0056] Step 128: Incorporate optical effects accurately into the generated synthetic imagery. At step 128, method 100 incorporates one or more optical effects into the generated synthetic imagery. Those skilled in the art will recognize what "accurately" may mean.

[0057] Optional step 132: The imaging sensor is positioned inside the cockpit of the vehicle. At optional step 132, method 100 may position the imaging sensor inside the cockpit of vehicle 22. Note that those skilled in the art will recognize other locations for imaging sensor 24, including but not limited to outside vehicle 22 and / or in front of or behind vehicle 22.

[0058] Step 134: Render one or more photo-realistic scenes on the generated synthetic imagery. At step 134, method 100 renders photo-realistic scenes on or within the generated synthetic imagery. Note that importantly, due to the generated synthetic imagery and / or the photo-realistic scenes, a test drive of vehicle 22 may not be required for data collection. Those skilled in the art will recognize why a test drive is not required for data collection.

[0059] Step 140: End / Loop. At step 140, method 100 ends or loops. Ending / looping may include traveling back to start step 110 or waiting until being called to run again, such as being called by one of controllers 20 or another part of connectivity system 10.

[0060] The detailed description and the drawings support and describe the subject matter herein. While some best modes and other embodiments have been described in detail, there are various alternative designs, embodiments, and configurations.

[0061] In addition, the features of any of the examples shown in the drawings or of the various examples mentioned in this description are not necessarily to be understood as independent of each other. On the contrary, it is possible that each feature described in one of the examples of an embodiment can be combined with one or more other desired features from other examples, resulting in other examples not described in words or with reference to the drawings. Thus, such other examples fall within the framework of the scope of the appended claims.

Claims

1. A method for simulating an image with aberration, comprising: Ray tracing a plane of an object to generate a planar image; Estimating a point spread function from the ray tracing; Creating one or more perceptual images; Feeding the perceptual image into a perceptual pipeline; And Generating a synthetic image from the planar image and the perceptual image.

2. The method according to claim 1, further comprising: Feeding one or more synthetic pulse images or scenes into a ray tracing simulator.

3. The method according to claim 2, further comprising: Implementing a blur effect; And Implementing a noise trade-off.

4. The method according to claim 3, further comprising: Providing an actual imaging sensor of a vehicle.

5. The method according to claim 4, wherein the imaging sensor of the vehicle is positioned inside the cockpit of the vehicle.

6. The method according to claim 5, further comprising: Incorporating an optical effect into the generated synthetic image.

7. The method according to claim 6, wherein due to the generated synthetic image, a test drive is not required for data collection.

8. The method according to claim 7, further comprising: Rendering one or more photo-realistic scenes on the generated synthetic image.

9. A non-transitory computer-readable storage medium having instructions recorded thereon, wherein the execution of the instructions by a processor causes the processor to: Ray trace a plane of an object; Estimate a point spread function from the ray tracing; Create one or more perceptual images; Implement a blur effect; Implement a noise trade-off; Feed the perceptual image into a perceptual pipeline; Generate a synthetic image from the perceptual image; Incorporate an optical effect into the generated synthetic image; Render one or more photo-realistic scenes on the generated synthetic image, wherein the imaging sensor is positioned inside the cockpit of the vehicle; and Feed one or more synthetic pulse images or scenes into a ray tracing simulator.

10. The non-transitory computer-readable storage medium having instructions recorded thereon according to claim 9, further comprising: Generating a point spread function sampled at multiple positions in one or more frames.

Images