Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that make it possible to improve the accuracy of a perception model for executing a simulation for a sensor that perceives an object. The information processing device comprises the perception model that performs a simulation for the sensor which perceives an object by using an electromagnetic wave, and that generates output data indicating the perception result of the object, on the basis of input data including at least one of spectral information and a propagation direction of the electromagnetic wave. For example, the present technology can be applied to an automatic driving simulation for executing a simulation for an automatic driving system.

Description

Technical Field

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and particularly relates to an information processing apparatus, an information processing method, and a program suitable for use when performing simulation of a sensor that perceives an object.

Background Art

[0002] Conventionally, in order to evaluate and verify the safety and the like of an automatic driving system that realizes automatic driving, in addition to a running test (hereinafter referred to as a real running test) that is actually performed by running a vehicle, a running test (hereinafter referred to as a virtual running test) that is virtually performed on a system that simulates the automatic driving system has been conducted (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, the automatic driving system executes processes including, for example, a perception step, a recognition step, a determination step, and an operation step. The perception step is, for example, a step of perceiving the situation around the vehicle. The recognition step is, for example, a step of specifically recognizing the situation around the vehicle. The determination step is, for example, a step of performing various determinations based on the result of recognizing the situation around the vehicle. The operation step is, for example, a step of performing automatic operation of the vehicle based on various determinations. Note that the perception step and the recognition step may be combined into one as a cognitive step in some cases.

[0005] In contrast, conventional virtual driving tests use images taken from the real world or images created using computer graphics (CG) (hereinafter referred to as CG images). With CG images, for example, the size and shape of objects can be reproduced, but it is difficult to reproduce blur, out-of-focus areas, lighting conditions, etc. Therefore, while it is possible to test the operation steps in virtual driving tests, it is difficult to test the perception, recognition, and judgment steps until the sensors to be used are actually installed in a vehicle and real-world driving tests are conducted.

[0006] Therefore, there is a need for the practical application of perceptual models that can generate images that reproduce, for example, lighting conditions, blur, and out-of-focus effects.

[0007] This technology was developed in light of these circumstances and aims to improve the accuracy of perceptual models that simulate the perception of objects. [Means for solving the problem]

[0008] The first aspect of this technology is an information processing device that simulates an imager and generates an imager model based on incident light data that includes at least one of the incident direction and spectral information of the incident light for each pixel. The system includes a perceptual model that generates perceptual data including the image data, and a recognition model that simulates the process of recognizing the surrounding situation based on the perceptual data. It is equipped with.

[0009] The first aspect of this technology involves an information processing method that simulates an imager and processes image data based on incident light data that includes at least one of the incident direction and spectral information of the incident light for each pixel. This includes generating perceptual data and simulating the process of recognizing the surrounding environment based on the perceptual data.

[0010] The first aspect of this technology involves a program that simulates an imager and, based on incident light data including at least one of the incident light direction and spectral information for each pixel, generates image data. This includes generating perceptual data and simulating the process of recognizing the surrounding environment based on the perceptual data. Have the computer perform the process. The second aspect of this technology, the information processing device, simulates a sensor that perceives objects using electromagnetic waves and displays the result of object perception based on input data including the intensity for each direction of electromagnetic wave propagation. perception Perceptual models that generate data And, based on the aforementioned perceptual data, a recognition model that simulates the process of recognizing the surrounding situation. It is equipped with. The second aspect of this technology involves an information processing method that simulates a sensor that perceives objects using electromagnetic waves, and displays the result of object perception based on input data including the intensity for each direction of electromagnetic wave propagation. perception Generate data This includes simulating the process of recognizing the surrounding situation based on the aforementioned perceptual data. The second aspect of this technology involves a program that simulates a sensor that perceives objects using electromagnetic waves, and displays the results of object perception based on input data including the intensity of electromagnetic waves in each propagation direction. perception Generate data This includes simulating the process of recognizing the surrounding environment based on the aforementioned perceptual data. Have the computer perform the process.

[0011] In the first aspect of this technology, an imager simulation is performed, and based on incident light data including at least one of the incident light direction and spectral information for each pixel, image data is generated. Perceptual data including this is generated, and a process of recognizing the surrounding situation is simulated based on the perceptual data. In the second aspect of this technology, a simulation of a sensor that perceives objects using electromagnetic waves is performed, and output data showing the result of object perception is generated based on input data including the intensity for each direction of electromagnetic wave propagation. Based on the aforementioned perceptual data, a process for recognizing the surrounding environment is simulated. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of motion blur. [Figure 2] This diagram shows an example of overexposure. [Figure 3] This figure shows an example of a sub-pixel structure. [Figure 4] This is a diagram illustrating the principles behind the generation of flares and ghosts. [Figure 5] This figure shows examples of flare and ghosting. [Figure 6]It is a timing chart of an imager of the rolling shutter method. [Figure 7] It is a diagram showing an example of rolling shutter distortion. [Figure 8] It is a diagram showing an example of flicker. [Figure 9] It is a graph showing an example of the spectral sensitivity characteristics of each pixel of the imager. [Figure 10] It is a graph showing an example of the spectral distribution of an incandescent lamp. [Figure 11] It is a graph showing an example of the spectral distribution of a low-pressure sodium lamp. [Figure 12] It is a block diagram showing a first embodiment of an automatic driving simulator. [Figure 13] It is a diagram showing an example of the arrangement of pixels of an imager model. [Figure 14] It is a table showing the correspondence between the real space, the automatic driving system, and each model of the imaging model. [Figure 15] It is a diagram showing an example of the signal processing of an imager simulated in a silver model. [Figure 16] It is a diagram showing an example of the signal processing of an imager simulated in a gold model. [Figure 17] [[ID=३३]]It is a diagram showing an example of the input / output data of an imager model. [Figure 18] It is a diagram showing an example of the data structure regarding the intensity of incident light. [Figure 19] It is a diagram showing an example of the data structure regarding the incident direction of incident light. [Figure 20] It is a diagram showing an example of the data structure regarding exposure characteristics. [[ID=४४]] [Figure 21] It is a diagram showing an example of the structure of a buffer between a rendering model and an imager model. [Figure 22] It is a block diagram showing a second embodiment of an automatic driving simulator. [Figure 23] It is a block diagram showing a third embodiment of an automatic driving simulator. [Figure 24]This is a block diagram showing an example of a computer configuration. [Modes for carrying out the invention]

[0013] The following describes the configurations for implementing this technology. The explanation will proceed in the following order. 1. Background of this technology 2. Embodiments 3. Variant 4. Others

[0014] <<1. Background of this technology>> First, the background of this technology will be explained with reference to Figures 1 to 11.

[0015] Conventional imager models used in virtual driving tests have difficulty reproducing the following phenomena that occur in images captured by real imagers. Note that an imager model is a type of perceptual model, a model that performs simulations of an imager.

[0016] Motion Blur A vehicle is moving at high speed, and there are other fast-moving objects, such as other vehicles, in its surroundings. When the relative speed between the vehicle and the surrounding objects is large, motion blur may occur in images taken from the vehicle, such as the vehicle in the leftmost frame of Figure 1.

[0017] Conventional imager models have difficulty reproducing this type of motion blur.

[0018] <Overexposure and underexposure> For example, when photographing extremely bright scenes, such as those illuminated by direct sunlight, overexposure may occur, where part of the image becomes completely white. For example, when photographing extremely dark scenes, such as those in the suburbs at night, underexposure may occur, where part of the image becomes completely black. For example, when photographing scenes with a large difference in brightness, overexposure or underexposure may occur in parts of the image, such as near the tunnel exit in Figure 2.

[0019] Conventional imager models have difficulty reproducing such overexposure and underexposure.

[0020] Furthermore, to prevent overexposure and underexposure, HDR (High Dynamic Range) compositing, for example, is used to expand the dynamic range of the captured image. Here, we will explain a specific example of HDR compositing.

[0021] For example, in HDR compositing, the dynamic range of the captured image is expanded by combining an image taken with a long exposure time with an image taken with a short exposure time.

[0022] For example, as shown in Figure 3, HDR compositing is performed by using an imager with a sub-pixel structure. An imager with a sub-pixel structure has a main pixel with a large light-receiving surface and low sensitivity, and a sub-pixel with a small light-receiving surface and high sensitivity. The dynamic range of the captured image is expanded by combining the low-sensitivity image captured using the main pixel with the high-sensitivity image captured using the sub-pixel.

[0023] In this example, the main pixels consist of Gr1, R1, B1, and Gb1 pixels. The sub-pixels consist of Gr2, R2, B2, and Gb2 pixels. Gr1, Gr2, Gb1, and Gb2 pixels are pixels with a green filter. R1 and R2 pixels are pixels with a red filter. B1 and B2 pixels are pixels with a blue filter.

[0024] Conventional imager models have difficulty reproducing captured images generated by this type of HDR compositing process.

[0025] <Flare, Ghost> For example, as shown in Figure 4, when strong light from an oblique direction enters the imager 21, flare and ghosting may occur in the captured image, as shown in Figure 5.

[0026] Conventional imager models have difficulty reproducing such flares and ghosting.

[0027] <Rolling shutter distortion> Figure 6 is a timing chart showing the transitions between the exposure and readout periods for each pixel row in a rolling shutter imager. The horizontal axis represents time, and the vertical axis represents the pixel row (Line). In each pixel row, the periods indicated by the white-filled frame represent the exposure period, and the periods indicated by the black-filled frame represent the readout period.

[0028] As shown in this example, in a rolling shutter imager, exposure begins sequentially from the first row of pixels at predetermined time intervals ΔT, and exposure continues for a predetermined period Td. After the exposure period ends for each pixel row, the charge accumulated in each pixel is read out during a predetermined readout period Tr.

[0029] Therefore, a shift in exposure time occurs for each pixel row. As a result, for example, when the relative speed between the vehicle and surrounding objects is high, rolling shutter distortion can occur, causing the shape of objects to become distorted, as seen in the headlights of the vehicle within the frame in Figure 7.

[0030] Conventional imager models have difficulty reproducing this type of rolling shutter distortion.

[0031] In a typical rolling shutter system, exposure starts sequentially for each row of pixels, but it is also possible to configure it so that exposure starts every n rows of pixels (2 or more).

[0032] <Flicka> For example, when photographing a light source with a fast blinking speed (such as an LED), or when photographing a scene illuminated by light from a light source with a fast blinking speed, flicker may occur in the captured image.

[0033] For example, when photographing a traffic light using LEDs, flicker occurs because each LED blinks at a different time, resulting in variations in the brightness of the traffic light's lamps, or even partial blind spots, as shown within the frame in Figure 8. For example, even if the LEDs blink at the same time, flicker may occur when photographing a traffic light using LEDs with the rolling shutter imager described above, because the LED blinking speed is fast.

[0034] Furthermore, flicker can occur, for example, when using a rolling shutter imager to photograph a scene illuminated by a rapidly flashing light source. In other words, there may be a mixture of pixel rows exposed under light conditions and pixel rows exposed without light conditions, resulting in variations in the brightness of the captured image.

[0035] Conventional imager models have difficulty reproducing this type of flicker.

[0036] <Light source with a special spectrum> Figure 9 shows an example of the spectral sensitivity characteristics of each pixel in an imager. Specifically, it shows the spectral sensitivity characteristics of an R pixel with a red filter, a G pixel with a green filter, a B pixel with a blue filter, and a W (White) pixel without a color filter. The horizontal axis represents wavelength, and the vertical axis represents relative sensitivity.

[0037] Figure 10 shows an example of the spectral distribution (spectrum) of an incandescent lamp with a color temperature Tc = 2856 K. The horizontal axis represents wavelength, and the vertical axis represents relative energy. As shown in this graph, the spectrum of an incandescent lamp is dispersed across the entire visible light region.

[0038] Figure 11 shows an example of the spectral distribution (spectrum) of a low-pressure sodium lamp used in tunnels and other similar applications. The horizontal axis represents wavelength, and the vertical axis represents relative energy. As shown in this graph, the low-pressure sodium lamp exhibits a pulsed spike in energy around 590 nm.

[0039] For example, when photographing a scene illuminated by a light source with a very narrow wavelength range, such as a low-pressure sodium lamp, the resulting image may have a different color tone than the scene actually seen by the eye, depending on the spectral sensitivity characteristics of each pixel in the imager.

[0040] Conventional imager models have difficulty reproducing images taken of scenes illuminated by light sources with such unique spectra.

[0041] In contrast, this technology makes it possible to reproduce phenomena that were difficult to reproduce with the conventional imager models mentioned above with high accuracy.

[0042] <<2. Embodiments>> Next, embodiments of this technology will be described with reference to Figures 12 to 21.

[0043] <Example configuration of autonomous driving simulator 101> Figure 12 shows one embodiment of an autonomous driving simulator 101 to which this technology is applied.

[0044] The autonomous driving simulator 101 is a system that performs simulations of autonomous driving systems to realize autonomous driving and evaluates and verifies the safety of the autonomous driving system. The autonomous driving simulator 101 includes a driving environment-electromagnetic wave propagation-sensor model 111 and an autonomous driving model 112.

[0045] The driving environment-electromagnetic wave propagation-sensor model 111 comprises a rendering model 121, a perception model 122, and a recognition model 123.

[0046] Rendering model 121 is a model that simulates the environment in which a vehicle operates during a virtual driving test, and simulates the electromagnetic waves (e.g., visible light, infrared light, radio waves, etc.) propagated to various sensors on the vehicle in the simulated environment.

[0047] For example, rendering model 121 simulates the environment in which the vehicle will travel, based on a 3D model of the vehicle subject to the virtual driving test, the characteristics of various objects, and the virtual driving test scenario. The characteristics of an object include, for example, its type, size, shape, texture (surface characteristics), and reflectivity. If an object has a light source, the characteristics of the object include, for example, the frequency and duty cycle of the light source's blinking. The virtual driving test scenario includes, for example, the route the vehicle will travel and the conditions of the virtual space around the vehicle on the route. The conditions of the virtual space include, for example, the position and movement of various objects (including people), the time of day, the weather, the road surface conditions, etc.

[0048] The rendering model 121 simulates electromagnetic waves propagating from the virtual space surrounding the vehicle to the various sensors installed on the vehicle in the simulated environment. These electromagnetic waves also include reflected waves from electromagnetic waves virtually emitted from the perception model 122 around the vehicle (e.g., reflected waves from millimeter-wave radar). The rendering model 121 supplies rendering data, including data representing the simulated electromagnetic waves, to the perception model 122.

[0049] The perception model 122 is a model that simulates the perception steps of an autonomous driving system. For example, the perception model 122 simulates the perception process in which various sensors perceive the surrounding environment of the vehicle (e.g., surrounding objects) based on electromagnetic waves simulated by the rendering model 121. The perception model 122 generates perception data that shows the results of perceiving the surrounding environment of the vehicle and supplies it to the recognition model 123.

[0050] The perception model 122 includes, for example, models corresponding to sensors installed in a vehicle that perceive objects using electromagnetic waves. For example, the perception model 122 includes, for example, an imager model 131, a millimeter-wave radar model 132, and a LiDAR (Light Detection and Ranging) model 133.

[0051] The imager model 131 is a model that performs a simulation of the imager (image sensor) installed in the vehicle. For example, the imager model 131 generates captured images (hereinafter referred to as virtual captured images) of the virtual space around the vehicle based on the light (incident light) contained in the electromagnetic waves simulated by the rendering model 121. The imager model 131 is a type of perceptual data and supplies virtual captured image data corresponding to the virtual captured image to the recognition model 123.

[0052] The millimeter-wave radar model 132 is a model that performs a simulation of the millimeter-wave radar installed in the vehicle. For example, the millimeter-wave radar model 132 simulates the process of transmitting a millimeter-wave signal within a predetermined range around the vehicle, receiving the reflected wave, and generating an IF (intermediate frequency) signal by mixing the transmitted and received waves. The millimeter-wave radar model 132 supplies the IF signal (hereinafter referred to as the virtual IF signal), which is a type of perceptual data, to the recognition model 123.

[0053] LiDAR model 133 is a model that performs a simulation of the LiDAR system installed in the vehicle. For example, LiDAR model 133 simulates the process of irradiating a predetermined area around the vehicle with laser light, receiving the reflected light, and generating point cloud data based on the reflected light. LiDAR model 133 supplies point cloud data (hereinafter referred to as virtual point cloud data), which is a type of perceptual data, to recognition model 123.

[0054] The recognition model 123 is a model that simulates the recognition steps of an autonomous driving system. For example, the recognition model 123 simulates the process of recognizing the situation around the vehicle based on virtual image data, virtual IF signals, and virtual point cloud data. For example, the recognition model 123 recognizes the type, position, size, shape, and movement of various objects (including people) around the vehicle. The recognition model 123 supplies data indicating the recognition results (hereinafter referred to as virtual recognition data) to the autonomous driving model 112.

[0055] Furthermore, the recognition model 123 may perform recognition processing individually based on the perceptual data for each sensor to generate virtual recognition data for each sensor, or it may perform sensor fusion (sensor fusion) of the perceptual data for each sensor and perform recognition processing based on the fused perceptual data.

[0056] The autonomous driving model 112 is a model that simulates the decision-making and operation steps of an autonomous driving system. For example, the autonomous driving model 112 simulates the process of judging the situation around the vehicle based on virtual recognition data and predicting the risks the vehicle may encounter. For example, the autonomous driving model 112 simulates the process of creating an action plan, such as a driving route, based on a planned route and predicted risks. For example, the autonomous driving model 112 simulates the process of executing automated vehicle operations based on the created action plan.

[0057] The autonomous driving model 112 feeds back information representing the vehicle's virtual state to the rendering model 121. The vehicle's virtual state includes, for example, the vehicle's driving conditions (e.g., speed, direction, braking, etc.) and the vehicle's driving position.

[0058] Furthermore, if the autonomous driving model 112 is configured such that the virtual recognition data supplied by the recognition model 123 is separated by sensor type, it performs a fusion of the recognition results indicated by each type of virtual recognition data (sensor fusion).

[0059] <Example of pixel arrangement for imager model 131> Figure 13 shows an example of the pixel arrangement of the imager model 131.

[0060] In this example, the pixels of imager model 131 are arranged according to a Bayer array.

[0061] Hereafter, a unit consisting of four pixels arranged in a 2x2 grid (Gr, R, B, and Gb pixels) will be referred to as a pixel unit. Therefore, in the imager model 131, pixels are arranged in units of pixel units. Furthermore, the color of the captured image is represented in units of pixel units.

[0062] <About Imaging Models> Next, we will describe the imaging model of the autonomous driving simulator 101.

[0063] The imaging model consists of all or part of the rendering model 121 in Figure 12 and the imager model 131. The imaging model simulates incident light incident on the imager model 13 and generates virtual image data based on the simulated incident light, etc.

[0064] Imaging models are broadly divided into two types: gold models and silver models.

[0065] The Gold Model is used when conducting tests that do not require real-time processing, such as tests of the perception and recognition steps. The Gold Model is further divided into the Gold+ (plus) model, the Gold Base model, and the Gold- (minus) model.

[0066] The Gold+ model is a model that corresponds to the simulation of the incident direction of light incident on the imager model 131.

[0067] The gold-based model is a model that corresponds to the spectral information of the light incident on the imager model 131 and the simulation of the exposure period of the imager model 131.

[0068] The Gold model is a model that corresponds to the simulation of the exposure period of the imager model 131.

[0069] The silver model is used, for example, when conducting tests that require real-time processing similar to that of an actual vehicle, such as testing decision steps and operation steps.

[0070] Figure 14 is a table showing the correspondence between the real-world space and autonomous driving system simulated by the autonomous driving simulator 101 and each model of the imaging model. In the figure, the column labeled "Embedded" shows the configuration of the real-world space and autonomous driving system. The column labeled "Simulation" shows the configuration of the real-world space and autonomous driving system that the imaging model corresponds to. The first row of the Simulation column shows the correspondence status of the Silver model, the second row shows the correspondence status of the Gold- model, the third row shows the correspondence status of the Gold Base model, and the fourth row shows the correspondence status of the Gold+ model.

[0071] The Silver model corresponds to the trajectory of an object in the real world, as well as the RGB components and properties of the object's surface (texture). In other words, the Silver model simulates the aforementioned properties of the object.

[0072] In the silver model, the incident light on the imager model 131 is represented by the R component, G component, and B component.

[0073] The silver model corresponds to the camera lens within the optical system that makes up the vehicle and autonomous driving system. In other words, the silver model simulates the effect of the camera lens on the light incident on the imager model 131.

[0074] The Silver model corresponds to a portion of the signal processing component of a real-world imager. In other words, the Silver model simulates a part of the imager's signal processing.

[0075] Figure 15 shows an example of imager signal processing simulated in the Silver model. In the Silver model, for example, HDR synthesis processing and post-HDR synthesis RAW processing are simulated as part of the detection system processing. Post-HDR synthesis RAW processing includes, for example, white spot and black spot correction processing. FW control processing is also simulated.

[0076] The Gold model corresponds to the trajectory of real-world objects, the RGB components and surface characteristics of the object's surface, and the flashing frequency and duty cycle of the light source. In other words, the Gold model simulates the aforementioned characteristics of objects and light sources.

[0077] In the Gold model, the incident light on the imager model 131 is represented by its R, G, and B components. Furthermore, the Gold model simulates the exposure characteristics of the imager for the light incident on the imager model 131.

[0078] The gold model corresponds to the camera lens within the optical system that makes up the vehicle and autonomous driving system. In other words, the gold model simulates the effect of the camera lens on the light incident on the imager model 131.

[0079] The Gold model corresponds to the signal processing component of a real-world imager. In other words, the Gold model simulates at least a portion of the imager's signal processing.

[0080] Furthermore, the gold model simulates more signal processing compared to the silver model.

[0081] Figure 16 shows an example of signal processing for an imager simulated in the Gold model. For example, the Gold model simulates the processing of the sub-pixel structure and spectral quantum efficiency processing described above, with reference to Figure 3. For example, quantum efficiency conversion processing and AD (Analog / Digital) conversion processing are simulated. For example, rolling shutter processing is simulated. For example, as part of the detection system processing, RAW processing before HDR synthesis, HDR synthesis processing, and RAW processing after HDR synthesis are simulated. RAW processing after HDR synthesis includes, for example, noise reduction. For example, PWL (Phase-Wave-Free) processing and FW processing are simulated.

[0082] The Gold-based model corresponds to the trajectory of real-world objects, their surface properties, and the spectral characteristics, flashing frequency, and duty cycle of light sources. In other words, the Gold-model simulates the aforementioned characteristics of objects and light sources.

[0083] In the gold-based model, the incident light on imager model 131 is represented by its spectral characteristics. Furthermore, the gold-based model simulates the exposure characteristics of the imager in relation to the incident light on imager model 131.

[0084] The gold base model corresponds to the camera lens within the optical system that makes up the vehicle and autonomous driving system. In other words, in the gold base model, the characteristics of the camera lens are simulated for light incident on the imager model 131.

[0085] The Gold-based model corresponds to the CFA (Color Filter Array) and signal processing components of a real imager. In other words, the Gold model simulates the characteristics of the CFA and the same signal processing as the Gold model for light incident on the imager model 131.

[0086] The Gold+ model corresponds to the trajectories of real-world objects, the surface properties of objects, and the spectral properties of light sources. In other words, the Gold- model simulates the aforementioned properties of objects and light sources.

[0087] In the Gold+ model, the incident light on the imager model 131 is represented by its spectral characteristics.

[0088] The Gold+ model corresponds to the windshield, headlights, camera module, and camera lens, which are part of the optical system that makes up the vehicle and autonomous driving system. In other words, in the Gold+ model, the characteristics of the windshield, headlights, camera module, and camera lens are simulated for light incident on the imager model 131.

[0089] The Gold+ model corresponds to the package, OCL (On-Chip Lens), CFA, and signal processing components of a real imager. In other words, the Gold model simulates the characteristics of the package, OCL, and CFA for light incident on imager model 131, and simulates the same signal processing as the Gold- and Gold-Base models.

[0090] <Types of input / output data for Imager Model 131> Figure 17 shows examples of the types of input data (rendering data) to the imager model 131 and the output data (virtual captured image data) from the imager model 131 for each imaging model.

[0091] In the Gold+ model, the input data includes spectral information and incident direction information. That is, the input data includes spectral information for each of a predetermined number of incident directions of the light incident on each pixel of the imager model 131. The output data includes virtual captured image data corresponding to a high-fidelity image. A high-fidelity image is, for example, an HD (High Definition), 4K, or 8K image.

[0092] In the gold-based model, the input data includes spectral information and exposure information. Specifically, the input data includes spectral information of the incident light incident on each pixel of the imager model 131 during its exposure period. The output data includes virtual captured image data corresponding to a high-fidelity image.

[0093] In the Gold model, the input data includes RGB information and exposure information. Specifically, the input data includes information on the R, G, and B components (RGB information) of the incident light incident on each pixel of the imager model 131 during the exposure period. The output data includes virtual captured image data corresponding to a high-fidelity image.

[0094] The format of the virtual image data is not particularly limited. For example, RAW format or YUV format may be used.

[0095] In the silver model, the input data includes RGB information of the incident light to each pixel of the imager model 131. The RGB information of the incident light is input at a frame rate of, for example, 30 fps (frames per second). The exposure characteristics of each pixel are not reflected in the RGB information of the incident light to each pixel. The input data also includes I2C data, embedded data, FuSa (Function Safety) data, and Cyber ​​Security data. The output data includes a standard fidelity image, I2C data, embedded data, FuSa data, and Cyber ​​Security data. The standard fidelity image is, for example, an SD (Standard Definition) image.

[0096] Furthermore, in the Gold+ model, exposure information may be included in the input data. That is, the input data may include spectral information for each of a predetermined number of incident directions (Multi Direction) of incident light incident on each pixel of the imager model 131 during the exposure period.

[0097] Furthermore, in the Gold+ model, Gold Base model, and Gold model, for example, I2C data, embedded data, FuSa data, and Cyber ​​Security data may be included as input and output data.

[0098] <Data structure between rendering model and imager model> Next, we will describe an example of the data structure that is input and output between the rendering model 121 and the imager model 131.

[0099] <Example of data structure related to the intensity of incident light> First, referring to Figure 18, we will describe an example of the data structure related to the intensity of incident light incident on each pixel of the imager model 131 from the rendering model 121.

[0100] WellInPix is ​​a constant that indicates the number of pixels in a pixel unit used to reproduce the colors of a captured image. As mentioned above, the pixels of the imager model 131 are arranged according to a Bayer array, and since each pixel unit consists of 4 pixels, WellInPix is ​​set to 4.

[0101] Spectral0 is a constant that indicates the minimum wavelength of the spectral information. In this example, Spectral0 is set to 400 nm.

[0102] SpectralI is a constant that indicates the bandwidth of the spectral information. In this example, SpectralI is set to 5 nm.

[0103] SpectralN is a constant that indicates the number of bands in the spectral information. In this example, SpectralN is set to 60.

[0104] Therefore, the maximum wavelength of the spectral information is 700 nm (= 400 nm + 5 nm × 60). Furthermore, the spectral information shows the intensity of incident light in a total of 60 bands, each with a bandwidth of 5 nm, within the wavelength range of 400 nm to 700 nm.

[0105] PhotonFlux, RGBLuminance, and RGBIrradiance are structures that indicate the intensity of incident light to each pixel in a single pixel unit.

[0106] PhotonFlux measures the intensity of incident light to each pixel constituting the pixel unit, and is measured by the photon flux density (in μmol / m³). 2 It is a structure represented by ·s). PhotonFlux contains a variable SP[WellInPix] which is a one-dimensional array of type uint. For example, SP[1] represents the photon flux density of incident light to the Gr pixel, SP[2] represents the photon flux density of incident light to the R pixel, SP[3] represents the photon flux density of incident light to the B pixel, and SP[4] represents the photon flux density of incident light to the Gb pixel.

[0107] RGB Luminance is the intensity of incident light on each pixel that makes up a pixel unit, measured in luminance (unit: cd / m²). 2 RGBLuminance is a structure represented by RGBLuminance. RGBLuminance contains SP[WellInPix], which is a one-dimensional array variable of type float. For example, SP[1] represents the luminance of the light incident on the Gr pixel, SP[2] represents the luminance of the light incident on the R pixel, SP[3] represents the luminance of the light incident on the B pixel, and SP[4] represents the luminance of the light incident on the Gb pixel.

[0108] RGBIrradiance is the intensity of incident light to each pixel that makes up a pixel unit, measured in irradiance (unit: W / m²). 2RGBIrradiance is a structure represented by RGBIrradiance. RGBIrradiance contains SP[WellInPix], which is a one-dimensional array variable of type float. For example, SP[1] represents the radiant intensity of incident light to the Gr pixel, SP[2] represents the radiant intensity of incident light to the R pixel, SP[3] represents the brightness of incident light to the B pixel, and SP[4] represents the radiant intensity of incident light to the Gb pixel.

[0109] For example, in the gold-based model, gold-model, and silver model, the intensity of incident light to each pixel of the imager model 131 is represented using one of the following structures: PhotonFlux, RGBLuminance, or RGBIrradiance. Furthermore, PhotonFlux, RGBLuminance, and RGBIrradiance can be converted to each other. Therefore, for example, it is possible to use the three structures interchangeably as needed, or to unify the intensity of incident light represented by different structures so that it can be represented by the same structure.

[0110] SpectralRadiance is a structure that represents the spectral information of the incident light to each pixel constituting a pixel unit. SpectralRadiance contains a two-dimensional array variable of type float, SP[WellInPix][SpectralN]. SP[i][j] indicates the spectral radiant intensity of the j-th band of the incident light to the i-th pixel of the pixel unit. For example, SP[1][1] indicates the spectral radiant intensity of the 400nm-405nm band of the incident light to the Gr pixel, SP[1][2] indicates the spectral radiant intensity of the 405nm-410nm band of the incident light to the Gr pixel, and SP[1]

[60] indicates the spectral radiant intensity of the 695nm-700nm band of the incident light to the Gr pixel.

[0111] For example, in the Gold+ model, spectral radiation is used to represent the spectral information of the incident light to each pixel of the imager model 131.

[0112] Furthermore, the wavelength range, band range, and number of bands in the spectral information can be changed. For example, it is possible to set the minimum wavelength of the spectral information to 400 nm, the bandwidth to 30 nm, and the number of bands to 10.

[0113] However, the smaller the bandwidth, the closer the simulation will be to the real world. For example, when reproducing a scene illuminated by a light source with a very narrow wavelength spectrum, such as the low-pressure sodium lamp shown in Figure 11 above, it is expected that the light of the light source's wavelength will be selectively extracted within a single bandwidth, and the influence of light of other wavelengths will be reduced to perform the simulation. In such cases, it is desirable to set the bandwidth to, for example, 5 nm or less. For example, it is desirable to set it to 0 nm < bandwidth ≤ 5 nm.

[0114] On the other hand, as the bandwidth decreases, the number of bands increases, and the processing time for the simulation increases.

[0115] In contrast, for example, the bandwidth may be made variable for each wavelength range. For instance, the bandwidth may be set narrowly near wavelengths corresponding to special light, and widened in other wavelength ranges. For example, the bandwidth may be set to 5 nm in the wavelength range around 590 nm for low-pressure sodium lamps, and to 30 nm in other wavelength ranges. This allows for simulations that are closer to the real world in the wavelength range of special light, while reducing the simulation processing time in other wavelength ranges.

[0116] Furthermore, it may be possible to change the bandwidth during simulation execution. For example, the bandwidth could be set to 5nm when running a simulation that closely resembles the real world, and to 30nm when processing time needs to be reduced. The bandwidth could be arbitrarily changed according to the purpose of the simulation.

[0117] For example, the wavelength range of the spectral information may include the near-infrared and infrared light ranges.

[0118] It is desirable that the number of bands in the spectral information be greater than the number of bands in imager model 131 (the imager being simulated). For example, in the current example, imager model 13 has three bands: R, G, and B. Therefore, it is desirable that the number of bands in the spectral information be set to four or more.

[0119] <Example of data structure related to the direction of incident light> Next, referring to Figure 19, we will describe an example of the data structure regarding the incident direction of incident light incident from the rendering model 121 to each pixel of the imager model 131.

[0120] Direction is a structure that represents the direction of incidence of light to each pixel of the imager model 131. Direction contains a two-dimensional array variable Dir[dx][dy], where Dir[dx][dy] indicates the direction of incidence of the light.

[0121] dx and dy indicate the incident direction of the incident angle, with the central axis of the imager substrate as the reference direction. The central axis of the substrate is, for example, an axis that passes through the center of the substrate and is perpendicular to the substrate. dx indicates the incident angle in the direction tilted to the left or right with respect to the reference direction. When the incident angle is tilted to the right, dx is a positive value, and when the incident angle is tilted to the left, dx is a negative value. dy indicates the incident angle in the direction tilted up or down with respect to the reference direction. When the incident angle is tilted upward, dy is a positive value, and when the incident angle is tilted downward, dy is a negative value.

[0122] For example, Dir[0][0] represents light coming from directly above (vertically), and Dir

[10]

[50] represents light coming from 10 degrees to the right and 50 degrees above.

[0123] For example, in the Gold+ model, Direction is used to represent the direction of incidence of light to each pixel of the imager model 131. For example, by combining Direction with SpectralRadiance as described above (see Figure 18), spectral information for each pixel of the imager model 131 in terms of the direction of incidence of light is represented.

[0124] In the Gold+ model, the incident light data represented by SpectralRadiance and Direction is converted within the imager model 131 to data represented by PhotonFlux, RGBLuminance, or RGBIrradiance, based on the wavelength characteristics of the CFA.

[0125] Furthermore, it is possible to combine Direction with PhotonFlux, RGBLuminance, or RGBIrradiance as described above, as shown in Figure 18.

[0126] <Example of data structure related to exposure characteristics> Next, with reference to Figure 20, an example of the data structure related to the exposure characteristics of each pixel in the imager model 131 will be described.

[0127] Exposure is a structure that represents the exposure characteristics of the imager model 131. Exposure includes uint type variables x0, y0, xw, and yw, as well as float type variables exposure0 and exposureN.

[0128] x0 indicates the starting row of the pixel range for which the exposure period is set.

[0129] y0 indicates the starting row of the pixel range for which the exposure period is set.

[0130] xw indicates the number of pixels in the horizontal (width) direction of the pixel range for which the exposure period is set.

[0131] yw indicates the number of pixels in the vertical direction (height direction) of the pixel range for which the exposure period is set.

[0132] exposure0 indicates the start time of the exposure period.

[0133] Exposure N indicates the length of the exposure period.

[0134] Therefore, Exposure indicates that pixels within a rectangular area of ​​width xw and height yw, starting from coordinate (x0,y0) of the pixel array of imager model 131 (the pixel at the upper left corner of the pixel range), are exposed for a period of length exposureN from the start time exposure0.

[0135] For example, if the imager model 131 uses a global shutter system, the coordinates (x0, y0) are set to (0, 0), the width xw is set to the number of pixels in the horizontal direction of the imager model 131's pixel array, and the height yw is set to the number of pixels in the vertical direction of the imager model 131's pixel array. This ensures that the exposure period for all pixels of the imager model 131 is set to the same duration.

[0136] For example, if the imager model 131 uses a rolling shutter system, the coordinates (x0, y0) are set to (0, yi), the width xw is set to the number of pixels in the horizontal direction of the imager model 131's pixel array, and the height yw is set to 1. This sets the exposure period for the yi row of pixels in the imager model 131. If exposure is performed in units of n rows (2 or more), the height yw is set to n.

[0137] For example, the imager model 131 uses Exposure to notify the rendering model 121 of information regarding the exposure period of each pixel. The rendering model 121 then performs a simulation of the incident light on each pixel during the notified exposure period.

[0138] <Example of buffer structure between rendering model and imager model> Next, with reference to Figure 21, an example of the data structure of the buffer provided between the rendering model 121 and the imager model 131 will be described.

[0139] Buffer1 is a structure that represents a buffer placed between the rendering model 121 and the imager model 131, and stores the data of the incident light to each pixel of the imager model 131. Buffer1 includes the Exposure structure described above (see Figure 20) and the pointer variables *Pix[ROI.xw][ROI.yw].

[0140] The Exposure setting determines the exposure period and exposure range for which incident light data will be stored in the buffer. In other words, the incident light data for each pixel within the exposure range set by Exposure is stored in the buffer during its exposure period.

[0141] Specifically, the period from the start time Exposure.exposure0 to the length Exposure.exposureN is set as the exposure period for which incident light data will be stored in the buffer. The rectangular area with coordinates (Exposure.x0, Exposure.y0) as the starting point, width Exposure.xw, and height Exposure.yw is set as the exposure range for which incident light data will be stored in the buffer.

[0142] *Pix[ROI.xw][ROI.yw] are pointer variables of a two-dimensional array type, which store data (PhotonFlux, RGBLuminance, RGBIrradiance, or SpectralRadiance) indicating the intensity of incident light to each pixel.

[0143] For example, data indicating the intensity of incident light during an exposure period of Exposure.ExposureN, starting from Exposure.exposure0, is stored in a two-dimensional array variable from Pix[Exposure.x0][Exposure.y0] to Pix[Exposure.x0+xw][Exposure.x0+yw] within the exposure range of a rectangle with coordinates (Exposure.x0, Exposure.y0), width Exposure.xw, and height Exposure.yw.

[0144] As described above, the spectral information, incident direction, and exposure period of the incident light incident on each pixel of the imager can be simulated on a pixel-by-pixel basis. Furthermore, the optical system composed of the vehicle and the autonomous driving system, the optical system of the imager, and the signal processing can be simulated on a pixel-by-pixel basis.

[0145] This makes it possible to simulate motion blur, overexposure, underexposure, HDR compositing, flare, ghosting, rolling shutter distortion, flicker, and light sources with special spectra in the virtual captured images output from the imager model 131.

[0146] <<3. Variant Example>> The following describes some modifications of the embodiments of the present technology described above.

[0147] <Variations related to autonomous driving simulators> Figure 22 shows an example configuration of an autonomous driving simulator 301, which is a first modified example of the autonomous driving simulator 101 in Figure 12. In the figure, parts corresponding to the autonomous driving simulator 101 are denoted by the same reference numerals, and their explanations are omitted as appropriate.

[0148] The autonomous driving simulator 301 differs from the autonomous driving simulator 101 in that it has a driving environment-electromagnetic wave propagation-sensor model 311 instead of the driving environment-electromagnetic wave propagation-sensor model 111. The driving environment-electromagnetic wave propagation-sensor model 311 differs from the driving environment-electromagnetic wave propagation-sensor model 111 in that it has a rendering model 321 and an optical model 322 instead of the rendering model 121. In other words, in the autonomous driving simulator 301, the rendering model 121 of the autonomous driving simulator 101 is divided into the rendering model 321 and the optical model 322.

[0149] The rendering model 321 simulates electromagnetic waves propagating from the virtual space surrounding the vehicle to the various sensors installed in the vehicle, without considering the influence of the optical system composed of the vehicle and the autonomous driving system. The rendering model 321 supplies rendering data, including data showing the simulated electromagnetic waves, to the optical model 322.

[0150] Optical model 322 is a model that simulates the optical system composed of the vehicle and the autonomous driving system. Optical model 322 simulates the influence of the optical system on the rendering data. Optical model 322 then supplies the rendering data to perception model 122.

[0151] Furthermore, the rendering model 321, the optical model 322, and the perception model 122 exchange necessary data with each other.

[0152] Figure 23 shows an example configuration of an autonomous driving simulator 301, which is a second modified example of the autonomous driving simulator 101 in Figure 12. In the figure, parts corresponding to the autonomous driving simulator 101 in Figure 12 are denoted by the same reference numerals, and their explanations are omitted as appropriate.

[0153] The autonomous driving simulator 351 differs from the autonomous driving simulator 101 in that it features a driving environment-electromagnetic wave propagation-sensor model 361 instead of the driving environment-electromagnetic wave propagation-sensor model 111. The driving environment-electromagnetic wave propagation-sensor model 361 differs from the driving environment-electromagnetic wave propagation-sensor model 111 in that it features a cognitive model 371 instead of the perception model 122 and recognition model 123. In other words, in the autonomous driving simulator 301, the perception model 122 and recognition model 123 are integrated into the cognitive model 371.

[0154] The cognitive model 371 performs integrated processing of the perceptual model 122 and the recognition model 123. It is also possible for the cognitive model 371 to include separate models for each sensor, similar to the perceptual model 122.

[0155] <Other variations> For example, the imager model 131 may perform a simulation of the exposure period. For example, time-series data showing the incident light to each pixel of the imager model 131 may be input from the rendering model 121 to the imager model 131, and the imager model 131 may select the time-series data of the incident light based on the exposure period of each pixel.

[0156] Furthermore, the imaging model (rendering model 121 and imager model 131) may switch between gold+ model, gold base model, gold- model, and silver model based on the scene to be simulated. That is, based on the scene to be simulated, the rendering model 121 selects the object to be rendered and changes the data structure of the generated rendering data, and the imager model 131 selects the functions and characteristics of the imager to be simulated.

[0157] For example, the silver model is selected when simulating a scene that requires real-time processing. For example, the gold-based model or gold-model is selected when simulating a scene where rolling shutter distortion occurs. For example, the gold-based model is selected when simulating a scene under special lighting conditions. For example, the gold+ model is selected when simulating a scene where flare or ghosting occurs.

[0158] The above explanation described the data structure of incident light incident on the imager model 131, but electromagnetic waves propagating to other models of the perception model 122 can also be represented by a similar data structure. For example, the intensity and spectral information of electromagnetic waves can be represented by PhotonFlux, RGBLuminance, RGBIrradiance, and SpectralRadiance in Figure 18. For example, the propagation direction of electromagnetic waves can be represented by Direction in Figure 19. For example, by combining SpectralRadiance and Direction, the intensity or spectral information for each propagation direction of electromagnetic waves can be represented. This electromagnetic wave data is included, for example, in the rendering data output from the rendering model 121.

[0159] The above explanation demonstrated an example of applying this technology to an autonomous driving simulator that simulates the autonomous driving system of a vehicle, but it can also be applied to simulators that perform other types of simulations.

[0160] For example, this technology can be applied to simulators for autonomous driving systems of various types of mobility devices other than vehicles. Examples of applicable mobility devices include trains, ships, aircraft, drones, motorcycles, personal mobility devices, robots, construction machinery, and agricultural machinery (e.g., tractors).

[0161] For example, this technology can be applied to simulators for driving systems other than autonomous driving of mobile devices.

[0162] For example, this technology can also be applied to simulators of systems that perform perceptual and recognition steps other than those of mobile devices.

[0163] <<4. Others>> <Example of computer configuration> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0164] Figure 24 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0165] In computer 1000, the CPU (Central Processing Unit) 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0166] An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0167] The input section 1006 consists of input switches, buttons, a microphone, an image sensor, etc. The output section 1007 consists of a display, a speaker, etc. The storage section 1008 consists of a hard disk or non-volatile memory, etc. The communication section 1009 consists of a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0168] In the computer 1000 configured as described above, the CPU 1001 loads, for example, a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes it, thereby performing the series of processes described above.

[0169] The program executed by computer 1000 (CPU 1001) can be provided by recording it on removable media 1011, such as a packaged media. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0170] In computer 1000, programs can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, programs can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0171] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0172] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

[0173] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0174] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0175] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0176] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0177] <Examples of configuration combinations> This technology can also be configured as follows:

[0178] (1) This perceptual model simulates a sensor that perceives objects using electromagnetic waves, and generates output data showing the result of perceiving an object based on input data that includes at least one of the following: the propagation direction of the electromagnetic wave and spectral information. Information processing device. (2) The aforementioned perception model includes an imager model that performs an imager simulation, The input data includes incident light data that includes at least one of the incident direction and spectral information of the incident light for each pixel. The imager model generates image data based on the incident light data. The information processing device described in (1) above. (3) The incident light data includes at least one of the incident direction and spectral information of the incident light incident on each pixel during its exposure period. The information processing device described in (2) above. (4) The imager model notifies the rendering model that generates the input data of information regarding the exposure period of each pixel. The information processing device described in (3) above. (5) The aforementioned imager model performs a simulation of a rolling shutter type imager. The information processing apparatus described in (3) or (4) above. (6) The incident light data includes the intensity of the incident light for each pixel in each direction of incidence. The information processing apparatus described in any of (2) to (5) above. (7) The incident light data includes spectral information for each pixel in the direction of incidence of the incident light. The information processing device described in (6) above. (8) The spectral information has more bands than the number of bands in the imager model. The information processing device described in any of (2) to (7) above. (9) The bandwidth of the aforementioned spectral information is greater than 0 nm and less than or equal to 5 nm. The information processing device described in any of (2) to (8) above. (10) The imager model selects the functions and characteristics of the imager to be simulated based on the scene to be simulated. The information processing apparatus described in any of (2) to (9) above. (11) A rendering model that simulates incident light entering the imager from the surrounding space. Furthermore, The rendering model changes the data structure of the incident light data based on the scene to be simulated. The information processing device described in (10) above. (12) The aforementioned input data includes the intensity for each direction of electromagnetic wave propagation. The information processing device described in any of (1) to (11) above. (13) The aforementioned input data includes spectral information for each direction of electromagnetic wave propagation. The information processing device described in (12) above. (14) A rendering model that simulates electromagnetic waves propagating from the surrounding space to the sensor. The information processing apparatus further comprises any of the above (1) to (13). (15) The aforementioned perception model performs a simulation of the sensors installed in the vehicle. An information processing device according to any of (1) to (14) above. (16) This system simulates a sensor that perceives objects using electromagnetic waves, and generates output data showing the result of object perception based on input data that includes at least one of the following: the propagation direction of the electromagnetic waves and spectral information. Information processing methods. (17) This system simulates a sensor that perceives objects using electromagnetic waves, and generates output data showing the result of object perception based on input data that includes at least one of the following: the propagation direction of the electromagnetic waves and spectral information. A program that causes a computer to perform a process.

[0179] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur. [Explanation of Symbols]

[0180] 101 Autonomous Driving Simulator, 111 Driving Environment-Electromagnetic Wave Propagation-Sensor Model, 112 Autonomous Driving Model, 121 Rendering Model, 122 Perception Model, 123 Recognition Model, 131 Imager Model, 132 Millimeter-Wave Radar Model, 133 LiDAR Model, 301 Autonomous Driving Simulator, 311 Driving Environment-Electromagnetic Wave Propagation-Sensor Model, 321 Rendering Model, 322 Optical Model, 351 Autonomous Driving Simulator, 361 Driving Environment-Electromagnetic Wave Propagation-Sensor Model, 371 Cognitive Model

Claims

1. The imager model includes an imager simulation that generates image data based on incident light data, which includes at least one of the incident direction and spectral information of the incident light for each pixel, and a perceptual model that generates perceptual data including the image data. Based on the aforementioned perceptual data, a recognition model is used to simulate the process of recognizing the surrounding environment. An information processing device equipped with the following features.

2. The incident light data includes at least one of the incident direction of the incident light incident on each pixel during the exposure period and the spectral information. The information processing apparatus according to claim 1.

3. The imager model notifies the rendering model that generates the incident light data of information regarding the exposure period of each pixel. The information processing apparatus according to claim 2.

4. The imager model is capable of adjusting the exposure period on a pixel-by-pixel basis. The information processing apparatus according to claim 2.

5. The imager model performs a simulation of the rolling shutter type imager. The information processing apparatus according to claim 4.

6. The incident light data includes the intensity of the incident light for each pixel in each direction of incidence. The information processing apparatus according to claim 1.

7. The incident light data includes the spectral information for each pixel in the direction of incidence of the incident light. The information processing apparatus according to claim 6.

8. The spectral information has more bands than the number of bands in the imager model. The information processing apparatus according to claim 1.

9. The bandwidth of the spectral information is greater than 0 nm and less than or equal to 5 nm. The information processing apparatus according to claim 1.

10. The spectral information has a variable bandwidth for each wavelength range. The information processing apparatus according to claim 1.

11. The imager model selects the functions and characteristics of the imager to be simulated based on the scene to be simulated. The information processing apparatus according to claim 1.

12. A rendering model that simulates incident light entering the imager from the surrounding space. Furthermore, The rendering model changes the data structure of the incident light data based on the scene to be simulated. The information processing apparatus according to claim 11.

13. The perception model simulates a sensor including the imager that perceives objects using electromagnetic waves, and generates perception data showing the result of perceiving an object based on input data including at least one of the propagation direction of electromagnetic waves and spectral information. The information processing apparatus according to claim 1.

14. A rendering model that simulates electromagnetic waves propagating from the surrounding space to the sensor. The information processing apparatus according to claim 13, further comprising:

15. An imaging model comprising part or all of the rendering model and the imager model comprises a model used for tests that do not require real-time processing, and a model used for tests that require real-time processing. The information processing apparatus according to claim 14.

16. The perception model performs a simulation of the sensors installed in the vehicle, The aforementioned recognition model simulates the process of recognizing the surrounding environment of the vehicle. The information processing apparatus according to claim 13.

17. The recognition model generates recognition data indicating the recognition result of the situation around the vehicle, Based on the aforementioned recognition data, an autonomous driving model is created to simulate the decision-making and operation steps of the autonomous driving system. The information processing apparatus according to claim 16, further comprising:

18. A rendering model for simulating electromagnetic waves propagating from the space surrounding the vehicle to the sensor. Furthermore, The autonomous driving model feeds back information representing the virtual state of the vehicle to the rendering model. The information processing apparatus according to claim 17.

19. The perception model further includes at least one of a millimeter-wave radar model that performs a simulation of a millimeter-wave radar and a LiDAR model that performs a simulation of LiDAR (Light Detection and Ranging). The information processing apparatus according to claim 13.

20. The millimeter-wave radar model simulates the process of transmitting a millimeter-wave signal within a predetermined range, receiving the reflected wave, and generating an intermediate frequency signal by mixing the transmitted wave and the received wave. The LiDAR model simulates a process in which laser light is shone within a predetermined range, the reflected light is received, and point cloud data is generated based on the reflected light. The perceptual data further includes at least one of the intermediate frequency signal and the point cloud data. The information processing apparatus according to claim 19.

21. The recognition model performs fusion of the perceptual data for each sensor and performs recognition processing based on the fused perceptual data. The information processing apparatus according to claim 19.

22. The imager is simulated to generate perceptual data, including image data, based on incident light data that includes at least one of the incident light direction and spectral information for each pixel. Based on the aforementioned perceptual data, the process of recognizing the surrounding situation is simulated. including Information processing methods.

23. The imager is simulated to generate perceptual data, including image data, based on incident light data that includes at least one of the incident light direction and spectral information for each pixel. Based on the aforementioned perceptual data, the process of recognizing the surrounding situation is simulated. including A program that causes a computer to perform a process.

24. A sensor that perceives objects using electromagnetic waves is simulated, and a perceptual model is generated that shows the result of perceiving an object based on input data including the intensity for each direction of electromagnetic wave propagation. Based on the aforementioned perceptual data, a recognition model is used to simulate the process of recognizing the surrounding environment. An information processing device equipped with the following features.

25. The aforementioned input data includes spectral information for each direction of electromagnetic wave propagation. The information processing apparatus according to claim 24.

26. This involves simulating a sensor that perceives objects using electromagnetic waves, and generating perceptual data that shows the result of perceiving an object based on input data including the intensity for each direction of electromagnetic wave propagation. Based on the aforementioned perceptual data, the process of recognizing the surrounding situation is simulated. Information processing methods including

27. This involves simulating a sensor that perceives objects using electromagnetic waves, and generating perceptual data that shows the result of perceiving an object based on input data including the intensity for each direction of electromagnetic wave propagation. Based on the aforementioned perceptual data, the process of recognizing the surrounding situation is simulated. A program that causes a computer to perform a process that includes [a specific action].

Images