Simulation device, simulation system, and control method

Abstract

The present technology relates to a simulation device, a simulation system, and a control method which make it possible to process different types of pieces of image data. The present technology comprises: an image input interface that generates an instance, manages the address of an image memory and the types of images in the instance, and determines the types of the images stored in the instance; and a processing unit that executes processing corresponding to the types of the images. The present technology may be applied to, for example, a simulation device that performs simulation by using image data from a rendering device.

Description

Simulation device, simulation system, and control method

[0001] The present technology relates to a simulation device, a simulation system, and a control method, and more particularly to a simulation device, a simulation system, and a control method that are capable of standardizing and handling a plurality of different types of image data, for example.

[0002] In the development of autonomous driving systems, image recognition algorithms are verified through simulations in virtual space. In verifying such image recognition algorithms, a sensor model of the image sensor is used to generate input images that closely resemble real-life images using CG (Computer Graphics) synthesis technology.

[0003] Patent Document 1 proposes that noise and the like that occurs when an HDR image sensor is used be reproduced so that the recognition performance of an image recognition algorithm can be correctly evaluated.

[0004] Japanese Patent Publication No. 2022-99651

[0005] In the simulation system proposed in Patent Document 1, the rendering system and the sensor model have a one-to-one correspondence. Because the sensor model is configured with specifications suited to the rendering system, it is difficult to process signals from a rendering system with different specifications. If the specifications of the rendering system are changed, the sensor model must be redesigned to suit the new rendering system.

[0006] The present technology has been made in view of such circumstances, and makes it possible to provide a highly versatile simulation device.

[0007] A simulation device according to one aspect of the present technology is a simulation device that generates an instance, manages an image memory address and an image type in the instance, and includes an image input interface that determines the type of image stored in the instance, and a processing unit that executes processing corresponding to the type.

[0008] A control method according to one aspect of the present technology is a control method in which an image input interface that controls input and output of image data generates an instance, manages an image memory address and an image type in the instance, and supplies the image data to a processing unit that executes processing corresponding to the type.

[0009] A simulation system according to one aspect of the present technology comprises a rendering device and a simulation device that performs a simulation using image data from the rendering device, wherein the simulation device generates an instance in response to an instruction from the rendering device, manages the address of an image memory using the instance, stores the image data and image type from the rendering device in the image memory, and includes a processing unit that executes processing corresponding to the type.

[0010] In a simulation device according to one aspect of the present technology, an instance is generated, an image memory address and an image type are managed in the instance, and processing corresponding to the type is executed based on the image data and type stored in the image memory.

[0011] In a control method according to one aspect of the present technology, an image input interface that controls the input and output of image data generates an instance, manages the address of an image memory and the type of image in the instance, determines the type of image stored in the instance, and supplies the image data to a processing unit that executes processing corresponding to the type.

[0012] A simulation system according to one aspect of the present technology comprises a rendering device and a simulation device that performs a simulation using image data from the rendering device, and the simulation device generates an instance in response to instructions from the rendering device, manages the address of an image memory using the instance, stores the image data and image type from the rendering device in the image memory, and executes processing corresponding to the image data and image type stored in the image memory.

[0013] The simulation device may be an independent device or an internal block constituting one device.

[0014] FIG. 1 is a diagram illustrating a configuration of an embodiment of a simulation system to which the present technology is applied. FIG. 2 is a diagram illustrating an operation of a simulation. FIG. 3 is a diagram illustrating a relationship between an instance and an image memory. FIG. 4 is a diagram illustrating processing of a sensor model. FIG. 5 is a diagram illustrating simulation processing of a sensor model. FIG. 6 is a diagram illustrating storage of data in an image memory. FIG. 7 is a diagram illustrating distortion reproduction. FIG. 8 is a diagram illustrating another configuration of a simulation system. FIG. 9 is a diagram illustrating distortion reproduction. FIG. 10 is a diagram illustrating an example configuration of a PC.

[0015] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described.

[0016] <Configuration Example of Simulation System> Fig. 1 is a diagram showing a configuration example of an embodiment of a simulation system to which the present technology is applied. The simulation system 10 shown in Fig. 1 can be applied, for example, to simulating the operation of a camera system in an autonomous driving system for automobiles or an Advanced Driver Assistance System (ADAS). The simulation system 10 is configured by hardware, software, or a combination of these.

[0017] 1 includes rendering systems (rendering devices) 21-1, 21-2, and 21-3, and a sensor model (simulation device) 31. The rendering systems 21-1 to 21-3 generate image data for each pixel of a pixel model used in the sensor model 31, and supply the image data to the sensor model 31.

[0018] The rendering system 21-1 generates image data of spectral irradiance, the rendering system 21-2 generates image data of RGB (Red, Green, Blue) irradiance, and the rendering system 21-3 generates image data of the number of photons. Any one of the rendering systems 21-1 to 21-3 is connected to the sensor model 31 and configured to supply data. In FIG. 1, the rendering system 21-1 is connected to the sensor model 31 and is indicated by a solid arrow, while the rendering systems 21-2 and 21-3 are not connected to the sensor model 31 and are indicated by dotted arrows.

[0019] Note that a state in which the rendering systems 21-1 to 21-3 are physically connected but no data is output is also included in the "not connected" state. This embodiment also includes a case in which the rendering systems 21-1 to 21-3 are connected in a state in which they can output data to the sensor model 31, but only one of the rendering systems 21 outputs data.

[0020] In this embodiment, the sensor model 31 is configured to be able to handle data from a plurality of rendering systems 21 that output different types of data.

[0021] The sensor model 31 includes an input image normalization unit 41, a spectral irradiance processing unit 42, an RGB irradiance processing unit 43, a photon number processing unit 44, and a pixel model processing unit 45. The spectral irradiance processing unit 42 includes a distortion reproduction unit 51 and a pixel model generation unit 52, the RGB irradiance processing unit 43 includes a distortion reproduction unit 53 and a pixel model generation unit 54, and the photon number processing unit 44 includes a distortion reproduction unit 55 and a pixel model generation unit 56.

[0022] The input image normalization unit 41 determines whether the image data supplied from the rendering system 21 is spectral irradiance data, RGB irradiance data, or photon count data, and based on the determination result, causes a subsequent processing unit to perform processing.

[0023] The input image standardization unit 41 controls the input and output of image data from the rendering system 21, determines the type of input image data, and appropriately controls so that processing according to the determination result is performed at a later stage. The input image standardization unit 41 functions as an image input interface within the sensor model 31. Note that, although the explanation will continue here using an example in which the input image standardization unit 41 is provided within the sensor model 31, it is also possible to provide the input image standardization unit 41 separately from the sensor model 31, between the rendering system 21 and the sensor model 31.

[0024] If it is determined that the data is spectral irradiance data, processing is performed by the spectral irradiance processing unit 42. A distortion reproduction unit 51 of the spectral irradiance processing unit 42 performs processing to generate pixel data in which distortion has been reproduced from the spectral irradiance data. A pixel model generation unit 52 performs processing to convert the spectral irradiance data into pixel model data. For example, the spectral irradiance data supplied from the rendering system 21-1 is a value for each wavelength, and the pixel model generation unit 52 of the spectral irradiance processing unit 42 integrates the value for each wavelength in accordance with the quantum conversion efficiency of the sensor, and then converts it into a value within the sensor model 31, such as the number of electrons, taking into account the pixel size and exposure time.

[0025] If it is determined that the data is RGB irradiance data, processing is performed by the RGB irradiance processing unit 43. A distortion reproduction unit 53 of the RGB irradiance processing unit 43 performs processing to generate pixel data in which distortion has been reproduced from the RGB irradiance data. A pixel model generation unit 54 performs processing to convert the RGB irradiance data into pixel model data. For example, the RGB irradiance data supplied from the rendering system 21-2 is an energy value dispersed into RGB, and the pixel model generation unit 54 of the RGB irradiance processing unit 43 performs matrix calculation processing on the energy value dispersed into RGB, and converts it into a value inside the sensor model 31, such as the number of electrons, taking into account the pixel size and exposure time.

[0026] If it is determined that the data is photon count data, processing is performed by the photon count processing unit 44. The distortion reproduction unit 55 of the photon number processing unit 44 performs processing to generate pixel data in which distortion has been reproduced from the photon count data. The pixel model generation unit 56 performs processing to convert the photon count data into pixel model data. For example, photon count data is supplied from the rendering system 21-3, and the pixel model generation unit 56 of the photon number processing unit 44 converts the input image represented by the photon count data into a value inside the sensor model 31, such as the number of electrons.

[0027] The pixel model data from the spectral irradiance processing unit 42, the RGB irradiance processing unit 43, or the photon number processing unit 44 is supplied to the pixel model processing unit 45, where it is converted into an imaging signal by undergoing predetermined processing, and is output to a downstream application (not shown).

[0028] <Operation of Simulation System> The operation of the simulation system 10 shown in Fig. 1 will be outlined with reference to the flowchart in Fig. 2, and the operation of the sensor model 31 will be described in detail with reference to the flowcharts in Fig. 4 and 5. In the following explanation, the case where image data of spectral irradiance is supplied from the rendering system 21 to the sensor model 31 will be taken as an example, and the explanation will be continued with the case where processing is performed by the spectral irradiance processing unit 42 as an example.

[0029] In step S21, the rendering system 21 instructs the sensor model 31 to generate instances. In step S31, upon receiving the instruction to generate instances from the rendering system 21, the sensor model 31 starts generating instances. The number of instances generated is the same as the number of input images required to generate one frame.

[0030] In step S32, the sensor model 31 returns the address of the instance to the rendering system 21.

[0031] The relationship between an instance and an image memory will be described with reference to Fig. 3. As shown in A of Fig. 3, the sensor model 31 generates an instance 101 managed by the sensor model 31. The image memory 102 (address of the image memory 102) managed by the sensor model 31 is registered in the generated instance 101. In this case, the instance 101 and the image memory 102 are in the sensor model 31, and the address of the instance 101 is returned to the rendering system 21.

[0032] 3B, an instance 101 managed by the sensor model 31 is generated, and an image memory 103 (the address of the image memory 103) managed outside the sensor model 31 is registered in the instance 101. In this case, the address of the instance 101 is returned to the rendering system 21.

[0033] In the following explanation, an example will be given in which the instance 101 and the image memory 102 are managed within the sensor model 31, and the address of the image memory 102 is registered in the instance 101, as shown in A of FIG.

[0034] In step S22 (FIG. 2), the rendering system 21 receives the address of the instance 101 from the sensor model 31, and in step S23 issues a control start request to the sensor model 31. In step S33, the sensor model 31 starts control when it receives the control start request from the rendering system 21. This control start request is made, for example, by executing the Tick() function.

[0035] In step S34, the sensor model 31 requests the rendering system 21 to output image data. In step S24, the rendering system 21 receives the request to output image data from the sensor model 31, and in step S25 registers the image data in the image memory 102 in the instance 101 and outputs the address of the instance 101.

[0036] In step S35, the sensor model 31 receives the image data and address from the rendering system 21, and in step S36, extracts the received image data and manages it within the sensor model 31. The GetImage() function, for example, is used to exchange image data. This exchange of image data is repeated the number of times equal to the number of input images required to generate one frame.

[0037] The sensor model 31 executes a process of generating a pixel model using the image data that it manages.

[0038] <Sensor Model Processing> The processing in the sensor model 31 will be further described with reference to the flowchart shown in FIG.

[0039] In step S101, the input image normalization unit 41 of the sensor model 31 is requested by the rendering system 21 to acquire the number of input images required to generate one frame. The sensor model 31 passes the number of input images required to generate one frame to the rendering system 21.

[0040] In step S102, the rendering system 21 requests the input image normalization unit 41 of the sensor model 31 to generate image storage class instances for sensor model input, the number of which corresponds to the number of input images required to generate one frame. The sensor model 31 generates an instance 101 and returns its address to the rendering system 21.

[0041] In step S103, the input image normalization unit 41 receives a control start request from the rendering system 21. In step S104, the input image normalization unit 41 requests the rendering system 21 to output an instance 101 in which image data is stored, and in response to the request, acquires image data supplied from the rendering system 21. The image data acquired at this time is image data corresponding to one of the number of images (image data that is the processing unit) acquired when the number of input images required to generate one frame is acquired.

[0042] In step S104, the input image normalization unit 41 extracts the input image data from the image memory 102 whose address is managed within the image storage class, in this case, the instance 101.

[0043] The processing up to this point will now be explained again. In step S102, when an instance is generated, the size of the image memory 102 is set by the rendering system 21 side.

[0044] As described with reference to A in Fig. 3, the address of the image memory 102 created by the sensor model 31 is registered in the instance 101 and managed by the sensor model 31. Alternatively, as described with reference to B in Fig. 3, the address of the image memory 102 outside the sensor model 31 is registered in the instance 101 from the rendering system 21 and managed by the sensor model 31.

[0045] In the case of B in Fig. 3, the width, height, and image type of the image are set by the rendering system 21 at the time of registration. The image size is managed by the instance 101 based on the image width, height, and image type. A GPU (Graphics Processing Unit) or a CPU (Central Processing Unit) can be specified as the image memory 102.

[0046] 6, the content of the data stored in the image memory 102, in other words, the storage method (data treated as one pixel), differs for each type of image. The image type is set when the rendering system 21 registers an image in the instance 101. This allows the sensor model 31 to acquire the image type stored in the instance.

[0047] When photon count data is stored in the image memory 102, data for one photon count per pixel is stored serially in pixel order, as shown in A of Fig. 6. When RGB irradiance data is stored in the image memory 102, data for R (Red), G (Green), B (Blue), and A (alpha channel) representing one pixel are stored serially in pixel order, as shown in B of Fig. 6. Note that the A (alpha channel) data may also be configured not to be stored in the image memory 102.

[0048] When spectral irradiance data is stored in the image memory 102, all wavelength data is stored serially for each image, as shown in Fig. 6C. All wavelength data is data for wavelength intervals from the minimum wavelength to the maximum wavelength. As shown in the right diagram of Fig. 6C, data from wavelength[0], which is the minimum wavelength, to wavelength[N], which is the maximum wavelength, is stored in one pixel.

[0049] In step S106, the input image normalization unit 41 determines whether or not the acquisition of image data has been repeated for the number of input images required to generate one frame. If it is determined in step S106 that the acquisition of image data for the number of input images required to generate one frame has not been completed, the process returns to step S104, and the subsequent processes are repeated. On the other hand, if it is determined in step S106 that the acquisition of image data for the number of input images required to generate one frame has been completed, the process proceeds to step S107.

[0050] In step S107, a simulation process of the sensor model is executed. The simulation process of the sensor model executed in step S107 will be described with reference to the flowchart of FIG.

[0051] In step S121, the input image normalization unit 41 determines the type and format of the image supplied from the rendering system 21. The image type may be, for example, spectral irradiance, RGB irradiance, or photon count. The format may be, for example, 16-bit floating point, 32-bit floating point, or 64-bit floating point.

[0052] As explained with reference to Figure 6, the image memory 102 stores data for one pixel in an area corresponding to one pixel, and the input image standardization unit 41 obtains the type of image in the instance 101 to determine the type of image, and also determines the type by referring to the image data stored in the image memory 102.

[0053] In step S122, it is determined whether or not to perform distortion reproduction. If it is determined in step S122 that distortion reproduction is to be performed, the process proceeds to step S123. In step S123, distortion reproduction is performed by the distortion reproduction unit 51 (FIG. 1). The process related to distortion reproduction will be described below with reference to FIG. 7.

[0054] The rendering system 21 generates a rendering image for a very short time, and the rendering image is supplied to the sensor model 31 for each very short time. Because the rendering image for a very short time is treated as a single input image data, steps S103 (FIG. 4) to S108 (FIG. 4) are repeatedly executed, and distortion correction calculations are performed after the execution has been performed a number of times equivalent to the time that constitutes one frame, or during the repeated execution.

[0055] The distortion reproduction unit 51 of the sensor model 31 reproduces the sensor distortion by integrating the supplied rendering image for the pixel to be exposed. In step S124, it is determined whether image data having the reproduced sensor distortion has been generated by the distortion reproduction unit 51, and if so, the image data is supplied to the pixel model generation unit 52. For example, if spectral irradiance data is supplied, the distortion reproduction unit 51 generates image data of the spectral irradiance in which the sensor distortion has been reproduced, and the generated image data is supplied to the pixel model generation unit 52.

[0056] On the other hand, if it is determined in step S124 that image data having the sensor distortion reproduced by the distortion reproduction unit 51 has not been generated, the image data is not supplied to the pixel model generation unit 52, and the input of the next rendering image at a very short time is awaited. That is, in this case, the process returns to step S103 (FIG. 4), and the subsequent processes are repeated to input the necessary rendering image.

[0057] On the other hand, if it is determined in step S122 that distortion reproduction is not to be performed, the process proceeds to step S125. In step S125, image conversion processing is performed. The image data is supplied to a processing unit that processes an image corresponding to the type determined by the input image normalization unit 41, and processing is performed. If the input image normalization unit 41 determines that the image type is spectral irradiance, the image data is supplied to the pixel model generation unit 52 of the spectral irradiance processing unit 42, and processing is performed to convert, for example, the spectral irradiance data into electron number data.

[0058] If the input image normalization unit 41 determines that the type of image is RGB irradiance, the image data is supplied to the pixel model generation unit 54 of the RGB irradiance processing unit 43, and a process is performed to convert, for example, the RGB irradiance data into electron number data.

[0059] If the input image normalization unit 41 determines that the type of image is photon count, the image data is supplied to the pixel model generation unit 56 of the photon count processing unit 44, and a process is performed to convert, for example, the photon count data into electron count data.

[0060] The data processed by the spectral irradiance processing unit 42, the RGB irradiance processing unit 43, or the photon number processing unit 44 is supplied to the pixel model processing unit 45. In step S126, the pixel model processing unit 45 performs predetermined processing on the supplied pixel model to generate an imaging signal. When this processing is executed in step S107 (FIG. 4), the processing of the sensor model 31 for one frame of image is completed, and a loop is executed in step S108, returning to step S103 (FIG. 4) for accepting a control start request for the next input.

[0061] As described above, the input image standardization unit 41 has the function of generating an instance in response to a request from the rendering system 21. The input image standardization unit 41 has the function of receiving and managing the address of the memory for the image from the rendering system 21. At the time of registration, the width, height, and type of the image are set by the rendering system 21. The input image standardization unit 41 has the function of passing image information such as width and height to the rendering system. The input image standardization unit 41 also has the function of passing the address of the image memory to the rendering system 21.

[0062] The input image normalization unit 41 has a function of accepting settings of wavelength information (minimum wavelength, maximum wavelength, wavelength resolution) of spectral irradiance from the rendering system 21, and passing the wavelength information for spectral irradiance to the rendering system 21. The input image normalization unit 41 has a function of accepting settings of information for R, G, B, and A of RGB irradiance from the rendering system 21, and passing the information for RGB irradiance to the rendering system 21. The input image normalization unit 41 has a function of accepting settings of information on the number of photons from the rendering system 21, and passing the information on the number of photons to the rendering system 21.

[0063] The input image normalization unit 41 or the instance can receive a pixel value gain setting from the rendering system 21 and request the sensor model 31 to multiply all pixel values ​​of the image data by the collective gain. The input image normalization unit 41 or the instance can have a function of passing the pixel value gain to the rendering system 21.

[0064] By having these functions, the input image normalization unit 41 can collectively handle image data from the rendering system 21 that handles different types of image data.

[0065] <Configuration of Sensor Model in Second Embodiment> Fig. 8 is a diagram showing an example of the configuration of a sensor model in the second embodiment. The sensor model 231 shown in Fig. 8 includes an input image normalization unit 241, a spectral irradiance processing unit 242, an RGB irradiance processing unit 243, a photon number processing unit 244, and a pixel model processing unit 245. The spectral irradiance processing unit 242 includes a pixel model generation unit 252, the RGB irradiance processing unit 243 includes a pixel model generation unit 254, and the photon number processing unit 244 includes a pixel model generation unit 256.

[0066] 1 in that the sensor model 231 is configured without the distortion reproduction units 51, 53, and 55. The sensor model 231 also includes a line shutter information output unit 246 that outputs line shutter information from the pixel model processing unit 245 to the rendering system 21.

[0067] The sensor model 231 in the second embodiment is not configured to perform distortion reproduction within the sensor model 231, but is configured to acquire image data on which distortion reproduction has been performed from the rendering system 21.

[0068] The processing related to distortion reproduction will be described with reference to Fig. 9. In order for the rendering system 21 to generate image data of an image in which distortion has been reproduced, the line shutter information output unit 246 of the sensor model 231 acquires shutter control information, in which an exposure start time (open) and an exposure end time (close) are associated for each line, from the pixel model processing unit 245 and supplies the information to the rendering system 21.

[0069] The rendering system 21 generates a frame in which the distortion is reproduced using the shutter control information supplied from the sensor model 231 , and supplies the image data to the sensor model 231 .

[0070] The sensor model 231 in the second embodiment operates basically in the same manner as the sensor model 31 in the first embodiment, and therefore a description thereof will be omitted.

[0071] According to this technology, by providing an input image normalization unit 41 (241) in the sensor model 31 (231), it is possible to unify the standard of images input to the sensor model 31, and to process signals from a plurality of rendering systems 21, in other words, image data of different types. Conventionally, there was a one-to-one correspondence between the rendering system 21 and the sensor model 31 (231), but it is now possible to have a multiple-to-one correspondence, and for example, even if the rendering system 21 is changed, the sensor model 31 (231) can process image data from the replaced rendering system 21.

[0072] Even when the rendering system 21 is changed, there is no need to redesign the sensor model 31 (231), and it is possible to provide a highly versatile sensor model 31 (231).

[0073] By providing the distortion reproduction unit 51 or the line shutter information output unit 246, it becomes possible to reproduce the distortion that occurs in an actual image sensor and is specific to the image sensor, thereby enabling a more accurate simulation.

[0074] <Regarding the recording medium> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed on a computer. Here, the computer includes a computer that is built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0075] 10 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. In the computer, a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, and a RAM (Random Access Memory) 2003 are interconnected by a bus 2004. An input / output interface 2005 is also connected to the bus 2004. An input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected to the input / output interface 2005.

[0076] The input unit 2006 includes a keyboard, a mouse, a microphone, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a non-volatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives removable media 2011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0077] In a computer configured as described above, the CPU 2001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004 and executing it.

[0078] The program executed by the computer (CPU 2001) can be provided by being recorded on, for example, a removable medium 2011 such as a package medium. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0079] In the computer, the program can be installed in the storage unit 2008 via the input / output interface 2005 by inserting the removable medium 2011 into the drive 2010. The program can also be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. Alternatively, the program can be installed in advance in the ROM 2002 or the storage unit 2008.

[0080] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0081] In this specification, a system refers to an entire device made up of multiple devices.

[0082] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0083] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0084] The present technology can also be configured as follows. (1) A simulation device comprising: an image input interface that generates an instance, manages an image memory address and an image type in the instance, and determines the image type from the instance; and a processing unit that executes processing corresponding to the image type. (2) The simulation device according to (1), which acquires image data of an image at a predetermined time from a rendering device, and reproduces sensor distortion using the image data of the image at the predetermined time. (3) The simulation device according to (1), which supplies shutter information consisting of an exposure start time and an exposure end time to the rendering device, and acquires image data of the image in which the distortion has been reproduced from the rendering device. (4) The simulation device according to any of (1) to (3), wherein the image type is one of spectral irradiance, RGB (Red, Green, Blue) irradiance, or photon count. (5) The simulation device according to any of (1) to (4), wherein the instance manages an image memory address secured within the simulation device, or manages an image memory address secured outside the simulation device. (6) The simulation device according to (4), wherein the image input interface stores, in the image memory, image data of the spectral irradiance representing one pixel for each pixel, image data of the RGB irradiance representing one pixel for each pixel, or image data of the number of photons representing one pixel for each pixel. (7) A control method, wherein an image input interface that controls input and output of image data generates an instance, manages an address of an image memory and an image type in the instance, determines the image type from the instance, and supplies the image data to a processing unit that executes processing corresponding to the image type.(8) A simulation system comprising a rendering device and a simulation device that performs a simulation using image data from the rendering device, wherein the simulation device generates an instance in response to an instruction from the rendering device, manages the address of an image memory using the instance, stores image data and an image type from the rendering device in the image memory, and determines the type of the stored image from the image memory; and comprises an image input interface; and a processing unit that executes processing corresponding to the type of the image.

[0085] 10 Simulation system, 21 Rendering system, 31 Sensor model, 41 Input image normalization unit, 42 Spectral irradiance processing unit, 43 RGB irradiance processing unit, 44 Photon number processing unit, 45 Pixel model processing unit, 51 Distortion reproduction unit, 52 Pixel model generation unit, 53 Distortion reproduction unit, 54 Pixel model generation unit, 55 Distortion reproduction unit, 56 Pixel model generation unit, 101 Instance, 102 Image memory, 103 Image memory, 231 Sensor model, 241 Input image normalization unit, 242 Spectral irradiance processing unit, 243 RGB irradiance processing unit, 244 Photon number processing unit, 245 Pixel model processing unit, 246 Line shutter information output unit, 252 Pixel model generation unit, 254 Pixel model generation unit, 256 Pixel model generation unit

Claims

1. A simulation device comprising: an image input interface that generates an instance, manages an image memory address and an image type in the instance, and determines the image type from the instance; and a processing unit that executes processing corresponding to the image type.

2. The simulation device according to claim 1, further comprising: acquiring image data of an image at a predetermined time from a rendering device; and reproducing the distortion of the sensor using the image data of the image at the predetermined time.

3. The simulation device according to claim 1, further comprising: supplying a rendering device with shutter information consisting of an exposure start time and an exposure end time; and acquiring image data of an image in which distortion has been reproduced from said rendering device.

4. The simulation device according to claim 1, wherein the type of image is any one of spectral irradiance, RGB (Red, Green, Blue) irradiance, and photon count.

5. The simulation device according to claim 1, wherein the instance manages an address of an image memory secured within the simulation device, or manages an address of an image memory secured outside the simulation device.

6. The simulation device according to claim 4, wherein the image input interface stores in the image memory the image data of the spectral irradiance representing one pixel for each pixel, stores the image data of the RGB irradiance representing one pixel for each pixel, or stores the image data of the number of photons representing one pixel for each pixel.

7. A control method in which an image input interface that controls the input and output of image data generates an instance, manages an image memory address and an image type in the instance, determines the image type from the instance, and supplies the image data to a processing unit that executes processing corresponding to the type.

8. A simulation system comprising: a rendering device; and a simulation device that performs a simulation using image data from the rendering device, wherein the simulation device generates an instance in response to an instruction from the rendering device, manages an address of an image memory in the instance, stores image data and an image type from the rendering device in the image memory, and determines the type of the stored image from the image memory; and comprises an image input interface; and a processing unit that executes processing corresponding to the type of the image.

Images