Information processing method and information processing device

Abstract

This information processing method verifies the consistency of a first image capturing device with a camera model by comparing the characteristics of the first image capturing device and the characteristics of the camera model by using first image data output from the first image capturing device that captures a specific subject and second image data output from the camera model to which two-dimensional input data, which is based on a measurement result obtained by measuring light from the specific subject, is input.

Description

[Technical Field]

[0001] The present technology relates to the technical field of an information processing method and an information processing device for verifying the consistency between a real environment and a simulation environment. [Background technology]

[0002] In the development of in-vehicle camera systems used in autonomous driving and safety support systems, evaluations are carried out using simulation environments to assess safety in various driving environments. For example, the driving environment is reproduced using a CG (Computer Graphics) model, data based on the CG model is input into a camera model that imitates an actual camera, and recognition processing is performed on the image data output from the camera model, and evaluations are carried out using the recognition results. In Patent Document 1 below, input data for a camera model is generated by converting actual image data captured in a real environment into virtual image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2019 / 0171223 Summary of the Invention [Problem to be solved by the invention]

[0004] However, because the simulation process uses various virtual models, such as driving environment models and optical models, the process of verifying consistency with the real environment becomes complicated, which tends to result in high time and human costs. In a simulation using a camera model, since various sensors are used by users, it is necessary to incorporate a model that conforms to each sensor used into the camera model. However, every time the camera model is changed, it is necessary to change the input virtual image data and verify consistency, which leads to increased costs.

[0005] This technology was developed in consideration of these problems, and aims to propose a method for efficiently verifying consistency between the real environment and the simulation environment. [Means for solving the problem]

[0006] The information processing method according to the present technology performs consistency verification between the first imaging device and the camera model by comparing characteristics of the first imaging device and characteristics of the camera model using first image data output from a first imaging device that has captured an image of a specific subject and second image data output from a camera model to which two-dimensional input data based on a measurement result of measuring light from the specific subject is input. the light measurement result is a measurement result for a predetermined area set within the angle of view of the first imaging device, and the two-dimensional input data is data in which the measurement result is arranged in an area corresponding to the predetermined area. It is something. The comparison of the characteristics of the first imaging device and the camera model may be performed, for example, by comparing the first image data with the second image data, or by comparing the recognition results obtained by applying image recognition processing to the first image data with the recognition results obtained by applying image recognition processing to the second image data.

[0007] The information processing method of the present technology performs calibration as a pre-processing step for verifying the consistency between a first imaging device as an in-vehicle camera and a camera model as a simulated camera by adjusting the yaw and pitch directions of the first imaging device so that the center of a first target installed in at least two locations in front of the first imaging device and at the same height as the first imaging device coincides with the center of the angle of view. By performing such calibration, it is possible to verify the consistency between an on-board camera system equipped with an on-board camera and a simulator system equipped with a camera model. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram showing a flow executed in the vehicle-mounted camera system for verifying consistency. [Figure 2] FIG. 10 is a diagram showing a flow executed in a simulator system for consistency verification. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a camera model. [Figure 4] 10 is a flowchart showing the overall flow of consistency verification. [Figure 5] 10 is a flowchart illustrating a specific example of processing for verifying the consistency of a camera model. [Figure 6] FIG. 10 is a schematic diagram showing a state in which a luminance box is being photographed by a first imaging device. [Figure 7] FIG. 10 is a diagram showing an example of a verification area set in RAW data output from a first imaging device. [Figure 8] FIG. 4 is a diagram showing an example of RAW data output from the first imaging device. [Figure 9] FIG. 10 is a diagram showing a state in which a measurement area set in a luminance box is being measured using a spectroradiometer. [Figure 10] FIG. 10 is a diagram showing an example of input data using spectral radiance values ​​as input data to a camera model. [Figure 11] FIG. 10 is a diagram showing an example of RAW data output from a camera model. [Figure 12] FIG. 10 is a diagram illustrating an example of a color chart. [Figure 13] FIG. 10 is a diagram showing an example of a gray chart. [Figure 14] FIG. 1 is a diagram showing an example of a method for measuring spectral radiance values ​​for a color chart or a gray chart. [Figure 15]17 is a flowchart showing a specific example of processing for calibrating the camera posture together with FIG. 16. [Figure 16] 16 is a flowchart showing a specific example of processing for calibrating the camera posture together with FIG. 15. [Figure 17] FIG. 1 is a schematic diagram showing two indicators arranged in front of a vehicle. [Figure 18] FIG. 10 is a schematic diagram showing a state in which indicators are arranged spaced apart in the width direction of the vehicle. [Figure 19] FIG. 10 is an explanatory diagram of calibration of the first imaging device in the roll direction. [Figure 20] 10 is a schematic diagram showing a state in which an index for adjusting the depression angle of the first imaging device is arranged. FIG. [Figure 21] FIG. 10 is a diagram showing an example of a filter included in the first imaging device, and this diagram shows a Bayer array filter. [Figure 22] FIG. 10 is a diagram showing an example of a filter included in the first imaging device, and this diagram shows a filter with an RCCB arrangement. [Figure 23] FIG. 10 is a diagram showing an example of a filter included in the first imaging device, and shows a filter with an RGBIR arrangement. [Figure 24] FIG. 10 is a diagram showing an example of a filter included in the first imaging device, and this diagram shows a complementary color filter. [Figure 25] FIG. 1 is a block diagram of a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order. <1. Consistency verification> <2. Process flow of each system> <3. Consistency verification flow> <4. Camera model consistency verification> 5. Camera system calibration <6. Variations> <7. Computer Equipment> <8. Summary> <9. This Technology>

[0010] <1. Consistency verification> This embodiment will be described with reference to the accompanying drawings. In this embodiment, a consistency verification is performed between an in-vehicle camera system S1 equipped with a first imaging device 1 mounted on a vehicle 100 and a simulator system S2 equipped with a camera model 2 that simulates an actual imaging device.

[0011] In order to realize driving assistance functions such as a collision mitigation braking function and an autonomous driving function of the vehicle 100, it is necessary to verify the control algorithms of the vehicle 100. These control algorithms are improved by actually driving the vehicle 100 under various driving environments, acquiring data and control results obtained from various sensors, and conducting repeated studies.

[0012] However, preparing an actual environment for running the vehicle 100 requires a great deal of cost, such as securing a place. For this reason, algorithms are generally evaluated using simulations known as MBD (Model Based Design).

[0013] MBD uses a driving environment model that simulates an actual driving environment, a sensor model that simulates an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) or CCD (Charge Coupled Device) that the first imaging device 1 has, and an optical model that simulates optical components such as lenses that the first imaging device 1 has.

[0014] Therefore, in order to properly verify the control algorithm using MBD, It is necessary to make the various models provided by the system more consistent with the actual models.

[0015] In this embodiment, a method for efficiently verifying the consistency between these various models and the actual environment will be described. Specifically, the consistency between the vehicle-mounted camera system S1 and the simulator system S2 will be verified.

[0016] <2. Process flow of each system> FIG. 1 shows the flow of the process executed in the vehicle-mounted camera system S1 for verifying the consistency, and FIG. 2 shows the flow of the process executed in the simulator system S2.

[0017] The vehicle-mounted camera system S1 captures an image of an actual driving environment scene A1 using a first imaging device 1. The first imaging device 1 includes optical members such as a lens, an image sensor, and the like. A signal obtained by imaging by the first imaging device 1 is input to the camera signal processing model A2 as an imaging signal (RAW data).

[0018] The camera signal processing model A2 performs processes such as generating an RGB (Red, Green, Blue) image, adjusting white balance, sharpness, and contrast. The captured image data output from the camera signal processing model A2 is input to the recognition model A3. The captured image data output from the camera signal processing model A2 is referred to as a first image G1.

[0019] The recognition model A3 performs a process of identifying and labeling the captured subject based on the input image data. That is, the recognition model A3 performs a process of detecting the subject captured in the image. The detected subject may be, for example, a pedestrian, a traffic light, a sign, or a vehicle. The detection result in the recognition model A3 is input to the subsequent integrated control model A4.

[0020] The integrated control model A4 uses the recognition results based on the image data to control the vehicle 100. This realizes a collision mitigation brake function, an ACC (Adaptive Cruise Control) function, and various warning functions.

[0021] On the other hand, in the simulator system S2, in order to simulate an actual driving environment scene A1, environment setting B1, scenario generation B2, and rendering B3 are performed.

[0022] In the environment setting B1, map information corresponding to the actual driving environment scene A1 in which the vehicle 100 actually drives, a data set simulating other vehicles other than the vehicle 100, pedestrians, signs, traffic lights, road surfaces, and other objects that are located on the driving route, etc. are set. These data sets include shape information such as polygons used in ray tracing in the subsequent rendering B3, as well as color information, material information, and spectral reflectance information.

[0023] In the subsequent scenario generation B2, a driving plan is set that simulates the driving behavior of the vehicle 100. The driving plan includes not only the route that the vehicle 100 will travel, but also operations such as steering and accelerator operation while driving. This can also be said as the behavior of the vehicle 100 based on the driving operations.

[0024] The driving plan generated in scenario generation B2 is input to rendering B3, which is the subsequent stage. In rendering B3, a 3DCG (3-dimensional CG) model is generated by performing a rendering process using ray tracing. The output result of the rendering process is input to camera model 2.

[0025] Here, the output data of the rendering process is the spectral radiance value for each pixel expanded in a two-dimensional array corresponding to the two-dimensional pixel array of the image sensor model provided in the camera model 2. Furthermore, the spectral radiance value is output at a fixed time interval corresponding to the frame rate of the first image capturing device 1. For example, if the first image capturing device 1, which is the target to be simulated by the camera model 2, captures images at 30 fps (frames per second), the spectral radiance value for each pixel expanded in a two-dimensional array is output 30 times per second.

[0026] The camera model 2 is a model for simulating the imaging function of the first imaging device 1 of the vehicle-mounted camera system S1, and as shown in FIG. 3, is configured to include an optical model B7 and a sensor model B8.

[0027] The optical model B7 is a model that imitates various lens systems, shutter mechanisms, IR (Infrared) cut filters, etc., which are optical members of the first imaging device 1, and performs processing to calculate the influence of the optical members on the spectral radiance value input to the camera model 2 and convert it into spectral irradiance for each pixel. Specifically, the optical model B7 performs processing such as projection correction, aperture correction, shading correction, and IR cut filter correction.

[0028] The spectral irradiance for each pixel is input to a sensor model B8 that simulates an image sensor included in the first imaging device 1. The sensor model B8 is configured with a filter model that simulates various optical filters such as color filters included in the image sensor, a pixel array model that simulates the light receiving operation and readout operation in the pixel unit, and the like, and calculates the influence of the color filter and pixel unit on the input spectral irradiance and outputs the result as RAW data for each pixel to the subsequent camera signal processing model B4. Specifically, the sensor model B8 performs color filter correction processing, photoelectric conversion processing, AD (Analog to Digital) conversion processing, etc.

[0029] Here, the RAW data output from the first imaging device 1 of the in-vehicle camera system S1 and the RAW data output from the camera model 2 of the simulator system S2 have the same data format. Therefore, the camera signal processing model A2 of the in-vehicle camera system S1 to which the RAW data is input can be the same as the camera signal processing model B4 of the simulator system S2. The captured image data output from the camera signal processing model B4 will be referred to as the second image G2.

[0030] Similarly, the recognition model A3 of the in-vehicle camera system S1 and the recognition model B5 of the simulator system S2 are the same, and the integrated control model A4 of the in-vehicle camera system S1 and the integrated control model B6 of the simulator system S2 are the same.

[0031] Therefore, if the detection results input to the integrated control model A4 and the integrated control model B6 are the same, the data output from the integrated control model A4 and the integrated control model B6 will also match.

[0032] Therefore, the flow from the environment setting B1 to the recognition model B5 in the simulator system S2 (the dashed line in Fig. 2) needs to properly simulate the flow from the actual driving environment scene A1 to the recognition model A3 in the in-vehicle camera system S1 (the dashed line in Fig. 1). Verification of this point is carried out by verifying the consistency between the in-vehicle camera system S1 and the simulator system S2.

[0033] In this embodiment, the consistency verification is carried out by determining whether or not the intermediate data and the like in each system match. Specifically, consistency verification is performed using RAW data output from the first imaging device 1 in the in-vehicle camera system S1, RAW data output from the camera model 2 in the simulator system S2, detection results output from the recognition model A3 in the in-vehicle camera system S1, and detection results output from the recognition model B5 in the simulator system S2.

[0034] By performing consistency verification in this manner, it is possible to perform verification in the simulator system S2 that is equivalent to verifying the algorithm in the vehicle-mounted camera system S1 equipped with the first imaging device 1 as an actual imaging device.

[0035] <3. Consistency verification flow> The flow of the consistency verification will be described with reference to Fig. 4. Note that the device that performs each process for the consistency verification may be a computer device included in the simulator system S2, or may be a device separate from the computer device included in the simulator system S2. The computer device is configured to include an arithmetic processing unit including a CPU (Central Processing Unit) and the like in order to execute each process for the consistency verification.

[0036] 4, the arithmetic processing unit of the computer device verifies whether the camera model 2 appropriately simulates the optical members and image sensor of the actual first imaging device 1. That is, the arithmetic processing unit verifies the consistency between the camera model 2 and the first imaging device 1.

[0037] The verification of the consistency between the camera model 2 and the first imaging device 1 determines whether the RAW data output from the camera model 2 matches the RAW data output from the first imaging device 1. Since it is difficult to completely match the RAW data output from the first imaging device 1, which is a real device, with the RAW data output from the camera model 2 using 3DCG, in this embodiment, the verification of consistency is performed using statistical data on each piece of RAW data. Specific processing details for verifying the consistency of the RAW data will be described later.

[0038] In step S102, the calculation processing unit performs a determination process based on the comparison result between the RAW data output from the camera model 2 and the RAW data output from the first imaging device 1. If the difference between the RAW data does not fall within a predetermined range, that is, if it is determined that the consistency verification has failed, the calculation processing unit performs a factor analysis and a model correction process in step S103.

[0039] In the process of step S103, a factor analysis is performed, and the results of the factor analysis are presented to the operator, and a model to be modified is presented, etc. Also, a process of changing and overwriting variables given to the model may be performed in step S103. Alternatively, the calculation processing unit may perform a process of presenting data to be used for the factor analysis to the operator in step S103, and the operator may perform the factor analysis based on the presented data.

[0040] After the factor analysis and model correction process is completed, the processing unit returns to step S101 and verifies the consistency between the camera model 2 and the first image capturing device 1.

[0041] On the other hand, if it is determined in step S102 that the consistency verification is successful, i.e., if it is determined that the difference between the RAW data is within a predetermined range, the calculation processing unit performs consistency verification using the detection results output from the recognition model A3 of the in-vehicle camera system S1 and the recognition model B5 of the simulator system S2.

[0042] Specifically, in step S104, the calculation processing unit calibrates the attitude of the first imaging device 1 and the attitude of the camera model 2. In the processing of step S104, the attitude of the first imaging device 1 is calibrated so that the first imaging device 1 captures an image in a predetermined direction. Then, the optical axis direction set in the camera model 2 is changed to coincide with the optical axis direction of the first imaging device 1. The specific processing content of step S104 will be described later.

[0043] In step S105, the processing unit verifies the consistency of the camera system in the driving environment. The camera system consistency verification verifies the consistency between the dashed line portion of the in-vehicle camera system S1 shown in Fig. 1 and the dashed line portion of the simulator system S2 shown in Fig. 3. Specifically, a process is executed to check whether the subject detection result output from the recognition model A3 of the in-vehicle camera system S1 matches the subject detection result output from the recognition model B5 of the simulator system S2.

[0044] The detection result is, for example, data in which information specifying a pixel region for the first image G1 input to the recognition model A3 is linked to label information that is the result of classifying the subject captured there. Specifically, a label "car" is linked to a certain predetermined pixel region.

[0045] The detection result may include multiple pairs of pixel region information and label information for one first image G1.

[0046] The detection result for the simulator system S2 is similar. That is, the detection result is obtained by linking information specifying the pixel area of ​​the second image G2 input to the recognition model B5 with label information about the captured subject.

[0047] The detection results may include information other than labels. For example, they may include brightness information for pixels or pixel regions. Specifically, information for identifying a pixel region, label information for identifying an object captured in that region, and brightness information for the object, such as maximum, minimum, average, and variance brightness values, may be linked together. If the objects captured in a specific region in the first image G1 and the second image G2 are classified into the same category and the brightness of the captured objects tends to be similar, the detection results obtained by the simulator system S2 can be used to verify the control algorithm in the same way as with a real device. If there are objects detected only by one recognition model, if the classification results of the detected objects are different, or if the brightness trends of the objects differ, it can be determined that the detection results obtained by the simulator system S2 cannot be used to verify the control algorithm in the same way as with a real device.

[0048] In step S106, the calculation processing unit performs a determination process based on the comparison result between the detection result output from recognition model A3 and the detection result output from recognition model B5. If the two detection results are significantly different, that is, if it is determined that the consistency verification has failed, the calculation processing unit performs a factor analysis and model correction process in step S107. However, since the consistency between the camera model 2 and the first imaging device 1 is ensured in the processing of steps S101 and S102, in step S107, factor analysis and model correction are performed for parts other than those indicated by dashed lines in both Figures 1 and 2.

[0049] In the processing of step S106, similar to the processing of step S103, a factor analysis may be performed, the results of the factor analysis may be presented to the worker, and the model to be modified may be presented, or the variables given to the model may be changed and overwritten, or the data to be used in the factor analysis may be presented to the worker to encourage the worker to perform the factor analysis.

[0050] After the factor analysis and model correction processing in step S107 are completed, the processing unit returns to step S105 and performs consistency verification of the camera system in the driving environment.

[0051] <4. Camera model consistency verification> The process flow for verifying the consistency of the camera model will be described with reference to Fig. 5. The consistency verification flow shown in Fig. 5 shows the processes of steps S101, S102, and S103 in Fig. 4, and particularly shows the specific process executed in step S101.

[0052] Note that, although an example will be described in which the aforementioned arithmetic processing unit executes each process shown in Figure 5, this is not limiting, and each process shown in Figure 5 may be executed by cooperation between the arithmetic processing unit of the computer device and the control unit of the first imaging device 1, or each process may be executed by a system configured including other devices.

[0053] The arithmetic processing unit determines whether or not an instruction to start shooting has been received in step S201 of Fig. 5. The instruction to start shooting may be input by an operator to a computer device having an arithmetic processing unit, or may be input from the first imaging device 1 that has detected that a subject to be photographed has been positioned at a predetermined position within the angle of view of the first imaging device 1.

[0054] The arithmetic processing unit repeatedly executes the process of step S201 until it receives an instruction to start imaging. Note that the instruction to start imaging is given by an operator when, for example, predetermined conditions are met. The predetermined conditions being met means, for example, that the placement of the measurement subject within the angle of view of the first imaging device 1 has been completed.

[0055] A specific description will be given with reference to Fig. 6. Fig. 6 shows a state in which the luminance box 3 is placed within the angle of view of the first imaging device 1, and in this state it is possible to capture an image of the luminance box 3 as a measurement subject. After placing the luminance box 3 at a predetermined position in front of the first imaging device 1, the operator instructs the processing unit to start capturing images.

[0056] The luminance box 3 has a light emitter disposed therein and one surface formed as a diffusion surface 3a that transmits and diffuses light emitted from the light emitter disposed therein. The luminance box 3 is disposed so that the diffusion surface 3a faces the first imaging device 1.

[0057] Returning to the explanation of Figure 5. In step S202, the arithmetic processing unit transmits an image capturing operation instruction to the first image capturing device 1. As a result, the image capturing operation in the first image capturing device 1 is executed.

[0058] In step S203, the arithmetic processing unit acquires the RAW data output from the first imaging device 1. In the RAW data, a predetermined area is set as a verification area Ar.

[0059] Next, in step S204, the arithmetic processing unit performs statistical processing for each verification area Ar set as a predetermined region in the RAW data. A specific description will be given with reference to Fig. 7. The RAW data is data corresponding to the two-dimensional arrangement of pixels formed on the image sensor of the first imaging device 1. That is, as shown in Fig. 7, the RAW data can be regarded as two-dimensional data. In the two-dimensional RAW data, a verification area Ar1 is set in the center, a verification area Ar2 is set near the upper left corner, a verification area Ar3 is set near the upper right corner, a verification area Ar4 is set near the lower left corner, and a verification area Ar5 is set near the lower right corner.

[0060] In the statistical processing of step S204, the average value, variance, etc. are calculated for each of the verification areas Ar1 to Ar5. The size of each verification area Ar is, for example, 100 pixels in both length and width.

[0061] Here, if the image sensor is equipped with a Bayer array color filter or the like, the RAW data includes data Dr indicating the intensity of the red light component of the light received for each pixel, data Dg indicating the intensity of the green light component, and data Db indicating the intensity of the blue light component. Specifically, the RAW data is two-dimensional data in which the data Dr, Dg, and Db are expanded in a predetermined pattern, as shown in Fig. 8. Therefore, for the verification areas Ar1 to Ar5, the data Dr, Dg, and Db are also expanded vertically and horizontally to form two-dimensional data.

[0062] Taking the verification area Ar1 as an example, the statistical processing in step S204 calculates the mean value and variance for the data Dr contained in the verification area Ar1, calculates the mean value and variance for the data Dg contained in the verification area Ar1, and calculates the mean value and variance for the data Db contained in the verification area Ar1.

[0063] That is, when calculating the average value and variance, six statistical data are calculated for one verification area Ar.

[0064] Returning to the explanation of Fig. 5, by calculating the statistical data for each verification area Ar and for each color pixel, the arithmetic processing unit completes the acquisition of data to be used for the consistency verification for the first imaging device 1 as the actual device.

[0065] Next, the arithmetic processing unit executes the processes from step S205 onwards to start acquiring data to be used for consistency verification for the camera model 2 of the simulator system S2. Specifically, in step S205, the arithmetic processing unit determines whether or not measurement preparation is complete.

[0066] The state in which preparation for measurement is complete refers to a state in which the spectroradiometer 4 is placed in a position facing the diffusion surface 3a of the luminance box 3, as shown in Fig. 9, for example. In other words, it is determined that preparation for measurement is complete when a state in which the spectroradiometer 4 can be used to measure the spectral radiance value of a specific region has been achieved. The process of step S205 is repeated until preparation for measurement is complete.

[0067] If it is determined that the measurement preparation is complete, or if a user operation indicating that the measurement preparation is complete is received, the calculation processing unit acquires the measured spectral radiance value in step S206. Note that an instruction to start measurement may be issued to the spectroradiometer 4 before step S206.

[0068] The spectral radiance meter 4 does not measure the spectral radiance value of the entire radiance box 3, but measures the spectral radiance value of a narrow area having a small solid angle. Therefore, in order to obtain the spectral radiance value to be compared with the five verification areas Ar set in the center and near each corner in Fig. 7, it is necessary to measure the spectral radiance value of the measurement area Br set at a position corresponding to the verification area Ar.

[0069] 9, the calculation processing unit performs the process of step S206 with the optical axis of the spectroradiometer 4 aligned with the center of the measurement area Br1, and then waits again in step S205 until measurement preparation is complete. Then, in response to the operator performing an operation to align the optical axis of the spectroradiometer 4 with the center of the measurement area Br2, the calculation processing unit performs the process of step S206 again. By repeating the processes of steps S205 and S206 in this manner, the spectral radiance value for each measurement area Br is obtained.

[0070] As described above, the measurement area Br1 is provided at a position corresponding to the verification area Ar. That is, the light passing through the center of the measurement area Br1 on the diffusing surface 3a affects the RAW data located at the center of the verification area Ar1 in the RAW data shown in Fig. 7, and also affects the spectral radiance value measured for the measurement area Br1.

[0071] 5, the calculation processing unit uses the acquired spectral radiance values ​​to generate input data for the camera model 2. The generated data is two-dimensional data corresponding to the pixel array of the sensor model B8 included in the camera model 2. An example is shown in Fig. 10. In the input data, the acquired spectral radiance values ​​are placed in corresponding areas Cr1 to Cr5 corresponding to each measurement area Br. Specifically, the spectral radiance value obtained from measurement area Br1 is placed in corresponding area Cr1, the spectral radiance value obtained from measurement area Br2 is placed in corresponding area Cr2, the spectral radiance value obtained from measurement area Br3 is placed in corresponding area Cr3, the spectral radiance value obtained from measurement area Br4 is placed in corresponding area Cr4, and the spectral radiance value obtained from measurement area Br5 is placed in corresponding area Cr5.

[0072] In addition, dummy data is used in areas other than the corresponding area Cr in the input data using spectral radiance values. The area where the dummy data is placed is an area that is not used for consistency verification. The dummy data may be a zero value or any other value.

[0073] 5, the calculation processing unit acquires the RAW data output from the camera model 2. The RAW data output from the camera model 2 is obtained by converting the spectral radiance value input to the camera model 2 into a spectral irradiance through calculation by the optical model B7, and then further converting the spectral irradiance through calculation by the sensor model B8.

[0074] A predetermined area in the RAW data is set as a verification area Ar'. For example, as shown in Fig. 11, the area of ​​RAW data output from a pixel area to which spectral radiance values ​​arranged in corresponding area Cr1 are input is set as verification area Ar'1. Similarly, the area of ​​RAW data output from a pixel area to which spectral radiance values ​​arranged in corresponding area Cr2 are input is set as verification area Ar'2, the area of ​​RAW data output from a pixel area to which spectral radiance values ​​arranged in corresponding area Cr3 are input is set as verification area Ar'3, the area of ​​RAW data output from a pixel area to which spectral radiance values ​​arranged in corresponding area Cr4 are input is set as verification area Ar'4, and the area of ​​RAW data output from a pixel area to which spectral radiance values ​​arranged in corresponding area Cr5 are input is set as verification area Ar'5.

[0075] 5, the calculation processing unit performs statistical processing for each verification area Ar'. For example, the average value, variance, etc. are calculated for each of verification areas Ar'1 to Ar'5. The size of verification area Ar'1 is set to be the same as the size of verification area Ar in FIG. Furthermore, the RAW data output from the sensor model B8 is two-dimensional data in which the data Dr, Dg, and Db are expanded, similar to the RAW data output from the first imaging device 1 (see FIG. 8). In the statistical processing of step S209, the average value and variance are calculated for each of the data Dr, data Dg, and data Db for one verification area Ar'. That is, when calculating the average value and variance, six statistical data are calculated for one verification area Ar'.

[0076] In step S210, the calculation processing unit performs a comparison process for the verification area Ar and the verification area Ar'. Specifically, the calculation processing unit performs a process of comparing the statistical data calculated for the verification area Ar1 in step S204 with the statistical data calculated for the corresponding verification area Ar' in step S209.

[0077] If the statistical data for all verification areas Ar and verification areas Ar' match or the difference is small, it can be determined that the camera model 2 is able to properly simulate the first imaging device 1.

[0078] In step S102, the calculation processing unit performs branching processing based on the comparison result. Specifically, if it is determined that the differences between all statistical data are small as a result of the comparison processing in step S210, it is determined that the match between the first image capture device 1 and the camera model 2 is ensured, that is, the match verification is passed, and the series of processing shown in FIG. 5 is terminated.

[0079] On the other hand, if it is determined that the difference is greater than the threshold value for at least some of the statistical data, it is determined that the match between the first image capture device 1 and the camera model 2 is insufficient, and the process returns to step S201 after performing factor analysis and correcting the various models in step S103. That is, the series of processes shown in FIG. 5 are repeated until the match between the first image capture device 1 and the camera model 2 is ensured.

[0080] The threshold used in the branching process in step S102 may be different for each type of statistical data. Specifically, the threshold used when comparing mean values ​​may be different from the threshold used when comparing variances.

[0081] Furthermore, in the example of verifying the consistency between the first image capturing device 1 and the camera model 2 described above, the brightness box 3 is used, but other things may also be used. For example, instead of using the brightness box 3 as the measurement subject, a color chart 5 as shown in FIG. 12 or a gray chart 6 as shown in FIG. 13 may be used.

[0082] 14, a light source 7 may be disposed to irradiate the color chart 5 or gray chart 6 with light. That is, light emitted from the light source 7 may be reflected by the color chart 5 or gray chart 6 and input to the first image capturing device 1 or the spectroradiometer 4.

[0083] 5. Camera system calibration An example of processing executed by the arithmetic processing unit to calibrate the camera attitude in step S104, which is a preliminary preparation for verifying the consistency of the camera system in the driving environment in step S105 in Fig. 4, will be described with reference to Fig. 15 and Fig. 16. Note that the connection between the flows in Fig. 15 and Fig. 16 is indicated by a connector C1.

[0084] The arithmetic processing unit that executes the series of processes shown in Figures 15 and 16 may be the same as the arithmetic processing unit that executes the processes shown in Figure 4, or may be a different processing device.

[0085] First, before the calculation processing unit executes the process of step S301, an operator places index A and index B at predetermined positions in front of the vehicle 100. Specifically, as shown in Fig. 17 , index A is placed so that its horizontal distance from the first imaging device 1 is distance L1 and its height from the ground coincides with the height Hc of the first imaging device 1 when mounted on the vehicle 100.

[0086] Next, as shown in FIG. 17, the worker installs the indicator B so that the horizontal distance from the first imaging device 1 is a distance L2 that is longer than the distance L1, and so that the height from the ground is a height Hc.

[0087] 17, the arithmetic processing unit adjusts the yaw and pitch directions of the first imaging device 1 so that the center of index A, the center of index B, and the center of the angle of view coincide with each other in step S301 of Fig. 15. This adjustment is realized, for example, by the arithmetic processing unit transmitting a control signal to the first imaging device 1 to adjust the imaging direction of the first imaging device 1.

[0088] Note that, instead of transmitting a control signal to the first imaging device 1, the arithmetic processing unit may transmit the control signal to a holding device that holds the first imaging device 1 in a state in which the imaging direction of the first imaging device 1 can be adjusted, or to a control device that controls the imaging direction of the first imaging device 1. This also applies to other control signals described later.

[0089] By adjusting the yaw direction and pitch direction so that the center of index A and the center of index B overlap the center of the angle of view, the optical axis of the first imaging device 1 can be adjusted horizontally.

[0090] Next, before the calculation processing unit executes the process of step S302, the worker sets up an index C at a predetermined position. Specifically, as shown in Fig. 18, the worker sets up the index C at a position moved parallel to the vehicle width direction of the vehicle 100 from the position where the index A is set up. At this time, the height of the center of the index C is set to the height Hc.

[0091] In step S302 of FIG. 15, the calculation processing unit transmits a control signal for adjusting the roll direction of the first imaging device 1 so that the center of the index C is positioned on a horizontal line passing through the center of the angle of view of the first imaging device 1. Specifically, as shown in FIG. 19, the center of index A is imaged at the center of the angle of view of the first imaging device 1, and the roll direction is adjusted so that the center of index C is imaged at the vertical intermediate position in the angle of view.

[0092] 15, the calculation processing unit determines whether or not to provide a depression angle in the attitude of the first imaging device 1. Whether or not to provide a depression angle may be set by an operator, or may be set automatically depending on a subject, such as a pedestrian or a preceding vehicle, that is captured in the angle of view of the first imaging device 1.

[0093] When providing a depression angle, an operator places an index D at a predetermined position, as shown in Fig. 20, for example. The distance between the index D and the first imaging device 1 is set to L3, and the height of the center position is set to Hd. The distance L3 and the height Hd may be set to any values.

[0094] 15, the processor transmits a control signal for adjusting the pitch direction of the first imaging device 1 so that the center of the index D is positioned on a horizontal line passing through the center of the angle of view of the first imaging device 1. At this time, adjustments are not made to the yaw direction or roll direction. This makes it possible to adjust the imaging direction of the first imaging device 1 without changing the yaw direction or roll direction that have already been set.

[0095] The arithmetic processing unit performs the processes from step S305 onward to calibrate the camera model 2 for the simulator system S2. 15, the calculation processing unit sets indexes A' and B' at predetermined positions on the environment map of the simulator system S2. The set positions of indexes A' and B' are based on the positional relationship of indexes A and B with respect to the first image capturing device 1. That is, the positional relationship between the camera model 2 and index A' is the same as the positional relationship between the first image capturing device 1 and index A, and the positional relationship between the camera model 2 and index B' is the same as the positional relationship between the first image capturing device 1 and index B. Although the distance L1 and the distance L2 may be changed, the indicators A' and B' are set so that the distances from the vehicle position on the environment map are different.

[0096] In step S306 of FIG. 15, the calculation processing unit adjusts the yaw and pitch directions of the camera model 2 so that the centers of the indexes A' and B' and the center of the angle of view of the camera model 2 overlap.

[0097] 16, the calculation processing unit sets an index C' at a predetermined position on the environment map of the simulator system S2. Specifically, the calculation processing unit sets the center position of the index C so that the positional relationship between the index A and the index C coincides with the positional relationship between the index A' and the index C'.

[0098] In step S308, the calculation processing unit adjusts the roll direction of the camera model 2 so that the center of the index C′ is positioned on a horizontal line passing through the center of the angle of view of the camera model 2.

[0099] In step S309, the calculation processing unit determines whether or not to set a depression angle in the attitude of the camera model 2. If a depression angle is set in the attitude of the first image capturing device 1, a depression angle is also set in the attitude of the camera model 2 in order to simulate a similar state.

[0100] If a depression angle is to be set, the processing unit sets an index D' at a predetermined position on the environment map of the simulator system S2 in step S310. The position and height of the index D' relative to the camera model 2 are set to match the position and height of the index D relative to the first image capturing device 1.

[0101] Next, in step S311, the calculation processing unit adjusts the pitch direction of the camera model 2 so that the center of the index D' is positioned on a horizontal line passing through the center of the angle of view of the camera model 2. At this time, adjustments are not made to the yaw direction or roll direction. This makes it possible to adjust the imaging direction of the camera model 2 without changing the yaw direction or roll direction that has already been set.

[0102] By performing calibration so that the imaging direction of the camera model 2 coincides with the imaging direction of the first imaging device 1, a camera model 2 that properly simulates the first imaging device 1 can be prepared.

[0103] By appropriately calibrating the first imaging device 1 and the camera model 2 as shown in FIGS. 15 and 16, it becomes possible to appropriately perform the consistency verification in step S105 of FIG.

[0104] <6. Variations> In the above example, the first imaging device 1 is provided with a Bayer array color filter (see FIG. 21), and the camera model 2 has a sensor model B8 that imitates the color filter. However, the filter provided in the first imaging device 1 may be one other than a Bayer array filter.

[0105] Specifically, the first imaging device 1 may be equipped with an RCCB array filter (see FIG. 22) in which filters corresponding to R (Red), C (Clear), and B (Blue) are arranged, an RGBIR array filter (see FIG. 23) in which filters corresponding to R (Red), G (Green), B (Blue), and IR (Infrared) are arranged, or a complementary color filter (see FIG. 24) in which filters corresponding to Cy (Cyan), Ye (Yellow), G (Green), and Mg (Magenta) are arranged. In that case, the sensor model B8 of the camera model 2 is also equipped with an RCCB array filter model, an RGBIR array filter model, or a complementary color filter model.

[0106] In the examples shown in FIGS. 7 and 9 to 11, five verification areas Ar, five measurement areas Br, five corresponding areas Cr, and five verification areas Ar' are set in predetermined positions within the field of view or the image. However, this is merely an example, and other setting examples are also possible. For example, if there is no need to check the consistency of peripheral areas within the field of view or the image, only one predetermined area may be set in the center of each area. Specifically, a verification area Ar1 may be set as the verification area Ar, a measurement area Br1 may be set as the measurement area Br, a corresponding area Cr1 may be set as the corresponding area Cr, and a verification area Ar'1 may be set as the verification area Ar'.

[0107] Alternatively, when checking the match only in the peripheral area, a predetermined area may be set in each area around the corner. Also, six or more predetermined areas may be set. For example, the angle of view may be divided into three in the vertical direction and three in the horizontal direction, for a total of nine areas, and a verification area Ar may be set in the center of each area.

[0108] <7. Computer Equipment> The configuration of a computer device including a processing unit that executes the processes shown in the above-mentioned FIGS. 4, 5, 15, and 16 will be described with reference to FIG.

[0109] The CPU 71 of the computer device functions as an arithmetic processing unit that performs the various processes described above, and executes the various processes according to programs stored in the ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from the storage unit 79 to the RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute the various processes, as appropriate. The CPU 71, ROM 72, RAM 73, and nonvolatile memory unit 74 are interconnected via a bus 83. To this bus 83, an input / output interface (I / F) 75 is also connected.

[0110] The input / output interface 75 is connected to an input unit 76 that includes an operator and an operation device. For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0111] Furthermore, the input / output interface 75 is connected integrally or separately to a display unit 77 made up of an LCD or organic EL panel or the like, and an audio output unit 78 made up of a speaker or the like. The display unit 77 is a display unit that displays various information, and is configured, for example, by a display device provided in the housing of the computer device, or a separate display device connected to the computer device. The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., i.e., GUI (Graphical User Interface), based on instructions from the CPU 71.

[0112] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like.

[0113] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0114] A drive 81 is also connected to the input / output interface 75 as required, and a removable storage medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 81 can read data files such as programs used in various processes from a removable storage medium 82. The read data files are stored in a storage unit 79, and images and sounds contained in the data files are output on a display unit 77 and an audio output unit 78. Furthermore, the computer programs and the like read from the removable storage medium 82 are installed in the storage unit 79 as needed.

[0115] In this computer device, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0116] The CPU 71 performs processing operations based on various programs, thereby executing information processing and communication processing required for an information processing device equipped with the above-described arithmetic processing unit. The information processing device is not limited to being configured with a single computer device as shown in Fig. 2, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0117] <8. Summary> As described in the above examples, the information processing method of the present technology is a method in which a computer device (for example, an information processing device equipped with an arithmetic processing unit such as a CPU 71) performs consistency verification between the first imaging device 1 and the camera model 2 by comparing the characteristics of the first imaging device 1 with the characteristics of the camera model 2 using a first image G1 output from the first imaging device 1 that captures an image of a specific subject (such as the above-mentioned brightness box, each chart, or a subject or preceding vehicle in an actual driving environment scene), and a second image G2 output from the camera model 2 to which two-dimensional input data (for example, spectral radiance values) based on the measurement results of measuring light from the specific subject is input. The comparison of the characteristics of the first imaging device 1 and the camera model 2 may be performed, for example, by comparing the first image G1 and the second image G2, or by comparing the recognition result (label information) obtained by applying image recognition processing to the first image G1 with the recognition result obtained by applying image recognition processing to the second image G2. By verifying the match between the first imaging device 1 and the camera model 2 through such a comparison process, it becomes possible to appropriately perform a simulation using the camera model 2. Note that if the first image G1 and the second image G2 match, the recognition results of the image recognition process for each image will be the same, and it is possible to appropriately simulate a real environment through a simulation using the camera model 2. However, since the data input to the camera model 2 is based solely on the measurement results (such as spectral radiance values ​​and spectral irradiance) of light from a subject, it is difficult to achieve a perfect match between the first image G1 and the second image G2. Therefore, it is also possible to compare statistical data obtained from the first image G1 with similar statistical data obtained from the second image G2, and determine that a match has been achieved if the difference between them is small. This facilitates the verification of match and reduces various costs. Alternatively, as described above, the recognition result obtained by applying image recognition processing to the first image G1 may be compared with the recognition result for the second image G2. Even if the first image G1 and the second image G2 differ slightly, as long as the recognition result is the same, subsequent control can be performed correctly. In other words, if the first image capture device 1 and the camera model 2 are matched to the extent that the same recognition result can be output, automatic driving control and driving assistance control can be performed appropriately. Furthermore, ensuring the first image capture device and the camera model are matched to the extent that the same recognition result can be output means that excessive match between the first image capture device and the camera model is not required, and therefore the camera model can be verified efficiently.

[0118] As described with reference to FIG. 9 and the like, the input data input to the camera model 2 may be the spectral radiance value of light from a specific subject. By inputting two-dimensional spectral radiance values ​​into the camera model 2, the camera model 2 can output a signal similar to that of the actual first image capturing device 1. This allows for proper consistency verification.

[0119] As described with reference to FIG. 3 etc., the camera model may include an optical model B7 and an image sensor model (sensor model B8). For example, the optical model B7 is a model that imitates various lenses, projection correction, aperture correction, shading correction, IR cut filters, etc. that are provided in the first imaging device 1. The image sensor model is a model that imitates color filters provided in each pixel, photoelectric conversion processing, AD (Analog to Digital) conversion processing, etc. In this way, each model is configured to include various corrections and processes that are incorporated in the first imaging device 1, which is an actual imaging device, so that a camera model 2 can be constructed that appropriately imitates the first imaging device 1. Furthermore, it is possible to easily change the camera model 2, which becomes necessary when some of these elements in the actual first imaging device 1 are changed.

[0120] As explained with reference to Figures 7 and 10, the measurement results of measuring light from the subject are taken as measurement results for a predetermined area (area corresponding to the verification area Ar) set within the angle of view of the first imaging device 1, and the two-dimensional input data may have the measurement results placed in an area (corresponding area Cr) corresponding to the verification area Ar. This eliminates the need to generate input data corresponding to all subjects captured within the angle of view of the first imaging device 1. Therefore, the cost required to generate two-dimensional input data can be reduced.

[0121] As explained with reference to Figures 7 and 11, consistency verification may be performed by comparing a pixel area (verification area Ar) corresponding to a specified area with an area (verification area Ar') corresponding to the pixel area (verification area Ar) in the second image G2. Consistency verification can be performed simply by comparing data (statistical data) output from a portion of the pixel area. Therefore, the amount of calculation required for matching verification can be reduced. It is also possible to configure the system so that the verification area Ar in the RAW data is compared with the data of the corresponding verification area Ar', and if a certain degree of agreement is found, image recognition processing and a comparison of the recognition results are performed. In this case, if it is determined that agreement between the first image capture device 1 and the camera model 2 cannot be ensured at the time of comparing the data for the verification areas Ar and Ar', there is no need to perform image recognition processing, thereby reducing the processing load.

[0122] As described with reference to FIG. 10 etc., two-dimensional input data (input data using spectral radiance values) may have dummy data placed in an area other than the area corresponding to the specified area (corresponding area Cr). This eliminates the need to use the measurement results to generate data that is to be placed in an area other than the area corresponding to the predetermined area. Therefore, the time cost and calculation cost for generating two-dimensional input data can be reduced.

[0123] As described with reference to Figures 3 and 7, the image sensor model (sensor model B8) may have a filter model, and consistency verification may be performed by performing statistical processing for each pixel group for each filter type in the second image G2. This allows for verification of consistency in accordance with the configuration of the actual first imaging device 1 equipped with a color filter and the like. Therefore, it is possible to appropriately perform consistency verification.

[0124] As explained with reference to Figure 8, etc., consistency verification may be performed by calculating the mean value and variance for each pixel group (i.e., for each Dr, Dg, or Db) through statistical processing, and comparing the mean value and variance with the respective threshold values. By using the average value, variance, etc., it is possible to verify the consistency with simple processing, thereby reducing the amount of calculation required to verify the consistency. Furthermore, by using the average value, variance, etc., it is possible to prevent erroneous determination due to noise, etc.

[0125] As described with reference to each of Figures 21 to 24, the filter model may be a model that employs at least one of a Bayer array, an RCCB array, an RGBIR array, and an array using a complementary color filter. By using various models as filter models, it is possible to verify the consistency of the camera model 2 according to the user's needs. Therefore, it is possible to carry out flexible consistency verification in accordance with various requirements, thereby improving convenience.

[0126] As explained with reference to the figures in the above-mentioned examples, the information processing method of the present technology is a pre-processing for verifying the consistency between the first imaging device 1 as an in-vehicle camera and the camera model 2 as a simulated camera, in which a computer device (e.g., an information processing device equipped with an arithmetic processing unit such as a CPU 71) performs calibration by adjusting the yaw and pitch directions of the first imaging device 1 so that the center of the first target (indicators A and B) installed in at least two locations in front of the first imaging device 1 and at the same height position (height Hc) as the first imaging device 1 coincides with the center of the angle of view. By performing such calibration, it is possible to verify the consistency between the vehicle-mounted camera system S1 equipped with the first imaging device 1 and the simulator system S2 equipped with the camera model 2. Therefore, if it can be guaranteed that the camera model 2 properly simulates the first imaging device 1 when mounted on a vehicle, the camera model 2 can be used to properly test the vehicle-mounted camera, thereby enabling various cost reductions.

[0127] As explained with reference to Figures 18 and 19, calibration may be performed by adjusting the roll direction of the first imaging device 1 so that the center of a second target (index C), which is installed at a position horizontally offset from the optical axis of the first imaging device 1 and at the same height as the first imaging device 1, coincides with the vertical midpoint in the angle of view. This allows calibration in the roll direction to be performed using a simple method. Therefore, the accuracy of the match verification can be improved.

[0128] As explained with reference to Figure 20 etc., the depression angle may be adjusted as calibration by adjusting the pitch direction so that the center of the third target (index D) installed in front of the first imaging device 1 coincides with the middle position in the vertical direction. This allows calibration to be performed not only on the first imaging device 1 whose optical axis direction is substantially horizontal, but also on the first imaging device 1 that captures the area ahead of the vehicle 100 from a slightly bird's-eye view. By setting the optical axis direction of the camera model 2 to match the optical axis direction of the calibrated first imaging device 1 in this way, it is possible to prepare a camera model 2 that appropriately simulates the first imaging device 1 set in various orientations. Therefore, it is possible to appropriately perform a simulation of the first imaging device 1.

[0129] As described with reference to FIGS. 15 and 16, the optical axis direction of the camera model 2 may be adjusted based on the optical axis direction of the first imaging device 1 adjusted by calibration. This allows the optical axis direction of the first imaging device 1 and the optical axis direction of the camera model 2 to coincide with each other. Therefore, it is possible to prepare a camera model 2 that appropriately simulates the first imaging device 1, and simulations can be performed using the camera model 2 instead of the first imaging device 1, allowing efficient verification of algorithms used in driving assistance control.

[0130] The information processing device of this embodiment includes an arithmetic processing unit (e.g., CPU 71) that performs consistency verification between the first imaging device 1 and the camera model 2 by comparing the characteristics of the first imaging device 1 with the characteristics of the camera model 2 using a first image G1 output from the first imaging device 1 that captures an image of a specific subject (such as the above-mentioned brightness box, each chart, or a subject or preceding vehicle in an actual driving environment scene) and a second image G2 output from the camera model 2 to which two-dimensional input data (e.g., spectral radiance values) based on the measurement results of measuring light from the specific subject is input.

[0131] The information processing device of this embodiment is equipped with an arithmetic processing unit (e.g., CPU 71) that performs calibration by adjusting the yaw and pitch directions of the first imaging device 1 so that the center of the first target (indicators A and B) installed in at least two locations in front of the first imaging device 1 and at the same height position (height Hc) as the first imaging device 1 coincides with the center of the angle of view as a pre-processing step for verifying the consistency between the first imaging device 1 as an in-vehicle camera and the camera model 2 as a simulated camera.

[0132] The program executed by the information processing device described above can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a device such as a computer device, or on a ROM in a microcomputer having a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disk Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0133] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0134] Furthermore, the above-described examples may be combined in any manner, and even when various combinations are used, the various effects described above can be obtained.

[0135] <9. This Technology> This technology can also be configured as follows. (1) Using first image data output from a first image capturing device that captures an image of a specific subject and second image data output from a camera model to which two-dimensional input data based on a measurement result of measuring light from the specific subject is input, a comparison is made between the characteristics of the first image capturing device and the characteristics of the camera model, thereby verifying the consistency between the first image capturing device and the camera model. Information processing methods. (2) The input data is a spectral radiance value of the light from the specific subject. The information processing method according to (1) above. (3) The camera model includes an optical model and an image sensor model. An information processing method according to any one of (1) to (2) above. (4) the light measurement result is a measurement result for a predetermined area set within an angle of view of the first imaging device, The two-dimensional input data is arranged such that the measurement results are arranged in an area corresponding to the predetermined area. An information processing method according to any one of (1) to (3) above. (5) The matching verification is performed by comparing a pixel area corresponding to the predetermined area with an area corresponding to the pixel area in the second image data. The information processing method described in (4) above. (6) The two-dimensional input data is a data set in which dummy data is placed in areas other than the areas corresponding to the predetermined areas. The information processing method according to any one of (4) to (5) above. (7) the image sensor model has a filter model; The consistency verification is performed by performing statistical processing for each pixel group for each filter type in the second image data. The information processing method described in (3) above. (8) The statistical processing calculates a mean value and a variance for each pixel group, and the mean value and the variance are compared with respective thresholds to perform the consistency verification. The information processing method according to (7) above. (9) The filter model is a model that employs at least one of a Bayer array, an RCCB array, an RGBIR array, and an array using a complementary color filter. The information processing method according to (7) above. (10) As a pre-processing for verifying the consistency between a first imaging device as an in-vehicle camera and a camera model as a simulated camera, calibration is performed by adjusting the yaw direction and pitch direction of the first imaging device so that the center of a first target installed in at least two locations in front of the first imaging device and at the same height as the first imaging device coincides with the center of the angle of view. Information processing methods. (11) The calibration is performed by adjusting the roll direction of the first imaging device so that the center of a second target, which is installed at a position offset in the horizontal direction with respect to the optical axis of the first imaging device and at the same height as the first imaging device, coincides with the middle position in the vertical direction of the angle of view. The information processing method according to (10) above. (12) The depression angle adjustment as the calibration is performed by adjusting the pitch direction so that the center of a third target installed in front of the first imaging device coincides with the intermediate position in the vertical direction. The information processing method according to (11) above. (13) The optical axis direction of the camera model is adjusted based on the optical axis direction of the first image capturing device adjusted by the calibration. The information processing method according to any one of (10) to (12) above. (14) and a calculation processing unit that performs a consistency verification between the first image capturing device and the camera model by comparing characteristics of the first image capturing device and characteristics of the camera model using first image data output from a first image capturing device that captures an image of a specific subject and second image data output from a camera model to which two-dimensional input data based on a measurement result of measuring light from the specific subject is input. Information processing device. (15) The system includes a calculation processing unit that performs calibration by adjusting the yaw direction and pitch direction of the first imaging device so that the center of a first target installed in at least two locations in front of the first imaging device and at the same height as the first imaging device coincides with the center of the angle of view as a pre-processing for verifying the consistency between the first imaging device as an in-vehicle camera and the camera model as a simulated camera. Information processing device. [Explanation of symbols]

[0136] 1 First imaging device 2 Camera Models B7 Optical Model B8 Sensor Model (Image Sensor Model) G1 1st image G2 2nd image Ar, Ar1, Ar2, Ar3, Ar4, Ar5 Verification Area Cr, Cr1, Cr2, Cr3, Cr4, Cr5 compatible areas Ar', Ar'1, Ar'2, Ar'3, Ar'4, Ar'5 Verification Area A, B Indicator (first target) C Indicator (second target) D indicator (third target)

Claims

1. verifying the consistency between the first imaging device and the camera model by comparing characteristics of the first imaging device and characteristics of the camera model using first image data output from a first imaging device that has captured an image of a specific subject and second image data output from a camera model to which two-dimensional input data based on a measurement result of measuring light from the specific subject is input; the light measurement result is a measurement result for a predetermined area set within an angle of view of the first imaging device, The two-dimensional input data is data in which the measurement results are arranged in an area corresponding to the predetermined area. Information processing methods.

2. The input data is a spectral radiance value of the light from the specific subject. The information processing method according to claim 1 .

3. The camera model includes an optical model and an image sensor model. The information processing method according to claim 1 .

4. The matching verification is performed by comparing a pixel area corresponding to the predetermined area with an area corresponding to the pixel area in the second image data. The information processing method according to claim 1 .

5. The two-dimensional input data is a data set in which dummy data is placed in areas other than the areas corresponding to the predetermined areas. The information processing method according to claim 1 .

6. the image sensor model has a filter model; The consistency verification is performed by performing statistical processing for each pixel group for each filter type in the second image data. The information processing method according to claim 3 .

7. The statistical processing calculates a mean value and a variance for each pixel group, and the mean value and the variance are compared with respective thresholds to perform the consistency verification. The information processing method according to claim 6.

8. The filter model is a model that employs at least one of a Bayer array, an RCCB array, an RGBIR array, and an array using a complementary color filter. The information processing method according to claim 6.

9. a calculation processing unit that performs consistency verification between the first image capturing device and the camera model by comparing characteristics of the first image capturing device and characteristics of the camera model using first image data output from a first image capturing device that captures an image of a specific subject and second image data output from a camera model to which two-dimensional input data based on a measurement result of measuring light from the specific subject is input, the light measurement result is a measurement result for a predetermined area set within an angle of view of the first imaging device, The two-dimensional input data is data in which the measurement results are arranged in an area corresponding to the predetermined area. Information processing device.

Images