Rendering system and autonomous driving verification system

The rendering system and autonomous driving verification system address the inefficiencies of rolling shutter modeling by controlling the drawing engine to focus on exposed line blocks, enhancing execution speed and enabling effective algorithm verification.

JP7596360B2Active Publication Date: 2024-12-09SONY SEMICON SOLUTIONS CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022509391
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-03-24
Filing Date
2021-02-15
Publication Date
2024-12-09
Estimated Expiration
2041-02-15

AI Technical Summary

Technical Problem

Existing autonomous driving systems face challenges in efficiently modeling image sensors with a rolling shutter function, as increasing temporal resolution for precise exposure leads to significant drops in execution speed and prolonged drawing times.

Method used

A rendering system and autonomous driving verification system that utilize an image sensor model with a rolling shutter function, controlling a rendering engine to draw only the exposed line block area, thereby reducing drawing time by limiting the drawing area to the exposure position.

Benefits of technology

This approach significantly shortens drawing time while maintaining precision, allowing for efficient verification of autonomous driving algorithms in virtual environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007596360000001
    Figure 0007596360000001
  • Figure 0007596360000002
    Figure 0007596360000002
  • Figure 0007596360000003
    Figure 0007596360000003
Patent Text Reader

Abstract

This rendering system is provided with: an image sensor model which performs exposure by line block unit, with one or multiple pixel rows as one line block, and has a rolling shutter function; and a rendering engine which processes image data and performs rendering. Also, the image sensor model controls the rendering engine so as to render only the region of the line block that is exposed.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a rendering system and an autonomous driving verification system. [Background technology]

[0002] When developing an autonomous driving system, the autonomous driving algorithm is verified through simulation in a virtual space. To verify the autonomous driving algorithm, a model of an image sensor that plays the role of the human eye is required, and an input image that closely resembles a real-life image is generated using CG (computer graphics) synthesis technology (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-60511 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, in an autonomous driving system, assuming that an image sensor mounted on a vehicle has a rolling shutter function that performs exposure in units of one or more pixel rows as a line block and acquires pixel values ​​for each line block, it is necessary to realize a rolling shutter in precise modeling. However, if the time resolution is increased to reproduce exposure in units of line blocks and full-screen drawing is performed, the execution speed drops significantly and drawing takes time.

[0005] Therefore, an object of the present disclosure is to provide a rendering system that makes it possible to shorten the drawing time when realizing a rolling shutter in modeling of an image sensor, and an autonomous driving verification system that uses the rendering system. [Means for solving the problem]

[0006] In order to achieve the above object, the rendering system of the present disclosure comprises: An image sensor model having a rolling shutter function that performs exposure in units of one or more pixel rows as one line block, and A rendering engine that processes and renders image data; And, The image sensor model controls the rendering engine so that only the exposed line block area is drawn.

[0007] In addition, the autonomous driving verification system of the present disclosure for achieving the above object includes: An image sensor model with a rolling shutter function that exposes one or more pixel rows as one line block, A rendering engine that processes and renders image data; and An application processor equipped with the autonomous driving software to be verified; Equipped with The image sensor model controls the rendering engine so that only the exposed line block area is drawn. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram of the viewpoint tracking method, which is one of the rendering techniques. [Diagram 2] FIG. 2 is a timing chart for explaining a conventional rolling shutter (line exposure). [Diagram 3] FIG. 3 is a block diagram illustrating an example of the configuration of the rendering system according to the first embodiment. [Figure 4] FIG. 4 is a timing chart illustrating a rolling shutter (line exposure) executed in the image sensor model of the rendering system according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating a process flow of the rendering content determination method executed in the rendering system according to the first embodiment. [Figure 6] FIG. 6 is an explanatory diagram of the viewpoint tracking in the rendering system according to the first embodiment. [Figure 7] FIG. 7 is a timing chart for explaining the rolling shutter (line exposure) executed in an image sensor model that employs a wide dynamic range of the DOL method. [Figure 8] FIG. 8A is a diagram showing the configuration of a pixel in an image sensor model that employs a wide dynamic range of a sub-pixel structure method, and FIG. 8B is a diagram for explaining eye-gaze tracking. [Figure 9] FIG. 9 is a flowchart showing the flow of processing of a rendering content determination method executed in a rendering system that employs a wide dynamic range of the sub-pixel structure method. [Figure 10] FIG. 10 is a block diagram illustrating an example of the configuration of a rendering system according to the second embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of an autonomous driving verification system according to use case 1. [Figure 12] FIG. 12 is a block diagram showing an example configuration of an autonomous driving verification system according to use case 2. [Figure 13] FIG. 13 is a block diagram showing an example configuration of an autonomous driving verification system according to use case 3. [Figure 14] FIG. 14 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied. [Figure 15] FIG. 15 is a diagram showing an example of the installation position of the imaging unit in the mobile object control system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a form for carrying out the technology of the present disclosure (hereinafter, referred to as "embodiment") will be described in detail with reference to the drawings. The technology of the present disclosure is not limited to the embodiment. In the following description, the same elements or elements having the same functions will be designated by the same reference numerals, and duplicated description will be omitted. The description will be given in the following order. 1. General description of the rendering system and autonomous driving verification system disclosed herein 2. About the rendering system 3. Rendering system according to an embodiment of the present disclosure 3-1. Example 1 (Example of a case where a lens model is interposed between an image sensor model and a rendering engine) 3-1-1. System configuration example 3-1-2. How to determine drawing content 3-1-3. Wide dynamic range 3-2. Example 2 (Example of a case where a lens model is not interposed between an image sensor model and a rendering engine) 4. Autonomous driving verification system according to an embodiment of the present disclosure 4-1. Use Case 1 (Example of simulation technology using MILS / SILS) 4-2. Use Case 2 (Example of simulation technology using HILS, part 1) 4-3. Use Case 3 (Example 2 of simulation technology being HILS) 5. Variations 6. Application Examples 7. Configurations that the present disclosure can take

[0010] <General Description of the Rendering System of the Present Disclosure> In the rendering system of the present disclosure, the image sensor model may have a function of tracking the trajectory of light reaching the viewpoint from the viewpoint back to the light source. The image sensor model may also be configured to perform scanning in units of line blocks and provide the rendering engine with information on the ray tracing direction and ray tracing period for each exposure position.

[0011] In the rendering system of the present disclosure including the above-mentioned preferred configuration, the rendering engine may be configured to calculate the arrival position of a ray based on information on the ray tracing direction provided from the image sensor model, obtain the spectral reflectance of the object at the arrival position, and further trace the ray in the reflection direction after reaching the object at the arrival position to obtain the spectral irradiance of the light source at the arrival position.The rendering engine may be configured to obtain the spectral irradiance of the sensor image plane of the image sensor model based on the spectral reflectance of the object and the spectral irradiance of the light source.

[0012] In addition, the rendering system of the present disclosure including the above-mentioned preferred configuration may have a lens model between the image sensor model and the rendering engine. In this case, the image sensor model may perform scanning in units of line blocks and provide information on the ray tracing direction and ray tracing period for each exposure position to the lens model. In addition, the lens model may calculate the ray tracing direction based on the information on the ray tracing direction provided by the image sensor model, and provide the calculation result to the rendering engine.

[0013] In addition, in the rendering system of the present disclosure including the above-mentioned preferred configuration, the rendering engine can be configured to calculate the arrival position of the light ray based on the calculation result of the ray tracing direction given by the lens model, obtain the spectral reflectance of the object at the arrival position, and further trace the light ray in the reflection direction after reaching the object at the arrival position to obtain the spectral irradiance of the light source at the arrival position.The rendering engine can be configured to obtain the spectral irradiance of the sensor image plane of the image sensor model based on the spectral reflectance of the object and the spectral irradiance of the light source.

[0014] In addition, in the rendering system of the present disclosure including the above-described preferred configuration, the image sensor model may have an on-chip lens on a pixel, and the viewpoint position may be corrected based on the pupil correction shift amount of the on-chip lens relative to the pixel. Also, the lens model may be configured to correct the viewpoint position based on the pupil correction shift amount of the on-chip lens.

[0015] In the autonomous driving verification system disclosed herein, the image sensor model can be configured to have a function of tracking the trajectory of light reaching the viewpoint from the viewpoint back to the light source.

[0016] In the autonomous driving verification system of the present disclosure, which includes the preferred configuration described above, the application processor can be implemented on the same image processing unit as the image sensor model and the rendering engine, or on a central processing unit, to verify the installed autonomous driving software.

[0017] Alternatively, in the autonomous driving verification system of the present disclosure including the above-mentioned preferred configuration, the image sensor model and the rendering engine may be configured to be implemented in the same image processing unit, and the application processor may be configured to be implemented in an electronic control unit electrically connected to the image processing unit via a camera interface to verify the autonomous driving software installed therein.

[0018] Alternatively, in the autonomous driving verification system of the present disclosure including the above-mentioned preferred configuration, the image sensor model and the rendering engine may be implemented in separate image processing units electrically connected via a display interface, and the application processor may be implemented on an electronic control unit electrically connected via a camera interface to the image processing unit in which the image sensor model is implemented, to verify the autonomous driving software installed therein.

[0019] <About the rendering system> When developing an autonomous driving system, the autonomous driving algorithm is verified through simulation in a virtual space. To verify the autonomous driving algorithm, an image sensor model is required. An image sensor model is a sensor model that models an image sensor on software.

[0020] The rendering system disclosed herein is an in-vehicle rendering system particularly suitable for use in verifying autonomous driving algorithms. Here, "rendering" refers to the process of creating an image by calculation of information about an object or the like given as numerical data. In virtual space autonomous driving verification, for example, the viewpoint tracking method, which depicts the image by tracing the light rays that reach the viewpoint in reverse, as shown in FIG. 1 (reference: Internet<URL:https: / / www.scratchapixel.com / lessons / 3d-basic-rendering / global-illumination-path-tracing> ) can be used. Figure 1 shows an example in which the reflection of a ray of light from the viewpoint (camera) up to three times is taken into consideration. The viewpoint tracking method makes it possible to precisely reproduce shadow areas and reflections of puddles in CG (computer graphics) drawings.

[0021] Image sensors (imaging devices) include sensors with a global shutter function that exposes all pixels at the same time and acquires pixel values, and sensors with a rolling shutter function that treats one or more pixel rows as one line block, exposes the pixels in line block units, and acquires pixel values.

[0022] In autonomous driving systems, there is a tendency to use sensors with a rolling shutter function as the image sensor installed in the vehicle, which has advantages such as being more cost-effective and being able to select long exposure times compared to sensors with a global shutter function. In this case, it is necessary to realize a rolling shutter for precise modeling. However, if the time resolution is increased to reproduce exposure in line block units and full-screen drawing is performed, the execution speed drops significantly and drawing takes a long time.

[0023] A timing chart explaining a conventional rolling shutter (line exposure) that performs exposure in line block units is shown in Fig. 2. In the timing chart in Fig. 2, A is a drawing range of a certain unit time (i.e., variable time resolution Δt), and a range that covers the accumulation time of B is drawn and added. In the drawing range A, the area from vertical coordinate y1 to y2 becomes a drawing range with a time resolution Δt that depends on the accumulation time.

[0024] <Rendering system according to an embodiment of the present disclosure> A rendering system according to an embodiment of the present disclosure is, for example, a system for verifying autonomous driving in virtual space, and includes an image sensor model with a rolling shutter function, and a rendering engine that processes and renders image data.

[0025] In the rendering system according to this embodiment, the image sensor model uses a known viewpoint tracking method (see FIG. 1) to control the rendering engine so that rendering is performed only in the exposed line block area. In this way, the rendering area when rendering by the rendering engine is limited to the exposed position of the line block under the control of the image sensor model, so that the rendering time can be reduced compared to the case of full-screen rendering in which one screen's worth of rendering is performed all at once.

[0026] The rolling shutter is, for example, a shutter operation in which exposure is performed in units of one pixel row, that is, in units of one line, in sequence, to obtain pixel values. However, the rolling shutter is not limited to exposure in units of one line, but may be a shutter operation in which exposure is performed in units of one line block, with multiple pixel rows as one line block. In the following, for ease of understanding, the shutter operation is described as a rolling shutter that performs exposure in units of one line in sequence, and exposure in units of one line may be referred to as "line exposure."

[0027] In the rolling shutter operation, the accumulation time is divided within the moving object blur permissible time, and the signal level is calculated by accumulating for the divided time, and this is carried out over the accumulation time of the pixel and then added up.

[0028] A specific example of the embodiment of the present disclosure for limiting the drawing area to the exposure position of the line block and shortening the drawing time will be described below.

[0029] [Example 1] The first embodiment is an example of a case where a lens model is provided between an image sensor model and a rendering engine. A block diagram of a configuration example of a rendering system according to the first embodiment is shown in FIG.

[0030] (System configuration example) As shown in Fig. 3, the rendering system 1A according to the first embodiment includes an image sensor model 10 and a rendering engine 20, as well as a lens model 30 interposed between the image sensor model 10 and the rendering engine 20. The image sensor model 10 includes an output interface I / F_A. The rendering engine 20 includes an input interface I / F_B. The lens model 30 includes an input interface I / F_C and an output interface I / F_D.

[0031] In the rendering system 1A according to the first embodiment having the above configuration, the image sensor model 10 has a rolling shutter function and a function of tracking the viewpoint by a viewpoint tracking method. In the viewpoint tracking method, rendering is performed by tracing the trajectory of a light ray that reaches each pixel of the image sensor model 10, which is the viewpoint, from the viewpoint back to the light source.

[0032] The image sensor model 10 sequentially scans in units of one line block (in this example, in units of one line), and transmits information on the ray tracing direction and ray tracing period for each line exposure position from the output interface I / F_A as a control signal to the preceding lens model 30. The ray tracing direction is determined taking into account internal sensor information such as pixels and on-chip lenses.

[0033] The lens model 30 receives a control signal transmitted from the image sensor model 10, i.e., information on the ray tracing direction and ray tracing period, via the input interface I / F_C. The lens model 30 then calculates the ray tracing direction based on the received control signal, and transmits the information on the calculated ray tracing direction as a control signal to the preceding rendering engine 20 from the output interface I / F_D.

[0034] The rendering engine 20 receives the control signal transmitted from the lens model 30, i.e., information on the calculated ray tracing direction, via the input interface I / F_B. Then, based on the received control signal, i.e., the information on the ray tracing direction, the rendering engine 20 calculates the arrival position of the light ray emitted from the image sensor model 10, and obtains the spectral reflectance of the object where the light ray arrives.

[0035] The rendering engine 20 further traces the light ray in the direction of reflection after reaching the object, and obtains the spectral radiance of the light source at the point of arrival. The rendering engine 20 obtains the spectral irradiance of the sensor image plane of the image sensor model 10 based on the spectral reflectance of the object at which the light ray reaches, and the spectral radiance of the light source. Note that it is possible to obtain not only the spectral irradiance, but also the radiant flux or radiance.

[0036] According to the rendering system 1A of the first embodiment having the above configuration, under the control of the image sensor model 10, the area of ​​the rendering engine 20 can be limited to the area that is line exposed, thereby shortening the drawing time compared to the case of full-screen drawing.

[0037] FIG. 4 is a timing chart for explaining the rolling shutter (line exposure) executed in the image sensor model 10 of the rendering system 1A according to the first embodiment.

[0038] In the timing chart of Fig. 4, the time Δt in area A is the time resolution, and this time is variable. The section in area B is an accumulation section for long-time accumulation. Area C is an example of a drawing area. In the drawing area C, the area of ​​vertical coordinates y1 to y2 is a drawing area with a time resolution Δt that depends on the accumulation time.

[0039] As is clear from the timing chart of FIG. 4, in the rendering system 1A of the first embodiment, under the control of the image sensor model 10, the rendering engine performs drawing only in the portion C where the time resolution Δt and the section of the region B overlap, i.e., the range where line exposure is performed.

[0040] (How to determine drawing content) Next, a description will be given of a rendering content determination method executed in the rendering system 1A according to the embodiment 1. The flow of processing of the rendering content determination method in the rendering system 1A according to the embodiment 1 is shown in the flowchart of FIG.

[0041] 6 is an explanatory diagram of viewpoint tracking in the rendering system 1A according to the first embodiment. As shown in FIG. 6, an image sensor model 10 is configured by arranging pixels 11, and further, an on-chip lens 12 is arranged on the pixels 11 in correspondence with the pixels 11. In this image sensor model 10, each pixel 11 serves as a viewpoint. In FIG. 6, the on-chip lens 12 is abbreviated as "OCL." The ray tracing direction is determined in consideration of sensor internal information such as the pixels 11 and the OCL 12.

[0042] In the flowchart of FIG. 5, the image sensor model 10 first sets a time resolution Δt (step S11), then selects a line exposure direction (vertical coordinates y1 to y2 that overlap with times t to t+Δt) (step S12), and then sets the pixel area at the coordinate position (x, y) as the viewpoint base point (step S13).

[0043] Next, the image sensor model 10 acquires the pupil correction shift amount of the on-chip lens 12 corresponding to the coordinate (x, y) position of the viewpoint (step S14), then sets a scanning range from the chief ray of the on-chip lens 12 (step S15), and then updates the scanning range to that of the chief ray of the lens model 30 (step S16).

[0044] That is, in the image sensor model 10, the viewpoint position is corrected based on the pupil correction shift amount of the on-chip lens 12 with respect to the pixel 11 at the viewpoint coordinate (x, y) position. Also, in the lens model 30, the viewpoint position is corrected based on the pupil correction shift amount of the on-chip lens 12 with respect to the pixel 11.

[0045] Next, the rendering engine 20 identifies the object and the light source at the position reached by each ray emitted from the light source (step S17), then obtains the spectral reflectance of the object reached by the ray and the spectral irradiance of the light source (step S18), and then convolves the spectrum and performs accumulation from time t to t + Δt (step S19).

[0046] Through the series of processes performed by the image sensor model 10, the rendering engine 20, and the lens model 30, the rendering system 1A according to the first embodiment can determine the rendering content.

[0047] (Regarding wide dynamic range) Incidentally, for example, in-vehicle cameras (image sensors) are required to be able to capture high-quality images with a wide (high) dynamic range even in scenes with large differences in brightness, such as the entrance and exit of a tunnel during the day.

[0048] As a method for realizing a wide dynamic range, for example, the following two methods can be exemplified. However, the present invention is not limited to these two methods, and for example, a method in which the two methods are combined can also be used.

[0049] One of them is a method to obtain multiple signals with different exposure times within one frame, for example, two signals with a relatively long accumulation time and a relatively short accumulation time, and then synthesize them to achieve a wide dynamic range (hereinafter referred to as the DOL (Digital OverLap) method).

[0050] The other is a method in which one pixel is divided into multiple sub-pixels of different areas, for example two sub-pixels: one with a relatively large area and one with a relatively small area, to obtain two signals with different sensitivities and then combine them to achieve a wide dynamic range (hereinafter referred to as the sub-pixel structure method).

[0051] Example of wide dynamic range using the DOL method The rolling shutter (line exposure) executed in the image sensor model 10 that employs the wide dynamic range of the DOL method will be described with reference to the timing chart of FIG.

[0052] 7, in an image sensor model 10 that employs the wide dynamic range of the DOL method, for example, two signals with different exposure times are obtained in one frame, that is, a relatively long-time accumulation signal and a relatively short-time accumulation signal. Then, during the time of the time resolution Δt, only the area C that overlaps with the long-time accumulation section and the area D that overlaps with the short-time accumulation section are rendered.

[0053] Example of wide dynamic range achieved by sub-pixel structure method In an image sensor model 10 that adopts a wide dynamic range using a sub-pixel structure, for example, as shown in Fig. 8A, one pixel 11 is divided into two sub-pixels with different areas, namely, a relatively large sub-pixel SP1 and a relatively small sub-pixel SP2. The relatively large sub-pixel SP1 is a high-sensitivity sub-pixel, and the relatively small sub-pixel SP2 is a low-sensitivity sub-pixel. Two signals with different sensitivities are obtained from these two sub-pixels SP1 and SP2, and a wide dynamic range is realized by synthesizing the two signals.

[0054] FIG. 8B is an explanatory diagram of the eye point tracking in the rendering system 1A in the case of a wide dynamic range of the sub-pixel structure method, and FIG. 9 is a flowchart showing the process flow of the rendering content determination method.

[0055] The image sensor model 10 first sets the time resolution Δt (step S21), then selects the line exposure direction (vertical coordinates y1 to y2 that overlap with times t to t+Δt) (step S22), and then sets the pixel area of ​​the sub-pixel SP1 at the coordinate position (x, y) as the viewpoint base point (step S23). That is, in the case of the sub-pixel structure method, the viewpoint base point is determined according to the positions of the high-sensitivity and low-sensitivity sub-pixels.

[0056] Here, in the image sensor model 10, the viewpoint position is corrected based on the pupil correction shift amount of the on-chip lens 12 for the subpixel SP1 at the viewpoint coordinate (x, y) position. Also, in the lens model 30, the viewpoint position is corrected based on the pupil correction shift amount of the on-chip lens 12 for the subpixel SP1.

[0057] Next, the image sensor model 10 acquires the pupil correction shift amount of the on-chip lens 12 corresponding to the coordinate (x, y) position of the viewpoint (step S24), then sets a scanning range from the chief ray of the on-chip lens 12 (step S25), and then updates the scanning range to that of the chief ray of the lens model 30 (step S26).

[0058] Next, the rendering engine 20 identifies the object and the light source at the position reached by each ray emitted from the light source (step S27), then obtains the spectral reflectance of the object reached by the ray and the spectral irradiance of the light source (step S28), and then convolves the spectrum and performs accumulation from time t to t+Δt (step S29).

[0059] By the series of processes by the image sensor model 10, the rendering engine 20, and the lens model 30 described above, the rendering content can be determined even in the rendering system 1A in the case of a wide dynamic range of the sub-pixel structure method.

[0060] [Example 2] The second embodiment is an example of a case where no lens model is interposed between an image sensor model and a rendering engine. A block diagram of a configuration example of a rendering system according to the second embodiment is shown in FIG.

[0061] (System configuration example) 10, a rendering system 1B according to the second embodiment includes an image sensor model 10 and a rendering engine 20. The image sensor model 10 includes an output interface I / F_A. The rendering engine 20 includes an input interface I / F_B.

[0062] The image sensor model 10 has a rolling shutter function and a function of tracking the viewpoint by the viewpoint tracking method, similarly to the rendering system 1A according to the first embodiment. The image sensor model 10 sequentially scans one line at a time, and transmits information on the ray tracing direction and ray tracing period for each line exposure position to the rendering engine 20 from the output interface I / F_A as a control signal.

[0063] The rendering engine 20 receives, via the input interface I / F_B, a control signal transmitted from the image sensor model 10, i.e., information on the ray tracing direction and ray tracing period. Then, based on the received information on the ray tracing direction, the rendering engine 20 calculates the arrival position of the light ray emitted from the image sensor model 10, and obtains the spectral reflectance of the object on which the light ray reaches.

[0064] The rendering engine 20 further traces the light ray in the direction of reflection after reaching the object, and obtains the spectral radiance of the light source at the point of arrival. The rendering engine 20 obtains the spectral irradiance of the sensor image plane of the image sensor model 10 based on the spectral reflectance of the object at which the light ray reaches, and the spectral radiance of the light source. Note that it is possible to obtain not only the spectral irradiance, but also the radiant flux or radiance.

[0065] The rolling shutter (line exposure) and the drawing content determination method executed in the image sensor model 10 of the rendering system 1B according to the second embodiment having the above configuration are the same as those in the rendering system 1A according to the first embodiment, and therefore the description thereof will be omitted. Also, the wide dynamic range is the same as that in the rendering system 1A according to the first embodiment.

[0066] According to the rendering system 1B of Example 2 having the above configuration, although there is no lens model equivalent to the lens model 30 of FIG. 3, as in the case of the rendering system 1A of Example 1, the area of ​​rendering by the rendering engine 20 can be limited to the area that is line exposed, and therefore it is possible to shorten the rendering time compared to the case of full-screen rendering.

[0067] <Autonomous driving verification system according to an embodiment of the present disclosure> The rendering system according to the embodiment of the present disclosure described above can be used in combination with software on a downstream chip (application processor) that performs image recognition using the output of the image sensor model 10 as input to construct an autonomous driving verification system that verifies autonomous driving algorithms.

[0068] Below, use cases of the rendering system according to the embodiment of the present disclosure for constructing an autonomous driving verification system will be described. Here, three use cases, use case 1, use case 2, and use case 3, will be illustrated.

[0069] [Use case 1] Use case 1 is an example in which the simulation technology is MILS (Model In The Loop Simulation) / SILS (Software In The Loop Simulation). A block diagram of a configuration example of an autonomous driving verification system related to use case 1 is shown in FIG.

[0070] The autonomous driving verification system for use case 1 has an application processor 40 implemented on the same GPU (Graphics Processing Unit) as the image sensor model 10 and the rendering engine 20, or on a CPU (Central Processing Unit) 50, and is configured to verify the autonomous driving software installed in the application processor 40.

[0071] In the autonomous driving verification system according to the use case 1 of the above system configuration, information is sent from the image sensor model 10 to the rendering engine 20 to reproduce the operation of the actual machine. In addition, the exposure timing is sent from the image sensor model 10 to the rendering engine 20 to accurately reproduce the timing of flickering and blur of LEDs (light-emitting diodes). Incidentally, LED traffic lights and signs blink periodically, and if the lighting timing does not match the exposure timing of the image sensor, the lighting period cannot be captured and the lights go out. If this occurs, there are cases where the LED traffic lights and signs cannot be correctly recognized. In addition, the blur refers to moving object blur, and if the accumulation time is long, the moving object may be blurred and may not be correctly recognized.

[0072] In the rendering engine 20, for example, when generating R (red), G (green), and B (blue) three primary color signals, the sensor color reproduction cannot be faithfully reproduced, but the sensor color reproduction can be faithfully performed by transmitting the sensor spectrum from the image sensor model 10 to the rendering engine 20. However, the calculation of the sensor spectrum may be performed by the image sensor model 10.

[0073] A simulated image is input from the rendering engine 20 to the image sensor model 10. The image sensor model 10 adds noise and sensor characteristics to the input image from the rendering engine 20 and supplies the image to the application processor 40. The application processor 40 is equipped with autonomous driving software to be verified and performs verification of the autonomous driving software.

[0074] [Use case 2] Use case 2 is an example (part 1) of a case where the simulation technology is Hardware In The Loop Simulation (HILS). A block diagram of a configuration example of an autonomous driving verification system according to use case 2 is shown in FIG.

[0075] The autonomous driving verification system according to the use case 2 has an image sensor model 10 and a rendering engine 20 mounted on the same GPU (image processing unit) 50, an application processor 40 constituting an advanced driver-assistance system (ADAS), an autonomous driving system (AD), or an autonomous driving system (AD), mounted on an ECU (electronic control unit) 60, and the application processor 40 is executed on the ECU 60 to verify autonomous driving. The GPU 50 and the ECU 60 are electrically connected via a camera I / F (interface) 70 such as SERDES or MIPI.

[0076] In the autonomous driving verification system according to use case 2 of the above system configuration, the operation of the actual vehicle is reproduced by transmitting information from the image sensor model 10 to the rendering engine 20. In addition, by transmitting exposure timing from the image sensor model 10 to the rendering engine 20, the timing of flickering and blurring of an LED (light-emitting diode) are accurately reproduced.

[0077] In the rendering engine 20, for example, when generating R, G, and B three primary color signals, the sensor color reproduction cannot be faithfully reproduced, but the sensor color reproduction can be faithfully performed by transmitting the sensor spectrum from the image sensor model 10 to the rendering engine 20. However, the calculation of the sensor spectrum may be performed in the image sensor model 10.

[0078] A simulated image is input from the rendering engine 20 to the image sensor model 10. The image sensor model 10 processes the input image from the rendering engine 20, such as encoding it according to the actual speed of the vehicle, and supplies it to the application processor 40 on the ECU 60 via the camera I / F 70. The application processor 40 is equipped with autonomous driving software to be verified, and performs verification of the autonomous driving software.

[0079] [Use case 3] Use case 3 is an example (part 2) of a case where the simulation technology is Hardware In The Loop Simulation (HILS). A block diagram of a configuration example of an autonomous driving verification system related to use case 3 is shown in FIG.

[0080] The autonomous driving verification system according to the use case 3 has a system configuration in which an image sensor model 10 and a rendering engine 20 are implemented on separate GPUs (image processing units) 50A and 50B, an advanced driver assistance system (ADAS), an autonomous driving system (AD), or an application processor 40 constituting the autonomous driving system (AD) is implemented on an ECU (electronic control unit) 60, and the processing of the application processor 40 is executed on the ECU 60 to verify autonomous driving. The GPU 50A and the GPU 50B are connected via a display I / F (interface) 80 such as HDMI (registered trademark) or DisplayPort. The GPU 50B and the ECU 60 are electrically connected via a camera I / F 70 such as SERDES or MIPI.

[0081] In the autonomous driving verification system according to use case 3 configured as above, information cannot be sent from the image sensor model 10 to the rendering engine 20. (However, this is not the case when there is an interface from the image sensor model 10 to the rendering engine 20 other than the display I / F 80.) Therefore, a moderate degree of reproducibility is ensured by approximating the phenomenon using metadata in the rendering engine 20. The application processor 40 is equipped with the autonomous driving software to be verified, and performs verification of the autonomous driving software.

[0082] Regarding the timing of blur and LED flicker, the degree of blur and LED lighting period are approximated using the phase difference and speed between the image sensor model 10 and the light source or moving object as variables. Also, by having the rendering engine 20 store the sensor spectrum in advance, the sensor color can be reproduced faithfully.

[0083] <Modification> Although the technology according to the present disclosure has been described above based on a preferred embodiment, the technology according to the present disclosure is not limited to the embodiment. The configuration and structure of the imaging device described in the above embodiment are examples and can be changed as appropriate.

[0084] <Application Examples> The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be used to verify the autonomous driving algorithm of any type of moving object, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.

[0085] FIG. 14 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.

[0086] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 14, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0087] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0088] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0089] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0090] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0091] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0092] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0093] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0094] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0095] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 14, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0096] FIG. 15 is a diagram showing an example of the installation position of the imaging unit 12031.

[0097] In FIG. 15, a vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as an imaging unit 12031.

[0098] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0099] 15 shows an example of the imaging ranges of the imaging units 12101 to 12104. An imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and an imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0100] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0101] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0102] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0103] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0104] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. In the vehicle control system described above, when the imaging unit 12031 has an image sensor having a rolling shutter function, the technology according to the present disclosure can be applied to shorten the drawing time compared to the case of full-screen drawing, and therefore the verification of autonomous driving can be performed quickly.

[0105] <Configurations that the present disclosure can take> The present disclosure may also be configured as follows.

[0106] A. Rendering System [A-1] An image sensor model having a rolling shutter function that performs exposure on a line block basis, with one or more pixel rows as one line block, and A rendering engine that processes and renders image data; Equipped with The image sensor model controls the rendering engine so that only the exposed line block area is drawn. Rendering system. [A-2] The image sensor model has the function of tracking the trajectory of light reaching the viewpoint, tracing it back from the viewpoint to the light source. The rendering system described in [A-1] above. [A-3] The image sensor model scans in units of line blocks and provides the rendering engine with information about the ray tracing direction and ray tracing period for each exposure position. The rendering system described in [A-2] above. [A-4] The rendering engine calculates the arrival position of the ray based on the ray tracing direction information provided by the image sensor model, and obtains the spectral reflectance of the object at the arrival position. The rendering system described in [A-3] above. [A-5] The rendering engine traces the light rays in the direction of reflection after reaching the object at the destination position, and obtains the spectral irradiance of the light source at the destination. The rendering system described in [A-4] above. [A-6] The rendering engine obtains the spectral irradiance of the sensor image plane of the image sensor model based on the spectral reflectance of the object and the spectral radiance of the light source. The rendering system described in [A-5] above. [A-7] Having a lens model between the image sensor model and the rendering engine, The rendering system described in [A-2] above. [A-8] The image sensor model scans in units of line blocks, and provides the lens model with information about the ray tracing direction and ray tracing period for each exposure position. The rendering system described in [A-7] above. [A-9] The lens model calculates the ray tracing direction based on the information about the ray tracing direction provided by the image sensor model, and provides the calculation result to the rendering engine. The rendering system described in [A-8] above. [A-10] The rendering engine calculates the arrival position of the ray based on the calculation result of the ray tracing direction given by the lens model, and obtains the spectral reflectance of the object at the arrival position. The rendering system described in [A-9] above. [A-11] The rendering engine follows the light rays in the direction of reflection after reaching the object at the destination position, and obtains the spectral irradiance of the light source at the destination. The rendering system according to [A-10] above. [A-12] The rendering engine obtains the spectral irradiance of the sensor image plane of the image sensor model based on the spectral reflectance of the object and the spectral radiance of the light source. The rendering system according to [A-11] above. [A-13] The image sensor model has an on-chip lens on a pixel, and corrects the viewpoint position based on the pupil correction shift amount of the on-chip lens relative to the pixel. A rendering system according to any one of [A-7] to [A-12] above. [A-14] The lens model corrects the viewpoint position based on the pupil correction shift amount of the on-chip lens. The rendering system described in [A-13] above.

[0107] <B. Autonomous Driving Verification System> [B-1] An image sensor model having a rolling shutter function that exposes one or more pixel rows as one line block, A rendering engine that processes and renders image data; and An application processor equipped with the autonomous driving software to be verified; Equipped with The image sensor model controls the rendering engine so that only the exposed line block area is drawn. Autonomous driving verification system. [B-2] The image sensor model has the function of tracking the trajectory of light reaching the viewpoint, tracing it back from the viewpoint to the light source. The autonomous driving verification system according to claim 15. [B-3] The application processor is implemented on the same image processing unit or central processing unit as the image sensor model and the rendering engine, and performs verification of the installed autonomous driving software. The autonomous driving verification system described in [B-1] or [B-2] above. [B-4] The image sensor model and the rendering engine are implemented in the same image processing unit. The application processor is implemented on the electronic control unit, which is electrically connected to the image processing unit via a camera interface, and performs verification of the installed autonomous driving software. The autonomous driving verification system described in [B-1] or [B-2] above. [B-5] The image sensor model and the rendering engine are implemented in separate image processing units electrically connected via a display interface, The application processor is implemented on an electronic control unit that is electrically connected to an image processing unit in which an image sensor model is implemented via a camera interface, and performs verification of the installed autonomous driving software. The autonomous driving verification system described in [B-1] or [B-2] above. [Explanation of symbols]

[0108] 1A Rendering system according to a first embodiment, 1B Rendering system according to a second embodiment, 10 Image sensor model, 11 Pixel, 12 On-chip lens (OCL), 20 Rendering engine, 30 Lens model, 40 Application processor, 50 GPU (image processing unit) / CPU (central processing unit), 60 ECU (electronic control unit), 70 Camera I / F, 80 Display I / F

Claims

1. An image sensor model having a rolling shutter function that performs exposure in units of one or more pixel rows as one line block, and A rendering engine that processes and renders image data; Equipped with The image sensor model controls the rendering engine so that only the exposed line block area is drawn. Rendering system.

2. The image sensor model has the function of tracking the trajectory of light reaching the viewpoint, tracing it back from the viewpoint to the light source. The rendering system of claim 1 .

3. The image sensor model scans in units of line blocks and provides the rendering engine with information about the ray tracing direction and duration for each exposure position.

3. The rendering system of claim 2.

4. The rendering engine calculates the arrival position of the ray based on the ray tracing direction information provided by the image sensor model, and obtains the spectral reflectance of the object at the arrival position.

4. The rendering system of claim 3.

5. The rendering engine follows the light rays in the direction of reflection after reaching the object at the destination position, and obtains the spectral irradiance of the light source at the destination.

5. The rendering system of claim 4.

6. The rendering engine obtains a spectral irradiance of a sensor image plane of the image sensor model based on the spectral reflectance of the object and the spectral radiance of the light source.

6. The rendering system of claim 5.

7. Having a lens model between the image sensor model and the rendering engine, 3. The rendering system of claim 2.

8. The image sensor model scans in units of line blocks and provides the lens model with information about the ray tracing direction and ray tracing period for each exposure position.

8. The rendering system of claim 7.

9. The lens model calculates the ray tracing direction based on the information about the ray tracing direction provided by the image sensor model, and provides the calculation result to the rendering engine.

9. The rendering system of claim 8.

10. The rendering engine calculates the arrival position of the ray based on the calculation result of the ray tracing direction given by the lens model, and obtains the spectral reflectance of the object at the arrival position.

10. The rendering system of claim 9.

11. The rendering engine follows the light rays in the direction of reflection after reaching the object at the destination position, and obtains the spectral irradiance of the light source at the destination.

11. The rendering system of claim 10.

12. The rendering engine obtains a spectral irradiance of a sensor image plane of the image sensor model based on the spectral reflectance of the object and the spectral radiance of the light source.

12. The rendering system of claim 11.

13. The image sensor model has an on-chip lens on a pixel, and corrects the viewpoint position based on a pupil correction shift amount of the on-chip lens relative to the pixel.

8. The rendering system of claim 7.

14. The lens model corrects the viewpoint position based on the pupil correction shift amount of the on-chip lens.

14. The rendering system of claim 13.

15. An image sensor model having a rolling shutter function that performs exposure in line block units, with one or more pixel rows as one line block; A rendering engine that processes and renders image data; and An application processor equipped with the autonomous driving software to be verified; Equipped with The image sensor model controls the rendering engine so that only the exposed line block area is drawn. Autonomous driving verification system.

16. The image sensor model has the function of tracking the trajectory of light reaching the viewpoint, tracing it back from the viewpoint to the light source. The autonomous driving verification system according to claim 15.

17. The application processor is implemented on the same image processing unit or central processing unit as the image sensor model and rendering engine, and performs validation of the installed autonomous driving software. The autonomous driving verification system according to claim 15.

18. The image sensor model and the rendering engine are implemented in the same image processing unit. The application processor is implemented on the electronic control unit, which is electrically connected to the image processing unit via a camera interface, and performs verification of the installed autonomous driving software. The autonomous driving verification system according to claim 15.

19. the image sensor model and the rendering engine are implemented in separate image processing units electrically connected via a display interface; The application processor is implemented on an electronic control unit that is electrically connected to an image processing unit in which an image sensor model is implemented via a camera interface, and performs verification of the installed autonomous driving software. The autonomous driving verification system according to claim 15.

Citation Information

Patent Citations

  • Simulation system, simulation program, and simulation method

    JP2018060511A

  • Imaging apparatus, imaging system, imaging method and imaging program

    JP2020039126A