A device for generating lens distortion images in a three-dimensional virtual space, its processing method, and a program.

Abstract

To make it possible to provide an appropriate lens distortion image to a vehicle control unit in a simulation device which allows a vehicle control unit of a vehicle which travels in a real space to travel in a three dimensional virtual space.SOLUTION: A device 33 executes an image process by using a memory 44 and generates an image containing a lens distortion like obtained by imaging a three dimensional virtual space with a wide angle camera 2 of a prescribed view point, acquires a view point image 50 viewing a three dimensional virtual space from the prescribed view point and deploys it in a memory 44, executes a lens distortion process, corresponding to the wide angle camera 2, to the deployed view point image 50, and outputs an image to which the lens distortion process is applied. In the lens distortion process, positions of each image pixels of the view point image 50 are moved with movement amounts corresponding to the lens distortion of the wide angle camera 2.SELECTED DRAWING: Figure 5

Description

Technical Field

[0004]

[0001] The present invention relates to a lens distortion image generation device for a three-dimensional virtual space, a processing method thereof, and a program.

Background Art

[0002] In vehicles such as automobiles, development for driving support or autonomous driving is in progress. In such vehicles, for example, the periphery of the vehicle during travel is imaged by a camera provided in the vehicle, an object outside the vehicle such as another moving body or a falling object is detected in the captured image, and driving control according to the detected driving situation is executed. In the development of the vehicle control unit of such a vehicle, it is conceivable to run the vehicle under development in a three-dimensional virtual space.

Prior Art Documents

Patent Documents

[0003] [

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, even if the current simulation device can simulate running the vehicle under development in a three-dimensional virtual space, in that simulation, an image equivalent to what the camera provided in the vehicle captures cannot be obtained with real-time performance at the same cycle as the camera provided in the vehicle.

[0005] For example, cameras used to image the outside of a vehicle typically have a wide-angle field of view of at least 120 degrees, preferably around 170 degrees. In this case, the camera used in the simulation device also needs to generate images that include lens distortion caused by the wide-angle camera, similar to the wide-angle camera installed in the vehicle. However, images captured in a three-dimensional virtual space by the camera provided in the simulation device are basically designed to be realistic and therefore do not include lens distortion caused by the wide-angle camera. In such images that do not include lens distortion caused by the wide-angle camera, the balance between the image size of the vehicle in the center of the image and the image size of the vehicle at the periphery of the image differs significantly from images that include lens distortion caused by the wide-angle camera. If such images, which differ from those of an actual vehicle, are provided to the vehicle control unit of a vehicle under development, it will not be possible to obtain appropriate simulation results that are relevant to practical use.

[0006] Furthermore, Patent Document 1 discloses a technology for an inspection system usable in production lines and the like that reflects the effects of lens distortion appearing in the actual image and forms a virtual camera image that is accurately aligned with the actual image. However, the technology described in Patent Document 1 does not involve driving a vehicle under development in a three-dimensional virtual space. Moreover, the inspection system described in Patent Document 1 does not have the constraint of real-time capabilities, such as when repeatedly providing images to the vehicle control unit of a vehicle under development, and it is unlikely that it can be used as is to obtain images for simulating vehicle driving control.

[0007] Furthermore, Patent Document 2 discloses a simulation program that causes a computer to execute the following steps: an image capture creation procedure for creating an image captured by an imaging unit virtually positioned at a predetermined viewpoint position within a 3D model based on data of the 3D model; a curvature calculation procedure for dividing the image captured by the image capture creation procedure into image regions of a predetermined angular shape and calculating the curvature of each image region based on the data belonging to each image region; a fineness adjustment procedure for adjusting the fineness of a mesh of a virtual plane virtually positioned at a predetermined distance from the viewpoint position based on the curvature of each image region calculated by the curvature calculation procedure; and a wide-angle image generation procedure for applying aberration data of a wide-angle lens used in the imaging unit to the image captured, according to the fineness of the mesh of the virtual plane adjusted by the fineness adjustment procedure, and generating a wide-angle image captured through the wide-angle lens. However, Patent Document 2 divides the captured image into multiple image regions and performs image processing on each image region in order to include lens distortion in the image. As a result, the value of each pixel in the processed image will be different from the value in the acquired image. For example, the color of the vehicle and the color of the road surface may be mixed, resulting in a value for a color that is neither of them. As a result, in images that include lens distortion, for example, the outline of the vehicle tends to be blurred. If machine learning is performed on blurred outlines or parameter settings are adjusted, when the developed vehicle control unit is actually implemented in a vehicle and used for driving control in real space, it may affect the vehicle recognition itself based on the wide-angle camera image, or errors may occur in the size and distance of the recognized vehicle. In vehicles that travel in real space, for example, it is necessary to detect external objects such as other moving objects and fallen objects around the vehicle based on images of the outside of the vehicle captured by a wide-angle camera, and to perform driving control according to the detected external objects. Furthermore, since Patent Document 2 divides the captured image into multiple image regions and performs multiple image transformation processes for each image region, the transformation process takes time for each image region. It is expected that achieving real-time performance, such as repeatedly supplying images to the vehicle control unit of a vehicle under development, will not be easy with such a process involving multiple image transformations.

[0008] Thus, in the development of vehicle control systems for vehicles that travel in real space, there is a need to be able to provide wide-angle lens distortion images when the vehicle is driven in a three-dimensional virtual space. [Means for solving the problem]

[0009] A lens distortion image generation apparatus for a three-dimensional virtual space according to one embodiment of the present invention is an apparatus that generates a lens distortion image including lens distortion by a wide-angle camera, as if the three-dimensional virtual space were captured by a wide-angle camera at a predetermined viewpoint, by performing image processing using memory, and comprises: a viewpoint image acquisition unit that acquires a viewpoint image of the three-dimensional virtual space viewed from the predetermined viewpoint and expands it into memory; a lens distortion processing unit that performs lens distortion processing corresponding to the wide-angle camera on the viewpoint image expanded in memory; and an output unit that outputs the image after lens distortion processing in memory as a lens distortion image including distortion by the wide-angle camera, wherein the lens distortion processing unit In the memory, a work area is generated consisting of at least all of the multiple image pixels of the viewpoint image and a plurality of corresponding pixels, with the center of the lens distortion as the origin and pixels from the first quadrant to the fourth quadrant. For each pixel of the viewpoint image expanded in the work area of ​​the memory, information is provided on the amount of positional movement of the plurality of pixels in the work area to move them toward the origin, which is the center of the lens distortion, and an erasure process is performed to make the plurality of pixels in the work area disappear as a whole toward the origin. Distortion adjustment pattern data is read into the memory and expanded to bring the amount of positional movement of the plurality of pixels in the work area after the erasure process closer to the amount of positional movement due to lens distortion by the wide-angle camera, and the information on the amount of positional movement for each pixel in the work area after the erasure process is adjusted by referring to the distortion adjustment pattern data expanded in the memory. The position of each image pixel of the viewpoint image stored in the memory is moved by an amount of movement corresponding to the lens distortion in the wide-angle camera, thereby generating an image in the memory that has undergone lens distortion processing corresponding to the wide-angle camera.

[0010] A processing method for a three-dimensional virtual space lens distortion image generation apparatus according to one embodiment of the present invention is a processing method for an apparatus that generates a lens distortion image of a three-dimensional virtual space, which includes lens distortion by a wide-angle camera, as if the three-dimensional virtual space were captured by a wide-angle camera at a predetermined viewpoint, by performing image processing using an apparatus that generates a lens distortion image having a memory, wherein the apparatus acquires a viewpoint image of the three-dimensional virtual space as seen from the predetermined viewpoint,、 In the aforementioned memory The generated work area has pixels in the first quadrant to the fourth quadrant, with the center of the lens distortion as the origin. unfold, For each pixel of the viewpoint image displayed in the working area of ​​the memory, information is provided about the amount of positional movement that brings the positions of multiple pixels in the working area toward the origin, which is the center of lens distortion, thereby performing an erasure process that causes the multiple pixels in the working area to disappear as a whole toward the origin. Distortion adjustment pattern data is read into the memory and displayed to bring the amount of positional movement of the multiple pixels in the working area after the erasure process closer to the amount of positional movement due to lens distortion by the wide-angle camera. By referring to the distortion adjustment pattern data displayed in the memory, the information about the amount of positional movement for each pixel in the working area after the erasure process is adjusted so that the position of each image pixel of the viewpoint image displayed in the memory is moved by an amount of movement corresponding to the lens distortion in the wide-angle camera. The memory performs lens distortion processing to generate an image with lens distortion processing corresponding to the wide-angle camera, and outputs the image with lens distortion processing in the memory as a lens distortion image that includes distortion from the wide-angle camera.

[0011] A program according to one embodiment of the present invention is a program for a computer to implement a processing method for a device that generates a lens distortion image having memory, which performs image processing to generate a lens distortion image that includes lens distortion from a wide-angle camera, such as when a three-dimensional virtual space is captured by a wide-angle camera at a predetermined viewpoint, wherein the processing method acquires a viewpoint image of the three-dimensional virtual space viewed from the predetermined viewpoint. 、 In the aforementioned memory The generated work area has pixels in the first quadrant to the fourth quadrant, with the center of the lens distortion as the origin. unfold, For each pixel of the viewpoint image displayed in the working area of ​​the memory, information is provided about the amount of positional movement that brings the positions of multiple pixels in the working area toward the origin, which is the center of lens distortion, thereby performing an erasure process that causes the multiple pixels in the working area to disappear as a whole toward the origin. Distortion adjustment pattern data is read into the memory and displayed to bring the amount of positional movement of the multiple pixels in the working area after the erasure process closer to the amount of positional movement due to lens distortion by the wide-angle camera. By referring to the distortion adjustment pattern data displayed in the memory, the information about the amount of positional movement for each pixel in the working area after the erasure process is adjusted so that the position of each image pixel of the viewpoint image displayed in the memory is moved by an amount of movement corresponding to the lens distortion in the wide-angle camera. The memory performs lens distortion processing to generate an image with lens distortion processing corresponding to the wide-angle camera, and outputs the image with lens distortion processing in the memory as a lens distortion image that includes distortion from the wide-angle camera. [Effects of the Invention]

[0012] In this invention, a viewpoint image is acquired from a predetermined viewpoint in a three-dimensional virtual space, lens distortion processing corresponding to a wide-angle camera is performed on the viewpoint image stored in memory, and the image with lens distortion processing in memory is output as a lens-distorted image that includes distortion from the wide-angle camera. In this case, the present invention generates an image with lens distortion processing corresponding to a wide-angle camera in memory by moving the position of each image pixel of the viewpoint image stored in memory by an amount of movement corresponding to the lens distortion in the wide-angle camera. Thus, in the present invention, by moving the positions of the respective image pixels of the acquired perspective images, an image subjected to lens distortion processing corresponding to a wide-angle camera is generated. Therefore, the value of each pixel of the image generated in the present invention uses the value of each image pixel of the acquired perspective image itself. As a result, in the present invention, the value of each pixel does not become a value not included in the original perspective image as in the case of image conversion processing. Moreover, in the present invention, there is no need to execute a process such as image conversion that is likely to take a long time. In the present invention, an image including lens distortion by a wide-angle camera can be repeatedly generated so as to obtain real-time performance as in the case of repeatedly providing images to the vehicle control unit of a vehicle under development. In a simulation of driving a vehicle using the present invention in a three-dimensional virtual space, it becomes possible to provide a wide-angle lens distortion image to the vehicle control unit of a vehicle driving in the real space.

Brief Description of the Drawings

[0013] [Figure 1] FIG. 1 is an explanatory diagram of a control system of an automobile having an autonomous driving support or an automatic driving function. [Figure 2] FIG. 2 is an explanatory diagram of an example of an imaging image outside the vehicle by the wide-angle camera of FIG. 1. [Figure 3] FIG. 3 is an explanatory diagram of the configuration of a driving simulation system that can be used for the driving learning of the vehicle control device of the automobile of FIG. 1. [Figure 4] FIG. 4 is an explanatory diagram of a computer device that can be used as the lens distortion image generation device of FIG. 3. [Figure 5] FIG. 5 is a flowchart of the vehicle exterior image generation control by the driving simulation system of FIG. 3. [Figure 6] FIG. 6 is an explanatory diagram of an example of a perspective image of the three-dimensional virtual space generated by the driving simulator of FIG. 3. [Figure 7] FIG. 7 is an explanatory diagram of the correspondence relationship between the driving situation of an automobile and a plurality of wide-angle imaging images generated in order according to the progress of the driving situation. [Figure 8] FIG. 8 is a flowchart of generation control of a wide-angle lens distortion image by the lens distortion image generation device of FIG. 3. [Figure 9] FIG. 9 is an explanatory diagram of an example of a work area generated by the generation control of FIG. 8. [Figure 10] FIG. 10 is an explanatory diagram of an example of disappearance processing to the origin for a plurality of pixels in the work area of FIG. 9. [Figure 11] FIG. 11 is an explanatory diagram of an example of adjustment processing for the movement amount of each pixel in the work area after the disappearance processing to the origin of FIG. 10. [Figure 12] FIG. 12 is a schematic explanatory diagram of the movement amount given to each pixel in the work area of FIG. 8.

DETAILED DESCRIPTION OF THE INVENTION

[0014] [[ID=2"> Hereinafter, embodiments of the present invention will be described based on the drawings.

[0015] FIG. 1 is an explanatory diagram of a control system of an automobile 1 having an autonomous driving support or an automatic driving function. The automobile 1 is an example of a vehicle. The automobile 1 in FIG. 1 has, as its control system, a wide-angle camera 2, a vehicle control device 3, an actuator 4, a vehicle motion sensor 5, a position information generation device 6, and a vehicle communication device 7. The vehicle control device 3 controls the running of the automobile 1. The vehicle control device 3 includes an out-of-vehicle object detection device 11, a three-dimensional map device 12, and a running control device 13.

[0016] In the automobile 1 traveling in the real space, the wide-angle camera 2 repeatedly captures the periphery of the automobile 1, for example, the outside of the vehicle in front, at short intervals for running control, and generates a wide-angle captured image. The automobile 1 travels on, for example, a road in the real space. There may be out-of-vehicle objects such as other moving objects and falling objects around on the road. The wide-angle camera 2 can capture these out-of-vehicle objects. The wide-angle camera 2 outputs the signal of the captured image to the vehicle control device 3. Here, the wide-angle captured image may be, for example, a 4K resolution image or an 8K resolution image. The wide-angle camera 2 may capture images at a frequency of at least 24 FPS, preferably 120 FPS or higher. Furthermore, the control system of the vehicle 1 may be equipped with other external sensors, such as a LiDAR, along with the wide-angle camera 2, or in place of the wide-angle camera 2, to detect the surroundings of the vehicle 1.

[0017] Actuator 4 controls the operation of devices such as a drive system, braking system, and steering system (not shown) provided in the automobile 1 for driving. The drive system may consist of, for example, an engine, motor, and transmission. The braking system may consist of a brake system, regenerative braking system, etc. The control system of the automobile 1 may be provided with multiple actuators 4, such as an actuator 4 for driving, an actuator 4 for braking, and an actuator 4 for steering.

[0018] The vehicle motion sensor 5 detects the movement or behavior of the automobile 1. Examples of such vehicle motion sensors 5 include acceleration sensors and velocity sensors. The acceleration sensor detects acceleration in the three axes of the vehicle 1: up, down, left, right, forward, and backward. The acceleration sensor may also detect pitch, roll, and yaw angles as part of the movement of the vehicle 1. In addition, for example, the vehicle motion sensor 5 may detect the operating state of the aforementioned device whose operation is controlled by the actuator 4. The movement of the automobile 1 changes according to the operating state of the device or changes therein. The detected values ​​of the operating state of the device or changes therein can be converted into values ​​that indicate the movement of the automobile 1. The vehicle motion sensor 5 outputs the detection results regarding the driving or behavior of the automobile 1 to the driving control device 13.

[0019] The location information generation device 6 receives radio waves from, for example, a GNSS (Global Navigation Satellite System) satellite and generates the current location and time of the automobile 1 in which the location information generation device 6 is installed. The location information generation device 6 outputs the generated current location and current time to the driving control device 13.

[0020] The vehicle communication device 7 communicates with the outside of the vehicle 1. The vehicle communication device 7 establishes communication channels with, for example, base stations of a mobile communication network, base stations of a mobile information distribution network, other vehicles or other mobile devices, mobile terminals carried by pedestrians, communication satellites, etc. The vehicle communication device 7 may send and receive data with a server device (not shown) through the established communication channels. The vehicle communication device 7 outputs received data, for example, received from the server device, to the vehicle control device 3. The vehicle communication device 7 also transmits data from the vehicle control device 3 to the server device, for example.

[0021] The external object detection device 11 receives the image captured by the wide-angle camera 2 as input. The external object detection device 11 analyzes the images captured by the wide-angle camera 2 and detects external objects, such as other cars, that are within the imaging range (angle of view) of the wide-angle camera 2. Furthermore, the external object detection device 11 detects the distance from the vehicle's position to the external object, as well as the size and type of the external object (other cars, motorcycles, bicycles, pedestrians, children, fallen objects, etc.) based on the imaging position and imaging range in the captured image of the detected external object. The external object detection device 11 may, for example, estimate an estimated frame for the image range of other cars in the captured image, and estimate the position, direction, and distance (distance) of other cars relative to the vehicle based on the estimated estimated frame. The external object detection device 11 outputs information about the detected external object to the driving control device 13. The external object detection device 11 may also be implemented in the automobile 1 as a processing unit of the wide-angle camera 2.

[0022] The three-dimensional map device 12 has three-dimensional map information about, for example, the roads on which the automobile 1 travels. High-precision three-dimensional map information is often used in the automated driving or driver assistance of the automobile 1. The three-dimensional map device 12 outputs three-dimensional map information of the area around the current position of the automobile 1 to the driving control device 13.

[0023] The driving control device 13 is connected to a wide-angle camera 2, an external object detection device 11, a three-dimensional map device 12, an actuator 4, a vehicle motion sensor 5, a position information generation device 6, and the like. The driving control device 13 determines the driving status of the vehicle 1 based on the input information and generates a control signal to the actuator 4 to realize driving according to the determination result. In this case, the driving control device 13 may control the driving of the vehicle 1 based on the driver's operation or by autonomous driving. The driving control device 13 may also control the driving of the vehicle 1 to support driving of the vehicle 1 by manual driving or remote autonomous driving. For example, in autonomous driving or driver assistance, the driving control device 13 estimates the future driving conditions of the vehicle after a predetermined control cycle based on information about the vehicle's current driving conditions obtained from a wide-angle camera 2, an external object detection device 11, a three-dimensional map device 12, a vehicle motion sensor 5, a position information generation device 6, etc., generates a control signal corresponding to the estimated driving conditions, and outputs it to the actuator 4. The driving control device 13 generates control signals for, for example, lane keeping control to maintain the driving lane, distance keeping control to maintain a safe distance from the vehicle ahead, control to decelerate and turn left or right at intersections according to the route to the autonomous driving destination, control to stop at red lights, control to accelerate and start at green lights, and detour control to avoid external objects such as fallen objects, and outputs them to the actuator 4. As a result, the vehicle 1 can be driven with driver assistance or autonomous driving. The driving control device 13 can detect external objects such as other moving objects and fallen objects around the vehicle 1 as it travels in real space, based on images of the outside of the vehicle captured by the wide-angle camera 2, and execute driving control according to the detected external objects.

[0024] Figure 2 is an explanatory diagram illustrating an example of an image 20 of the outside of the vehicle captured by the wide-angle camera 2 in Figure 1. The wide-angle camera 2 is installed facing forward, for example, inside the passenger compartment of the automobile 1, and captures images of the area outside the vehicle in front of it.

[0025] The image 20 captured by the wide-angle camera 2 in Figure 2, which shows the area outside the vehicle, includes an image component of an intersection in front of the vehicle. Other vehicles 21 on the intersecting roads are traveling through the intersection from left to right in the image. Above the intersection is a monorail track 23. On the road the vehicle is traveling on, there is a pedestrian crossing pattern 22 just before the intersection. As shown in Figure 2, the camera in car 1 that images the area outside the vehicle uses a wide-angle lens to capture images with a wide field of view. Real-world objects outside the vehicle captured at the periphery of the wide-angle image will have significantly more distorted image components compared to objects outside the vehicle captured at the center of the wide-angle image. Wide-angle lens distortion occurs in the wide-angle image. Furthermore, it is desirable that the wide-angle camera 2 used to capture images of the outside of the vehicle for driving control in the automobile 1 has a field of view of at least 120 degrees, and preferably one with a field of view of about 170 degrees.

[0026] Incidentally, in the development of such a vehicle 1, the driving control device 13 needs to appropriately determine detection patterns and detection parameters in order to improve the accuracy of detection of external objects by the external object detection device 11. In addition, it is necessary to appropriately determine the values ​​of the control signals that the driving control device 13 generates according to the driving conditions. It is also possible to determine these setting information values ​​using the results of a three-dimensional virtual space driving simulation, rather than having the developer fine-tune them based on driving tests in real space. The driving control device 13 may also use machine learning based on the results of the driving simulation. By using driving simulation, the developer can set highly accurate settings for the driving control device 13 while reducing the need for real-world verification tests. The developer can also set settings for the driving control device 13 that are usable well even in driving conditions where verification tests are not possible.

[0027] However, currently available simulation systems cannot obtain images of the outside of a vehicle 1 traveling in a three-dimensional virtual space that include wide-angle lens distortion, as would be obtained from a wide-angle camera 2 mounted on the vehicle 1. Simulation systems fundamentally seek realism in a three-dimensional virtual space and therefore provide viewpoint images that do not contain distortion. Such simulation systems cannot be directly used for developing the driving control of vehicle 1. For example, viewpoint images captured from a predetermined viewpoint in a three-dimensional virtual space created by simulation are inherently free of distortion, as they are primarily intended to pursue image realism. In such distortion-free images, the balance between the image size of other cars in the central part of the viewpoint image and the image size of other cars in the outer edges of the viewpoint image is significantly different from images captured by wide-angle camera 2, where lens distortion occurs. Even if the vehicle control device 3 of automobile 1 is given a viewpoint image that is significantly different from the image captured in the actual vehicle, and a simulation is performed, it is not possible to obtain reliable driving simulation results that can be used for driving in real space. For example, if the size or position of external objects in an image changes, not only will the accuracy of recognizing the size and type of external objects decrease, but significant errors will also occur in the direction and distance of external objects relative to the vehicle itself. Thus, in the development of the vehicle control device 3 for a car 1 that travels in real space, it is required to be able to provide wide-angle lens distortion images when the car 1 is driven in a three-dimensional virtual space.

[0028] Figure 3 is an explanatory diagram of the configuration of a driving simulation system 30 that can be used for driving learning of the vehicle control device 3 of the automobile 1 shown in Figure 1. The driving simulation system 30 shown in Figure 3 includes a vehicle simulator 31, a driving simulator 32, and a lens distortion image generation device 33. The driving simulation system 30 performs a driving simulation in which the driving control device 13 of the automobile 1 shown in Figure 1 is driven in a three-dimensional virtual space.

[0029] The vehicle simulator 31 is connected to the driving control device 13 of the automobile 1. The vehicle simulator 31 receives control information generated by the driving control device 13 to be provided to the actuator 4. The vehicle simulator 31 generates position information and attitude information of the controlled vehicle 1 based on the control information. The vehicle simulator 31 outputs the generated position information and attitude information of the controlled vehicle 1 to the vehicle 1's driving control device 13, three-dimensional map device 12, and driving simulator 32. Furthermore, the vehicle simulator 31 may store information on a virtual driving scenario for the automobile 1 in a three-dimensional virtual space, generate operation information based on the driving scenario from the position of the automobile 1 after control, and output the generated operation information to the driving control device 13. The driving scenario may be, for example, information on a driving path from point A to point B in a three-dimensional virtual space. In this case, the driving control device 13 of the automobile 1 will basically execute control to drive from point A to point B in the three-dimensional virtual space according to the driving scenario, and will execute driving control in a manner that avoids interference with external objects such as other automobiles during that driving control.

[0030] The driving simulator 32 is connected to the vehicle simulator 31. The driving simulator 32 receives, along with the position and attitude information of the vehicle 1 after control from the vehicle simulator 31, as well as scenario data and high-precision spatial data of other moving objects moving in a three-dimensional virtual space. The driving simulator 32 generates a three-dimensional virtual space around the position of the controlled vehicle 1 based on the vehicle 1's position information and high-precision spatial data. The three-dimensional virtual space may include the road on which the vehicle 1 is traveling and surrounding structures. The driving simulator 32 also places other moving objects in the three-dimensional virtual space based on scenario data for other moving objects. The driving simulator 32 sets the position and orientation of the viewpoint of the wide-angle camera 2 of the controlled vehicle 1 in the generated three-dimensional virtual space based on the attitude information of the controlled vehicle 1. Here, the driving simulator 32 may fine-tune the position and orientation of the viewpoint of the wide-angle camera 2 of the controlled vehicle 1 based on the difference between the mounting position of the wide-angle camera 2 on the vehicle 1 and the mounting position of the position information generation device 6 on the vehicle 1. The vehicle simulator 31 may also generate the position information and attitude information of the viewpoint of the wide-angle camera 2 of the controlled vehicle 1 along with the position information and attitude information of the controlled vehicle 1 and output them to the driving simulator 32. The driving simulator 32 generates a viewpoint image by capturing the three-dimensional virtual space from a set viewpoint. The viewpoint image may have the same field of view as the wide-angle camera 2, or a wider field of view. Roads, structures, and other moving objects in the three-dimensional virtual space are included in the viewpoint image at a position directly viewed from the viewpoint, and at the size directly viewed from the viewpoint. The roads, structures, and other moving objects in the three-dimensional virtual space are included in the viewpoint image as highly realistic image components with high resolution and virtually no distortion.

[0031] The lens distortion image generation device 33 is connected to the driving simulator 32. The viewpoint image generated by the driving simulator 32 is input to the lens distortion image generation device 33. The lens distortion image generation device 33 converts the viewpoint image into a wide-angle lens distortion image. The lens distortion image generation device 33 outputs the generated wide-angle lens distortion image to the vehicle control device 3's external object detection device 11 and driving control device 13, etc. As a result, the external object detection device 11 and the driving control device 13 of the vehicle control device 3 receive an image that captures a three-dimensional virtual space and includes wide-angle lens distortion equivalent to that obtained when captured by the wide-angle camera 2 of the automobile 1. The vehicle control device 3 of the automobile 1, which is traveling in real space, can perform control to drive the automobile 1 in a three-dimensional virtual space while obtaining a wide-angle lens distortion image.

[0032] Furthermore, the automobile 1 may be equipped with multiple cameras for capturing images of the area outside the vehicle. In this case, the driving simulation system 30 may have the same number of lens distortion image generation devices 33 as the number of cameras installed in the automobile 1. In this case, the multiple lens distortion image generation devices 33 each generate a lens distortion image in a three-dimensional virtual space that contains distortion components equivalent to those of the wide-angle camera 2 corresponding to each device. The driving simulator 32 may also be equipped with the same number of cameras as the automobile 1.

[0033] Figure 4 is an explanatory diagram of a computer device 40 that can be used as the lens distortion image generation device 33 in Figure 3. The computer device 40 in Figure 4 has input / output ports 41, a timer 42, a CPU 45, RAM 44, a storage device 43, and an internal bus to which these are connected.

[0034] The input / output port 41 is the physical input / output of the computer device 40. The vehicle control device 3 and other computer devices 40 may be connected to the input / output port 41. Timer 42 measures time and duration. The storage device 43 may be, for example, a ROM, a recording disk, or a non-volatile semiconductor memory. The storage device 43 records the program 47 executed by the CPU 45 and data. In Figure 4, the storage device 43 shows gradient pattern data 48. RAM44 is a volatile semiconductor memory. The CPU 45 reads the program 47 recorded in the storage device 43 into the RAM 44, expands it, and executes it. In this way, the CPU 45 functions as the control unit of the computer device 40. The CPU 45, as the control unit, then uses the RAM 44 to perform image processing and generates a lens distortion image that includes lens distortion from the wide-angle camera 2, as if the three-dimensional virtual space were captured by the wide-angle camera 2 at a predetermined viewpoint. At this time, the CPU 45, as the control unit, reads data such as gradient pattern data 48 recorded in the storage device 43 into the RAM 44, expands it, and uses it for processing based on the program 47.

[0035] The vehicle simulator 31 and driving simulator 32 of the aforementioned driving simulation system 30 can also be implemented in a computer device 40 as shown in Figure 4, similar to the lens distortion image generation device 33. Furthermore, each of the above-mentioned devices 31 to 33 in the driving simulation system 30 may be implemented in a single computer device 40 as shown in Figure 4.

[0036] Figure 5 is a flowchart of the external image generation control by the driving simulation system 30 shown in Figure 3. The CPU 45 in Figure 4, as the CPU 45 of the driving simulation system 30, is capable of repeatedly executing the external image generation control shown in Figure 5.

[0037] In step ST1, the CPU 45 determines whether or not it has acquired new vehicle information from the vehicle simulator 31 as the driving simulator 32. If new vehicle information has not been acquired, the CPU 45 repeats this process. If new vehicle information is acquired, the CPU 45 proceeds to step ST2.

[0038] In step ST2, the CPU 45 generates a new three-dimensional virtual space around the position of the car 1 acquired in step ST1, which serves as the driving simulator 32.

[0039] In step ST3, the CPU 45 sets the position and orientation of the viewpoint of the wide-angle camera 2 of the automobile 1 in the three-dimensional virtual space generated in step ST2 as the driving simulator 32.

[0040] In step ST4, the CPU 45 generates a viewpoint image, which is captured from the viewpoint of the three-dimensional virtual space, as the driving simulator 32.

[0041] In step ST5, the CPU 45, acting as a lens distortion image generation device 33, converts the viewpoint image into a wide-angle lens distortion image. This generates an image that includes wide-angle lens distortion equivalent to that captured by the wide-angle camera 2 of the automobile 1, which captures the area outside the vehicle.

[0042] In step ST6, the CPU 45, acting as a lens distortion image generation device 33, outputs the generated wide-angle image including lens distortion to the vehicle control device 3.

[0043] In step ST7, the CPU 45 determines whether or not to terminate the external image generation control. For example, the CPU 45 may decide to terminate the external image generation control if the elapsed time since the last acquisition of vehicle information, as measured by the timer 42, is equal to or greater than the vehicle information acquisition cycle. If the external image generation control is not terminated, the CPU 45 returns to step ST1. The CPU 45 repeats the processes from step ST1 to step ST7 until it determines that the external image generation control should be terminated. By repeating the external image generation control in this manner, the driving simulation system 30 can continuously output images from the wide-angle camera 2 outside the vehicle, which change in accordance with the driving stage of the automobile 1 in the three-dimensional virtual space, to the vehicle control device 3 of the automobile 1. When the CPU 45 determines that the external image generation control has finished, it terminates this control.

[0044] Figure 6 is an explanatory diagram illustrating an example of a viewpoint image 50 of the three-dimensional virtual space generated by the driving simulator 32 shown in Figure 3. The viewpoint image 50 of the three-dimensional virtual space in Figure 6 corresponds to the image 20 of the outside of the vehicle captured by the wide-angle camera 2 in real space in Figure 2. Furthermore, the viewpoint image 50 only needs to have at least the same number of pixels as the image 20 captured by the wide-angle camera 2, and preferably has a greater number of pixels than the image 20 captured by the wide-angle camera 2.

[0045] In Figure 6, the other vehicles 51 on the intersecting roads at the intersection have image components that are close to their original shape with minimal distortion in the viewpoint image 50. Furthermore, the monorail track 53 above the intersection and the road surface pattern 52 of the pedestrian crossing are almost entirely free of distortion and retain their original, easily visible shape. Even if such a wide-angle viewpoint image 50 without lens distortion is provided to the vehicle control device 3 of the automobile 1, it is difficult to obtain good simulation results for the vehicle control device 3, which actually performs control based on the captured image 20 that includes wide-angle lens distortion.

[0046] Figure 7 is an explanatory diagram illustrating the correspondence between the driving conditions of car 1 and multiple wide-angle captured images 54-56 that are generated sequentially as the driving conditions progress. Figure 7 shows images 54-56 captured sequentially by multiple wide-angle cameras 2 taken by the car 1 while it is moving in real space. Time flows from T1 to T3.

[0047] In the real-world driving situation 60 shown in Figure 7, car 1 is driving in the center lane of a straight, three-lane road. In addition to its own car, other cars 61-63 are driving in the center lane, right lane, and left lane, respectively. The images captured by the wide-angle camera 2 include images 57-59 of other cars 61-63 in each lane, accompanied by wide-angle lens distortion, as shown in the example of schematic real-space images 54-56 from timing T1 to timing T3 in the figure. In these images, cars are schematically represented by rectangles. Furthermore, the images 59 of the other car 63 in the right lane and the image 58 of the other car 62 in the left lane are located at the periphery of the image, and are therefore significantly distorted compared to the image 57 of the other car 61 in the center lane, which is located in the center of the image.

[0048] Furthermore, Figure 7 also shows a schematic viewpoint image 70 of the three-dimensional virtual space corresponding to the real space in Figure 7. In the viewpoint image 70 of Figure 7, the images 71-73 of other cars 61-63 in each lane included in the image show less distortion compared to the images 57-59 in the real-space captured images 54-56. The statue 71 of another vehicle 61 in the center lane is smaller in size than the statue 73 of another vehicle 63 in the right lane, or the statue 72 of another vehicle 62 in the left lane. Although the other car 61 in the center lane is actually located at the same distance from car 1 as the other car 62 in the right lane and the other car 63 in the left lane, in the viewpoint image 70, its image 71 is relatively small, causing it to be perceived as being further away than the other car 63 in the right lane and the other car 62 in the left lane.

[0049] Figure 8 is a flowchart showing the control of generating wide-angle lens distortion images using the lens distortion image generation device 33 shown in Figure 3. The CPU 45 in Figure 4, as the CPU 45 of the lens distortion image generation device 33, may repeatedly perform the generation control of the wide-angle lens distortion image shown in Figure 8 in step ST5 of Figure 4, for example.

[0050] In step ST11, the CPU 45 determines whether or not there are any unprocessed viewpoint images, which are generated repeatedly by the driving simulator 32 and have the position of the wide-angle camera 2 of the car 1 as the viewpoint. If there are no unprocessed viewpoint images, CPU45 repeats this process. When the driving simulator 32 generates a new viewpoint image, the CPU 45 determines that there is an unprocessed viewpoint image and proceeds to step ST12.

[0051] In step ST12, the CPU 45, acting as a viewpoint image acquisition unit, acquires a viewpoint image newly generated by the driving simulator 32 and stores it in the RAM 44. Subsequently, the CPU 45, acting as a lens distortion processing unit, begins executing lens distortion processing corresponding to the wide-angle camera 2 on the viewpoint image stored in the RAM 44.

[0052] In step ST13, the CPU 45 creates a work area in the RAM 44. The work area should preferably be created with a number of pixels equal to or greater than the number of image pixels of the viewpoint image. The working area is the region used to obtain the amount of movement of each image pixel in the viewpoint image, in order to obtain a lens distortion image from the viewpoint image.

[0053] Figure 9 is an explanatory diagram of an example of a work area 80 generated by the generation control shown in Figure 8. The working area 80 in Figure 9 is a planar region defined by the U-axis (horizontal axis) and the V-axis (vertical axis). The center of the working area coincides with the origin of the planar region. Here, the origin can be the center of lens distortion, and in this case, it becomes the processing center for the vanishing process described later. Furthermore, the work area 80 comprises the first to fourth quadrants surrounding the origin in the planar region. The size of each quadrant of the work area 80 may be, for example, the same as the viewpoint image. The sizes of the regions from the first quadrant to the fourth quadrant may be the same. In such a work area 80, multiple pixels corresponding to all of the multiple image pixels of the viewpoint image can be obtained.

[0054] In step ST14, the CPU 45 sets the effective work range for the work area generated in RAM 44. The effective working range may be set to a shape similar to the viewpoint image or the lens distortion image, so as to correspond to these images. The effective working range should be at least the range corresponding to the image captured by the wide-angle camera 2, and preferably the range that will be included in the lens-distorted image by the process of moving the image pixels of the viewpoint image. In Figure 9, the effective work area 81 is defined as a portion of the work area.

[0055] In step ST15, the CPU 45 performs origin vanishing processing for each pixel of the work area generated in RAM 44. This assigns an initial value for the amount of movement to each pixel in the work area.

[0056] Figure 10 is an explanatory diagram illustrating an example of vanishing to the origin for multiple pixels in the working area 80 of Figure 9. Figure 10 shows the movement vectors of each pixel toward the origin, superimposed on the work area 80 in Figure 9, indicated by dashed arrows. The amount of movement of pixels closer to the origin is smaller than the amount of movement of pixels farther from the origin. The amount of movement of a pixel increases as its distance from the origin increases. In this way, the CPU 45, through origin vanishing processing, provides information on the amount of positional movement that moves the positions of multiple pixels in the work area 80 toward the origin, which is the center of lens distortion. The amount of positional movement of each pixel increases as the distance from the origin to that pixel's position increases. As a result, multiple pixels in the work area 80 are given an initial value for the amount of movement that moves them as a whole toward the origin, which acts as the vanishing point in perspective.

[0057] From step ST16, the CPU 45 starts the process of adjusting the amount of movement. The CPU 45 first reads the one-dimensional gradient pattern data 48 from the storage device 43 and expands it into RAM 44, which is used as memory. The gradient pattern data 48 is a type of distortion adjustment pattern data used to bring the amount of positional movement of multiple pixels in the work area after the vanishing process closer to the amount of positional movement due to lens distortion by the wide-angle camera 2.

[0058] In step ST17, the CPU 45 selects one pixel from the work area after the disappearance process for the movement amount adjustment process. The CPU 45 selects one pixel from the work area for which the movement amount adjustment process has not been performed. The CPU 45 can select all of the multiple pixels in the work area one by one in sequence, for example, by selecting pixels one by one from the side of the origin in a range extending linearly from the origin in the work area after the vanishing process, and then rotating the direction of the line sequentially within a 360-degree range around the origin.

[0059] In step ST18, the CPU 45 calculates the normalized distance from the origin in the working area for the selected pixel by dividing the distance from the origin by a predetermined reference distance. Here, the reference distance may be, for example, half the length of the diagonal of the effective working area set in step ST14. This allows the normalized distance for all pixels in the working area included in the effective working area to take values ​​in the range of 0 to 1.

[0060] In step ST19, the CPU 45 obtains the gradient value of the position corresponding to the normalized distance from the one-dimensional gradient pattern data 48. The obtained gradient value can be used as an adjustment rate for the amount of movement of each pixel's position.

[0061] In step ST20, the CPU 45 multiplies the adjustment rate obtained from the one-dimensional gradient pattern data 48 by the amount of movement assigned as an initial value to the selected pixel to calculate a new amount of movement for the selected pixel. Then, the CPU45 updates the movement amount of the selected pixel based on the calculated movement amount. As a result, the selected pixels will contain information about the amount of movement adjusted by the gradient values ​​contained in the one-dimensional gradient pattern data 48.

[0062] In step ST21, the CPU 45 determines whether or not the selection of pixels for the work area after the erasure process has been completed. For example, the CPU 45 may determine that pixel selection is complete if it has finished selecting all pixels in the work area, or if it has finished selecting all pixels in the work area included in the valid work range. If the selection of pixels is not yet complete, the CPU 45 returns to step ST17. The CPU 45 repeats the process from step ST17 to step ST21 until the selection of all pixels within a predetermined range of the work area is complete. Once the selection for all pixels is complete, the CPU 45 finishes the adjustment process of the movement amount from step ST16 to step ST21 and proceeds to step ST22. Thus, when the CPU 45 proceeds to step ST22, all of the multiple pixels in the work area expanded in RAM 44 contain information about the amount of movement that has been adjusted using the gradient value contained in the one-dimensional gradient pattern data 48 as an adjustment factor.

[0063] In step ST22, the CPU 45 executes a process to move the position of each image pixel in the viewpoint image loaded in RAM 44. At this time, the CPU 45 refers to the information on the amount of positional movement for each pixel in the work area generated in RAM 44. As a result, the position of each image pixel in the viewpoint image shifts to correspond to the lens distortion caused by the wide-angle camera 2. The RAM 44 then generates an image that has been processed to correct for the lens distortion caused by the wide-angle camera 2. Subsequently, the CPU 45 completes the lens distortion processing corresponding to the wide-angle camera 2 on the viewpoint image loaded into RAM 44, and proceeds to step ST23.

[0064] In step ST23, the CPU 45 outputs the image with lens distortion correction applied, which is generated in the RAM 44, to the vehicle control device 3 as an output unit. The image, which has undergone wide-angle lens distortion processing, is output to the vehicle control device 3 as a lens distortion image that includes distortion from the wide-angle camera 2.

[0065] Note that the gradient pattern data 48 that is read into RAM 44 and expanded in the process shown in Figure 8 is one-dimensional. In contrast, the gradient pattern data 48, which is loaded into RAM 44 and expanded, can also be made two-dimensional by corresponding it to a planar area as a working area. By making the gradient pattern data 48 one-dimensional, as in this embodiment, the time required for reading and unfolding in step ST16 is significantly reduced. This reduction in reading and unfolding time can more than offset the increase in processing time caused by making the gradient pattern data 48 one-dimensional in the processing from step ST17 to step ST21. As a result, the processing time from step ST16 to step ST21 in Figure 8 can be significantly reduced compared to when the same processing is performed using two-dimensional gradient pattern data. Furthermore, the processing time shown in Figure 8 is significantly reduced, enabling the driving simulation system 30 to repeatedly output images to the vehicle control device 3 of the automobile 1 that have the same high resolution as the wide-angle camera 2 installed in the automobile 1, and that include wide-angle lens distortion with the same field of view as the wide-angle camera 2, at the same frequency as the wide-angle camera 2 installed in the automobile 1.

[0066] Figure 11 is an explanatory diagram illustrating an example of the adjustment process for the amount of movement of each pixel in the work area after the vanishing process to the origin in Figure 10. Figure 11 shows the effective working range 81 set for the working area 80, and the one-dimensional gradient pattern data 48. A circumscribed circle 83 is shown around the effective working area 81, circumscribing the rectangular effective working area 81. The radius of the circumscribed circle 83 is the reference distance L used to normalize the distance from the origin for pixels selected from the effective working area 81. The circumscribed circle 83 may also circumscribe the rectangular working area 80.

[0067] The one-dimensional gradient pattern data 48 has multiple pattern pixels arranged in one dimension. The number of pattern pixels may be the same as the number of pixels in the working area 80 at the radius of the circumscribed circle 83. The multiple pattern pixels then have gradient values ​​that increase monotonically within a range from, for example, 0 for black to 1 for white, according to the arrangement order of the one-dimensional gradient pattern data 48. The gradient pattern graph 91 shown in Figure 11 displays a gradient curve 92. The horizontal axis of graph 91 represents the normalized distance, and the vertical axis represents the adjustment rate of the displacement. The gradation curve 92 is a curve connecting the gradation values ​​of multiple pattern pixels contained in the one-dimensional gradation pattern data 48. The gradation curve 92 extends from the origin of the graph in an upward-right direction. The gradation curve 92 is modeled after the difference between the image distortion caused by the amount of positional movement information in multiple pixels of the work area 80 after the vanishing process, and the lens distortion caused by the wide-angle camera 2.

[0068] In this case, the CPU 45 calculates the normalized distance for the selected pixel using the radius of the circumscribed circle 83 in Figure 11 as the reference distance L. The CPU 45 calculates the normalized distance by dividing the length PR of the line segment from the origin to the selected pixel by the reference distance L, which is the radius of the circumscribed circle. The CPU 45 uses the calculated normalized distance to obtain the gradient value of one pattern pixel from the one-dimensional gradient pattern data 48, as schematically shown in the gradient pattern graph 91 in Figure 11. In the one-dimensional gradient pattern data 48, the black edge is considered to be the origin. In this case, the longer the normalized distance, the larger the gradient value obtained by the CPU 45. The CPU 45 then adjusts the movement amount of the selected pixel by performing a calculation that uses the acquired gradient value as the adjustment rate for the movement amount. The CPU 45 multiplies the movement amount initially given to the selected pixel by the adjustment rate. The CPU 45 updates the movement amount of the selected pixel with the movement amount obtained by the multiplication.

[0069] Figure 12 is a schematic diagram illustrating the amount of movement applied to each pixel of the work area 80 in Figure 8. Figure 12 shows a graph where the horizontal axis is the normalized distance and the vertical axis is the displacement normalized by the maximum displacement. Figure 12 also shows a straight line (line 100 showing the amount of movement due to origin vanishing processing) that represents the amount of positional shift assigned to each pixel as an initial value by origin vanishing processing, and a curve 101 that represents the amount of movement due to wide-angle lens distortion. Here, the maximum initial value given to each pixel during the origin vanishing process is the value of the positional displacement given to the pixel furthest from the origin, which is a pixel in the outer edge of the image. This maximum positional displacement becomes 1 when normalized by the maximum displacement value. Furthermore, in the origin vanishing process, the minimum initial value given to each pixel is the amount of positional displacement given to the pixel closest to the origin. When this maximum positional displacement is normalized by the maximum value of the displacement, it becomes a positive value close to 0. As a result, in the viewpoint image, similar to the image captured by the wide-angle camera 2 which includes lens distortion, pixels in the central part of the image component near the origin move by a small amount, while pixels in the outer part of the image component farther from the origin move by a large amount. Then, the gradient value of each pattern pixel in the one-dimensional gradient pattern data 48 can be, for example, a value corresponding to the ratio between the value on the straight line 100, which shows the amount of movement due to the origin vanishing process in Figure 12, and the value on the curve 101, which shows the amount of movement due to wide-angle lens distortion.

[0070] When trying to obtain a wide-angle lens-distorted image from a viewpoint image, basically, all that is needed is to prepare a value for the amount of positional displacement corresponding to the wide-angle lens distortion for each normalized distance of each image pixel. For example, distortion adjustment Pattern data The one-dimensional gradient pattern data 48 can simply contain the values ​​of the positional displacement corresponding to such wide-angle lens distortion. However, in wide-angle lens distortion images, the distortion is not uniform throughout the entire image. For example, as is clear from comparing Figure 2 and Figure 6, the distortion is small in the center of the image and large in the periphery. Therefore, the value of the positional shift corresponding to wide-angle lens distortion ranges widely from very small values ​​close to 0 to very large values ​​greater than 1. Distortion adjustment is performed using values ​​within this range. Pattern data In order to store a large number of digits in each pattern pixel, distortion adjustment is necessary. Pattern data The amount of data must be increased. As the amount of data increases, loading and unpacking it will take even more time.

[0071] Therefore, in this embodiment, for each pixel in the work area, the amount of positional movement due to the origin vanishing process is assigned as an initial value according to the straight line 100 which indicates the amount of movement due to the origin vanishing process. Furthermore, in this embodiment, the adjustment ratio value 102 for converting the assigned initial amount of movement into the value of the curve 101 which indicates the amount of movement of wide-angle lens distortion is performed by multiplication, and distortion adjustment Pattern dataThis is stored in the one-dimensional gradient pattern data 48. As a result, the value of each pattern pixel in the one-dimensional gradient pattern data 48 falls within the range of 0 to 1. Compared to the case where the values ​​of the movement amount described above are used as is, distortion adjustment Pattern data This reduces the amount of data. The value of each pattern pixel is limited to a small number of decimal places.

[0072] As described above, in this embodiment, a viewpoint image of a three-dimensional virtual space viewed from a predetermined viewpoint is acquired and loaded into RAM44, lens distortion processing corresponding to the wide-angle camera 2 is performed on the loaded viewpoint image, and a lens distortion image including lens distortion is generated and output in RAM44. In this embodiment, the position of each image pixel of the viewpoint image loaded into RAM44 is moved by an amount of movement corresponding to the lens distortion in the wide-angle camera 2, thereby performing lens distortion processing corresponding to the wide-angle camera 2 in RAM44. In this embodiment, by moving the position of each image pixel in the acquired viewpoint image, an image with lens distortion processing corresponding to the wide-angle camera 2 is generated without changing the color component values ​​of each image pixel. Therefore, the values ​​of each image pixel in the image generated by this embodiment use only the values ​​of each image pixel in the acquired viewpoint image. As a result, in this embodiment, the generated wide-angle lens-distorted image does not include color components whose values ​​are not included in the original viewpoint image, as would occur if the image itself were transformed. Color component values ​​not present in the original viewpoint image can blur the outlines of other vehicles in the image, reducing the accuracy of image analysis and recognition of external objects such as other vehicles. If machine learning is performed on blurred outlines or parameter settings are adjusted, the developed driving control device 13 may be affected when it is actually implemented in the vehicle 1 and driving control is performed in real space. This could lead to errors in the size and distance of other vehicles being recognized. In a vehicle 1 driving in real space, it is crucial to reliably detect external objects such as other moving objects and falling objects around the vehicle based on the external images captured by the wide-angle camera 2, and to perform driving control according to the detected external objects. Furthermore, in this embodiment, a wide-angle lens-distorted image is generated by shifting the positions of image pixels included in the viewpoint image. Therefore, it eliminates the need for computationally intensive and time-consuming processing, such as when the viewpoint image itself is converted. As a result, this embodiment makes it possible to repeatedly generate images that include wide-angle lens distortion in order to achieve real-time performance, even in applications that require real-time performance, such as repeatedly providing wide-angle lens distortion images to the driving control device 13 of a vehicle 1 under development. This embodiment can be suitably used in simulations of driving control for a driving control device 13 of an automobile 1 to periodically provide wide-angle lens distortion images in a real-time manner.

[0073] In particular, in this embodiment, a work area is generated in RAM44, consisting of multiple pixels corresponding to at least all of the multiple image pixels of the viewpoint image, with the center of the lens distortion as the origin and the pixels in the first quadrant to the fourth quadrant. In addition, in this embodiment, information is generated for each pixel of the work area generated in RAM44, indicating the amount of positional movement for each pixel corresponding to the lens distortion by the wide-angle camera 2. Furthermore, in this embodiment, by referring to the information on the amount of positional movement for each pixel of the work area generated in RAM44, a process is executed to move the position of each image pixel of the viewpoint image displayed in RAM44 by the amount of movement corresponding to the lens distortion in the wide-angle camera 2. Thus, in this embodiment, instead of directly performing various processes on the viewpoint image itself, a work area corresponding to the viewpoint image is generated, and within that work area, the amount of movement corresponding to the lens distortion in the wide-angle camera 2 is obtained for each pixel, and in the final stage, this is reflected in the movement of the position of each image pixel in the viewpoint image. For this reason, in this embodiment, these series of processes can be executed at high speed and in a stable time without being affected by the amount of data in the viewpoint image. In this embodiment, when generating an image including wide-angle lens distortion, the amount of movement of the position of each image pixel is obtained at high speed and in a stable processing time in a work area unaffected by the viewpoint image, and real-time performance equivalent to the imaging period in an actual vehicle can be obtained as the period of the wide-angle lens distortion image repeatedly provided to the driving control device 13 of the automobile 1 under development.

[0074] Furthermore, in this embodiment, each pixel of the work area generated in RAM44 is given information about the amount of positional movement that moves the positions of multiple pixels in the work area toward the origin, which is the center of lens distortion, through an annihilation process. As a result, each pixel of the work area can obtain information about the amount of positional movement that is not directly related to the viewpoint image as an initial value. Furthermore, in this embodiment, the initial positional displacement values ​​given to each pixel in the work area are not common to all of the multiple pixels in the work area, but are set to different values ​​that move the multiple pixels in the work area collectively toward the origin, which acts as the vanishing point in perspective. As a result, the distortion adjustment pattern data used in the subsequent process of adjusting the positional displacement of each pixel in the work area can use a ratio of 1 or less that brings the positional displacement of the multiple pixels in the work area after the vanishing process closer to the positional displacement due to lens distortion by the wide-angle camera 2. In contrast, if the initial positional displacement given to each pixel in the work area is a common value for all pixels in the work area, for example, 0, then the distortion adjustment pattern data used in subsequent processing must be a value that corresponds to the positional displacement caused by the lens distortion from the wide-angle camera 2. The lens distortion from the wide-angle camera 2 is large near the outer edges of the image and small near the center of the image. For this reason, each value in the distortion adjustment pattern data needs to have many digits capable of storing values ​​ranging from very small values ​​close to 0 to large values ​​greater than or equal to 1. In this embodiment, an initial value is given to each pixel that moves multiple pixels in the work area collectively toward the origin, which acts as the vanishing point in perspective. Therefore, each value in the distortion adjustment pattern data only needs to be a value that adjusts the amount of movement, and values ​​in the range of 0 to 1 can be used. In this embodiment, the number of digits in each value of the distortion adjustment pattern data can be reduced, thereby reducing the amount of data and the data expansion time.

[0075] Furthermore, in this embodiment, distortion adjustment Pattern data As a result, a one-dimensional gradient pattern data 48 is used, in which the value monotonically increases within the range of 0 to 1, following the difference between the image distortion caused by the amount of positional movement information in multiple pixels of the work area after the vanishing process and the lens distortion from the wide-angle camera 2. The one-dimensional gradient pattern data 48 can be made to have a length L that is half the length of the diagonal of the circumscribed circle 83 of the two-dimensional effective work area 81 (work area 80). In contrast, distortion adjustment Pattern dataFor example, it is possible to use two-dimensional gradient pattern data corresponding to a two-dimensional effective working range 81 (working area 80). However, the amount of data in the two-dimensional gradient pattern data is significantly larger than that of the one-dimensional gradient pattern data 48. The process of reading and expanding it in RAM 44 takes an excessive amount of time. The inventors' verification has shown that this processing time from reading to expansion is longer than the time required for the series of processes from reading to obtaining the value in this embodiment. In this embodiment, since one-dimensional gradient pattern data 48 is used and the above-described series of calculations are performed, the total processing time can be reduced compared to the case where two-dimensional gradient pattern data is used.

[0076] Furthermore, in this embodiment, a similar effective working area corresponding to the lens distortion image is set for the working area generated in RAM44, and the normalized distance, obtained by normalizing the length PR of the line segment from the origin to the pixel with respect to the radius L of the circumscribed circle of the effective working area, is used to obtain the position value corresponding to the normalized distance from the one-dimensional gradient pattern data 48. In this embodiment, by normalizing the position of each pixel in the work area, an appropriate adjustment ratio value corresponding to the distance of each pixel's position from the origin can be obtained from the one-dimensional gradient pattern data 48. In this embodiment, the one-dimensional gradient pattern data 48 can be used in common for the process of obtaining the adjustment ratio for all of the multiple pixels in the work area. In this embodiment, one-dimensional gradient pattern data 48 is used to obtain an adjustment ratio value for the amount of positional movement of all multiple pixels in the work area, thereby obtaining the amount of movement corresponding to the lens distortion by the wide-angle camera 2 for all multiple image pixels of the viewpoint image.

[0077] In this embodiment, which has these effects, a viewpoint image is acquired from a predetermined viewpoint in a three-dimensional virtual space, lens distortion processing corresponding to the wide-angle camera 2 is performed on the viewpoint image expanded in memory 44, and the image with lens distortion processing in memory 44 is output as a lens-distorted image that includes distortion from the wide-angle camera 2. In this embodiment, the position of each image pixel of the viewpoint image expanded in memory 44 is moved by an amount of movement corresponding to the lens distortion in the wide-angle camera 2, thereby generating an image with lens distortion processing corresponding to the wide-angle camera 2 in memory 44. In this embodiment, by moving the position of each image pixel in the acquired viewpoint image, an image with lens distortion processing corresponding to the wide-angle camera 2 can be generated. In this embodiment, the values ​​of each pixel in the generated image are the same as the values ​​of each image pixel in the acquired viewpoint image. As a result, in this embodiment, the values ​​of each pixel are not excluded from the original viewpoint image, as can happen when image conversion processing is performed. In this embodiment, it is less likely that the outline of the imaged object will be blurred and the image will become unclear. Furthermore, in this embodiment, no conversion process is performed to obtain new pixels from the image, so no such time is required. In this embodiment, an image including lens distortion from a wide-angle camera 2 of a vehicle such as an automobile 1 can be repeatedly provided to the vehicle control unit 3 of the vehicle under development. Moreover, in this embodiment, the image including lens distortion can be repeatedly provided in a manner that is equivalent to real-time performance of a camera installed in the vehicle under development. In a simulation of the vehicle in this embodiment, which is driven in a three-dimensional virtual space, the vehicle can provide the vehicle control unit 3 under development with images that include wide-angle lens distortion, with the same image quality, frequency, and real-time capabilities as those obtained by an onboard camera when a vehicle is driving in real space. In contrast, simulation devices developed to date can only simulate the operation of a vehicle under development in a three-dimensional virtual space. In this case, a vehicle autonomously driving in a three-dimensional virtual space cannot capture images of its surroundings using an onboard camera, detect external objects such as other moving objects or falling objects in the captured images, or perform driving control according to the detected "external" conditions. Even if an autonomous vehicle can simulate driving in a three-dimensional virtual space, it cannot simulate driving control using images captured by an onboard camera or equivalent images, just as it would when driving on a real course.

[0078] The embodiments described above are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications or changes are possible without departing from the spirit of the invention.

[0079] In the embodiment described above, the lens distortion image of the generated three-dimensional virtual space is directly input to the driving control device 13 that controls the driving of the automobile 1. In addition, for example, the lens distortion image of the generated three-dimensional virtual space may be input to the control system of another device installed in the automobile 1, such as the wide-angle camera 2, and then input to the driving control device 13 via the control system of the wide-angle camera 2. Furthermore, some of the functions of the driving control device 13 may be performed by other devices provided in the automobile 1, such as a wide-angle camera 2. For example, the process of detecting external objects such as other moving objects or fallen objects around the vehicle based on images of the outside of the vehicle captured by the wide-angle camera 2 may be performed in the control system of the wide-angle camera 2. [Explanation of symbols]

[0080] 1...Automobile (vehicle), 2...Wide-angle camera, 3...Vehicle control device (vehicle control unit), 4...Actuator, 5...Vehicle motion sensor, 6...Location information generation device, 7...Vehicle communication device, 11...External object detection device, 12...Three-dimensional map device, 13...Driving control device, 20...Captured image, 21...Other vehicles, 22...Pedestrian crossing road surface pattern, 23...Monorail track, 30...Driving simulation system, 31...Vehicle simulator, 32...Driving simulator, 33...Lens distortion image generation device, 40...Computer device, 41...Input / output port, 42...Timer, 43...Storage device, 44...RAM (memory) 45...CPU, 47...Program, 48...Gradient pattern data, 50...Viewpoint image, 51...Other cars, 52...Pedestrian crossing road surface pattern, 53...Monorail track, 54-56...Captured images, 57-59...Images of other cars in captured images, 60...Driving conditions in real space, 61-63...Other cars, 70...Viewpoint image, 71-73...Images of other cars in viewpoint images, 80...Work area, 81...Effective working range, 83...Circumscribed circle, 91...Gradient pattern graph, 92...Gradient curve, 100...Line showing the amount of movement due to origin vanishing processing, 101...Curve showing the amount of movement due to wide-angle lens distortion

Claims

1. A device that performs image processing using memory to generate a lens distortion image that includes lens distortion from a wide-angle camera, such as when a three-dimensional virtual space is captured by a wide-angle camera at a predetermined viewpoint, A viewpoint image acquisition unit acquires viewpoint images of the three-dimensional virtual space viewed from a predetermined viewpoint and stores them in memory. A lens distortion processing unit that performs lens distortion processing corresponding to the wide-angle camera on the viewpoint image loaded in the memory, An output unit that outputs the image processed for lens distortion in the memory as a lens-distorted image that includes distortion from the wide-angle camera, It has, The aforementioned lens distortion processing unit is In the memory, a working area is generated consisting of at least all of the image pixels of the viewpoint image and a plurality of corresponding pixels, with the center of the lens distortion as the origin and the pixels from the first quadrant to the fourth quadrant. For each pixel of the viewpoint image displayed in the working area of ​​the memory, information is provided about the amount of positional movement that brings the positions of multiple pixels in the working area toward the origin, which is the center of lens distortion, and an erasure process is performed to make the multiple pixels in the working area disappear as a whole toward the origin. Distortion adjustment pattern data, which is used to bring the amount of positional movement of multiple pixels in the work area after the disappearance process closer to the amount of positional movement due to lens distortion by the wide-angle camera, is read into the memory and expanded. By referring to the distortion adjustment pattern data deployed in the memory, the position of each pixel in the work area after the loss process is adjusted, thereby moving the position of each image pixel in the viewpoint image deployed in the memory by an amount of movement corresponding to the lens distortion in the wide-angle camera, and generating an image in the memory that has undergone lens distortion processing corresponding to the wide-angle camera. A device for generating lens distortion images in a three-dimensional virtual space.

2. The distortion adjustment pattern data is one-dimensional gradient pattern data in which the value increases in the range of 0 to 1 in accordance with the difference between the image distortion caused by the amount of positional movement information in a plurality of pixels of the work area after the erasure process and the lens distortion by the wide-angle camera, The aforementioned lens distortion processing unit is The one-dimensional gradient pattern data is read into the memory and expanded, For multiple pixels in the work area after the disappearance process, the distance from the origin is generated. For each pixel in the work area after the vanishing process, a value is obtained from the one-dimensional gradient pattern data based on the distance from the origin. The values ​​from the aforementioned one-dimensional gradient pattern data are used as adjustment rates for the amount of positional movement for each pixel, and the information regarding the amount of positional movement for each pixel in the work area after the disappearance process is updated to adjust this information. A device for generating lens distortion images in a three-dimensional virtual space, as described in claim 1.

3. The aforementioned lens distortion processing unit is In order to obtain the distance from the origin for each pixel in the work area after the loss process, a similar effective work area corresponding to the lens distortion image is set for the work area generated in the memory, The length of the line segment from the origin to the pixel is normalized using the radius of the circumscribed circle of the effective working area as the reference distance. Using this normalized distance, the position value corresponding to the normalized distance is obtained from the one-dimensional gradient pattern data as an adjustment rate for the amount of movement of each pixel's position. A device for generating lens distortion images in a three-dimensional virtual space, as described in claim 2.

4. The wide-angle camera is used to capture images of the outside of a vehicle for the purpose of driving control in a vehicle traveling in real space. A device for generating lens distortion images in a three-dimensional virtual space, according to any one of claims 1 to 3.

5. A method for processing an image using a device that generates lens distortion images having memory, wherein the device generates a lens distortion image that includes lens distortion from a wide-angle camera, such that a three-dimensional virtual space is captured by a wide-angle camera at a predetermined viewpoint, The three-dimensional virtual space is viewed from the predetermined viewpoint, and a viewpoint image is acquired and generated in the memory. This image is then expanded into a work area having pixels from the first quadrant to the fourth quadrant, with the center of the lens distortion as the origin. For each pixel of the viewpoint image displayed in the working area of ​​the memory, information is provided about the amount of positional movement that brings the positions of multiple pixels in the working area toward the origin, which is the center of lens distortion, and an erasure process is performed to make the multiple pixels in the working area disappear as a whole toward the origin. Distortion adjustment pattern data, which is used to bring the amount of positional movement of multiple pixels in the work area after the disappearance process closer to the amount of positional movement due to lens distortion by the wide-angle camera, is read into the memory and expanded. By referring to the distortion adjustment pattern data deployed in the memory, and adjusting the information on the amount of positional movement for each pixel in the work area after the loss processing, the position of each image pixel in the viewpoint image deployed in the memory is moved by an amount of movement corresponding to the lens distortion in the wide-angle camera, thereby executing lens distortion processing to generate an image in the memory that has undergone lens distortion processing corresponding to the wide-angle camera. The image processed for lens distortion in the aforementioned memory is output as a lens-distorted image that includes distortion from the wide-angle camera. A processing method for a lens distortion image generation device in a three-dimensional virtual space.

6. A program for a computer to implement a processing method for an apparatus that generates a lens distortion image having memory, which performs image processing to generate a lens distortion image that includes lens distortion from a wide-angle camera, such as when a three-dimensional virtual space is captured by a wide-angle camera at a predetermined viewpoint, The aforementioned processing method is: The three-dimensional virtual space is viewed from the predetermined viewpoint, and a viewpoint image is acquired and generated in the memory. This image is then expanded into a work area having pixels from the first quadrant to the fourth quadrant, with the center of the lens distortion as the origin. For each pixel of the viewpoint image displayed in the working area of ​​the memory, information is provided about the amount of positional movement that brings the positions of multiple pixels in the working area toward the origin, which is the center of lens distortion, and an erasure process is performed to make the multiple pixels in the working area disappear as a whole toward the origin. Distortion adjustment pattern data, which is used to bring the amount of positional movement of multiple pixels in the work area after the disappearance process closer to the amount of positional movement due to lens distortion by the wide-angle camera, is read into the memory and expanded. By referring to the distortion adjustment pattern data deployed in the memory, and adjusting the information on the amount of positional movement for each pixel in the work area after the loss processing, the position of each image pixel in the viewpoint image deployed in the memory is moved by an amount of movement corresponding to the lens distortion in the wide-angle camera, thereby executing lens distortion processing to generate an image in the memory that has undergone lens distortion processing corresponding to the wide-angle camera. The image processed for lens distortion in the memory is output as a lens-distorted image that includes distortion from the wide-angle camera. program.

Images