Image processing device, operation evaluation device, operation evaluation system, image processing method, and program

Abstract

The aim is to reduce the impact of discrepancies between the camera's mounting position and the driver's viewpoint during driving within the vehicle's interior environment, thereby enabling accurate evaluation of the driver's driving conditions and other factors. [Solution] The controller acquires camera images captured by a camera mounted on the vehicle, obtains the camera's position, obtains the driver's eye position, and based on the camera's position and the driver's eye position, converts the camera image into an image from the driver's perspective as seen by the vehicle's driver, thereby generating an evaluation image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a driving evaluation apparatus, a driving evaluation system, an image processing method, and a program.

Background Art

[0002] Conventionally, it has been proposed to estimate the state of a user of a mobile terminal, for example, to estimate the state of the user including a state of reduced attentiveness based on the movement of the user's line of sight with respect to an image displayed on the mobile terminal (see, for example, Patent Document 1 below).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] The above conventional technology is considered to be applicable to, for example, estimating the state of a driver driving a vehicle. That is, based on a camera image captured by a camera mounted on the vehicle and the line of sight of the driver driving the vehicle, an application for evaluation such as whether the line of sight is appropriately sent to a place where attention is required is also conceivable.

[0005] However, in the environment of a vehicle cabin, the mounting position of the camera in the vehicle does not coincide with the position of the driver's eyes while driving. Therefore, if a point (referred to as a projection point) is projected onto the camera image in the direction of the driver's line of sight, the position (object) that the driver is actually seeing will differ from the position of the projection point (the object at the position of the projection point) in the camera image. Consequently, there is a risk that various evaluations and other processing based on the camera image and the driver's line of sight may not be performed properly. The purpose of the disclosed embodiment is, for example, to reduce the influence of the discrepancy between the camera mounting position and the position of the driver's viewpoint while driving in the environment of a vehicle cabin, and to enable accurate evaluation of the driver's driving conditions, etc. [Means for solving the problem]

[0006] Embodiments of the disclosure are exemplified by an image processing apparatus comprising a controller. The controller acquires a camera image captured by a camera mounted on a vehicle, acquires the position of the camera, acquires the position of the driver's eyes in the vehicle, and, based on the camera position and the eye position, converts the camera image into an image from the driver's perspective as seen by the driver to generate an evaluation image. [Effects of the Invention]

[0007] This image processing device generates evaluation images that closely resemble the driver's viewpoint, thus improving the performance of devices that use evaluation images, particularly driver assistance systems that utilize the relationship between evaluation images and the driver's gaze. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 illustrates a vehicle to which an image processing device is applied. [Figure 2] Figure 2 illustrates the hardware configuration of an information processing device and an information communication system including the information processing device. [Figure 3] Figure 3 illustrates an image processing method and an operation evaluation method using the information processing device of the first embodiment. [Figure 4]Figure 4 illustrates the coordinate relationships when converting a camera image to an image in the driver's viewpoint coordinate system (driver's viewpoint image). [Figure 5] Figure 5 illustrates an alternative method for converting the pixels of a camera image to the pixels of an image from the driver's perspective. [Figure 6] Figure 6 illustrates a camera image captured by a front camera and an image converted from this camera image to a driver's viewpoint in the second embodiment. [Figure 7] Figure 7 illustrates the reflection of interior equipment on the windshield. [Figure 8] Figure 8 illustrates the relationship between the reflected image on the windshield and sunlight. [Figure 9] Figure 9 illustrates an image processing method and an operation evaluation method using the information processing device of the second embodiment. [Modes for carrying out the invention]

[0009] The following descriptions of each embodiment will explain the image processing apparatus G, the operation evaluation apparatus, the image processing method, and the program.

[0010] <First Embodiment> Referring to Figures 1 to 3, an image processing device G according to the first embodiment and an information processing device 1 as an example of a driving evaluation device to which the image processing device G is applied will be described. Figure 1 is a diagram illustrating a vehicle C to which the image processing device G is applied.

[0011] Vehicle C is equipped with a forward-facing camera 17A that captures the direction of travel when Vehicle C is moving forward, and a driver camera 17B that captures the face of the driver while driving. Vehicle C is also equipped with an information processing device 1 that processes the camera image captured by the forward-facing camera 17A, which captures the direction of travel of Vehicle C, and the image of the driver captured by the driver camera 17B.

[0012] The front camera 17A is installed at the position P1 near the upper part of the windshield GL and captures the scenery in the traveling direction of the vehicle C. On the other hand, a driver camera 17B is installed on the rear side of the vehicle with respect to the front camera 17A, and captures the face image of the driver during driving, particularly, both eyes. Then, the information processing device 1 mounted on the vehicle C estimates the propriety of the way the driver distributes their line of sight, the degree of attention during driving, etc. based on the camera image captured by the front camera 17A and the face image captured by the driver camera 17B.

[0013] More specifically, the information processing device 1 creates, for example, a saliency map based on the camera image. The saliency map can be said to be a map showing prominent parts and visually emphasized areas in the camera image including the scenery in the traveling direction of the vehicle C. Also, the saliency map is used as a tool for analyzing where in the visual information is likely to attract a person's attention (that is, "prominent").

[0014] Therefore, for example, when the ratio at which the driver's line of sight is positioned in an attention - attracting area in the saliency map is higher than the ratio at which it is positioned in other areas, the information processing device 1 may make an evaluation of propriety. Propriety includes, for example, that the way the driver distributes their line of sight is proper and that the attention during driving is sufficient. On the other hand, for example, when no significant data is obtained indicating that the ratio at which the driver's line of sight is positioned in an attention - attracting area in the saliency map is higher than the ratio at which it is positioned in other areas, the information processing device 1 may make an evaluation of inappropriateness.

[0015] However, if a saliency map or the like is created using the camera image including the scene in the traveling direction of the vehicle C as it is, the obtained saliency map may not reflect the scene that appears in the actual viewing angle of the driver. This is because the position P1 near the upper part of the windshield GL where the front camera 17A is installed does not coincide with the viewpoint position P2 of the driver. Here, the position P1 is the position of the center of the lens of the front camera 17A. In the example of FIG. 1, the viewpoint position P2 of the driver is closer to the rear of the vehicle than the position P1 of the front camera 17A, and the height of the viewpoint position P2 of the driver is lower than the position P1 of the front camera 17A. Further, when the position P1 of the front camera 17A is located on the center plane that bisects the vehicle C in the left-right and vertical (vertical) directions, the viewpoint position P2 of the driver is shifted to the right side (or left side) of the center plane and is behind the vicinity of the steering wheel.

[0016] As described above, as a result of the position P1 of the front camera 17A not coinciding with the viewpoint position P2 of the driver, there is a possibility that a shift may occur between the position of the prominent region in the camera image and the position of the prominent region in the scene that appears in the actual viewing angle of the driver. Also, in the scene that appears in the actual viewing angle of the driver, there may be a portion of the image that is not in the camera image.

[0017] For example, the pillar PL of the vehicle C may not be captured in the camera image by the front camera 17A installed at the position P1 near the upper part of the windshield GL. On the other hand, a part of the scene that appears in the actual viewing angle of the driver often includes the inner surface of the vehicle compartment side of the pillar PL of the vehicle C.

[0018] Furthermore, for example, the camera image from the front camera 17A rarely shows interior structures reflected on the windshield GL. On the other hand, depending on the position of the sun, some of the scenery that appears in the driver's actual field of vision may include interior structures and equipment, such as the dashboard. The information processing device 1 employs an image processing method that minimizes the differences exemplified above, that is, the differences between the camera image from the front camera 17A and the scenery that appears in the driver's actual field of vision. The information processing device 1 then evaluates the appropriateness of the driver's gaze distribution, or the driver's attention level while driving.

[0019] Figure 2 illustrates the hardware configuration of an information processing device 1 and an information communication system 3 including the information processing device 1. In this embodiment, the information processing device 1 may be an in-vehicle audio-visual navigation device. The configuration of the information processing device 1 may be the same as that of a normal computer. The information processing device 1 has a CPU 11, a main memory unit 12, and external devices connected through an interface (I / F), and performs information processing using a computer program. A computer program is also simply called a program.

[0020] The CPU 11 executes programs loaded into the main memory 12 and provides the functions of the information processing device 1. The CPU 11 is also called a processor. However, the CPU 11 is not limited to a single processor and may be a multi-processor configuration. Furthermore, the CPU 11 may be a single processor connected by a single socket and may be a multi-core configuration. In addition, at least some of the processing of the information processing device 1 may be provided by dedicated processors such as a Digital Signal Processor (DSP), Graphics Processing Unit (GPU), numerical processing processor, vector processor, image processing processor, Application Specific Integrated Circuit (ASIC), etc. Also, at least Some of these are dedicated large-scale integrations (LSIs) such as Field-Programmable Gate Arrays (FPGAs). ), or other digital circuits. Furthermore, at least a portion of the information processing device 1 may include analog circuits.

[0021] The main memory unit 12 is also simply called memory, and it is where the computer program executed by the CPU 11 is stored. The main memory unit 12 stores data processed by the CPU 11. The main memory unit 12 includes Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), and Read Only Memory. (ROM), etc. Here, ROM includes rewritable flash memory, etc. Note that the CPU 11 and main memory unit 12 can be referred to as the controller 10.

[0022] Examples of external devices include an external storage unit 13, a display unit 14, an operation unit 15, a communication unit 16, and a camera 17. The external devices do not have to be integrated with the controller 10. Each of the external devices may be connected to the housing of the controller 10 via an interface.

[0023] The external storage unit 13 is used, for example, as a storage area that assists the main memory unit 12, and stores computer programs executed by the CPU 11, data processed by the CPU 11, etc. The external storage unit 13 can be a hard disk drive, a solid state drive (SSD), etc. Furthermore, the information processing device 1 may be provided with a drive device for a removable storage medium. Examples of removable storage media include Blu-ray discs, Digital Versatile Discs (DVDs), Compact Discs (CDs), and flash memory cards.

[0024] The display unit 14 is, for example, a liquid crystal display, an electroluminescent panel, or an organic light-emitting diode (OLED). In this embodiment, the display unit 14 may be, for example, the screen of an in-vehicle audio-visual navigation system. In addition, a speaker may be provided as an external device along with the display unit 14.

[0025] The operating unit 15 is, for example, a keyboard, a pointing device, etc. In this embodiment, a touch panel is exemplified as a pointing device.

[0026] The communication unit 16 exchanges data with other devices on the network N1. For example, the CPU 11 obtains parameters for controlling information processing through the communication unit 16. The communication unit 16 supports Long Term Evolution (LTE), 4th Generation Mobile Communication System ( The device may also be a wireless communication device that accesses network N1, which includes wireless access networks (RANs) such as 4G, 5th Generation Mobile Communication System (5G), and 6G. In other words, network N1 includes wired and wireless networks, such as public networks like the Internet and dedicated networks like Virtual Private Networks (VPNs). It may include wires. Network N1 may be accessed from a wireless Local Area Network (LAN). Also, the communication unit 16 may perform communication processing known as telematics.

[0027] However, the communication unit 16 may be an interface connected to the in-vehicle network (CAN). Furthermore, the in-vehicle network may include the above-mentioned wireless communication device, navigation device, Global Positioning System (GPS), and Global Navigation Satellite System (GNSS). ), a drive recorder, etc. may be connected. Furthermore, if the information processing device 1 is a navigation device, GPS, GNSS, etc. may be provided as external devices connected to the interface (I / F) in Figure 2.

[0028] Camera 17 includes two cameras: a forward camera 17A and a driver camera 17B. The forward camera 17A and the driver camera 17B are collectively referred to simply as camera 17. As shown in Figure 1, the forward camera 17A captures the view in the direction of travel of vehicle C and provides the captured camera images to the CPU 11 at a frame period defined by the specifications. The driver camera 17B is, for example, an in-vehicle camera that acquires the facial image of the user, who is the driver of vehicle C, at each frame period and provides it to the CPU 11. However, camera 17 (17A, 17B) may be a recording device connected to a drive recorder. Camera 17 is Ch The camera may have an large-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor. The camera 17 may also be an infrared camera. Furthermore, for the purpose of detecting the gaze of the driver who is the subject, the camera 17 may be an infrared camera, and may also be equipped with a laser for shining onto the driver's eyeball (cornea) to generate a reference point at the spot position.

[0029] Server 2 is connected to the information processing device 1 via network N1. The configuration of Server 2 is the same as that of the information processing device 1. However, Server 2 does not have to have a camera 17. That is, Server 2 may have the same configuration as the CPU 11, the main memory unit 12, and external devices connected via an interface (I / F). Also, Server 2 may have the same configuration as the controller 10.

[0030] Server 2 may be a device virtually configured by multiple computers, such as a cloud. Furthermore, at least some of the functions of the information processing device 1, for example, at least some of the processing shown in Figures 3 and 9 (described later), may be executed on Server 2. The information processing device 1 and Server 2 cooperate to form the information communication system 3.

[0031] Figure 3 illustrates an image processing method and an operation evaluation method using the information processing device 1 of the first embodiment. The CPU 11 of the information processing device 1 executes the processing shown in Figure 3 using a program that is executablely deployed in the main memory 12. In Figure 3, the processing from S1 to S6 can be called image processing, and the information processing device 1 that executes the processing from S1 to S6 can be called the image processing device G. Furthermore, at least a portion of the processing in Figure 3 can be executed by the server 2 in Figure 2. Therefore, the information processing device 1 and the server 2 may cooperate via the network N1 and execute information processing as an information communication system 3.

[0032] In the following process, the information processing device 1 performs processing as an example of an evaluation device or driving evaluation device that performs evaluation based on the gaze position, which is the position where the driver's gaze is directed in the evaluation image. In addition, in the following process, the information processing device 1 performs processing as an example of an image processing device G that generates a saliency map used in this evaluation device.

[0033] In this process, the information processing device 1 receives a camera image input from the forward camera 17A that captures the view in the direction of travel of the vehicle C (S1). Through the process in S1, the controller 10 of the information processing device 1 acquires the camera image captured by the forward camera 17A mounted on the vehicle C.

[0034] Furthermore, the information processing device 1 acquires the positional relationship between the position P1 (the position of the center of the lens) of the front camera 17A and the driver's viewpoint position P2 (S2). The information processing device 1 may, for example, receive the position P1 of the front camera 17A from a user interface consisting of a display unit 14 and an operation unit 15 and store it in a database formed in the main memory unit 12 or an external memory unit 13. The position P1 of the front camera 17A may be input, for example, with respect to the position of the driver's eyes, as follows: (L1cm forward, L2cm to the left, L3cm above). In this case, the information processing device 1 can identify the position P1 of the front camera 17A in the forward direction of the vehicle C on a horizontal plane at the height of the driver's eyes, with respect to the center position of both of the driver's eyes. That is, the information processing device 1 recognizes that the lens center position of the front camera 17A is located at a position L1cm forward and L2cm to the left of the center position of both of the driver's eyes (viewpoint position P2), and at a position L3cm from the horizontal plane at the height of both eyes. Furthermore, it may be assumed that the lens of the front camera 17A is fixed facing forward of the vehicle C, and that the information processing device 1 recognizes the direction of the optical axis of the front camera 17A.

[0035] However, the information processing device 1 uses a certain origin in three-dimensional space as a reference to determine the position of the forward camera 17A. The positional relationship between the lens center (position P1) and the driver's viewpoint (position P2) may be received as separate inputs, and this information may be acquired individually. Alternatively, the information processing device 1 may determine the driver's viewpoint (position P2) from the driver's facial image captured by the driver camera 17B.

[0036] Furthermore, the information processing device 1 may acquire images of the vehicle interior using the driver camera 17B or other in-vehicle cameras, and from the images of the vehicle interior, it may acquire the position P1 of the front camera 17A (the position of the center of the lens) and the driver's viewpoint position P2 (the center between the eyes). In this case, the information processing device 1 may determine the coordinates of positions P1 and P2 in three-dimensional space, using the position of the in-vehicle camera used for shooting as the origin.

[0037] The information processing device 1 may then calculate the positional relationship (for example, relative coordinates) between the acquired position P1 of the forward camera 17A and the driver's viewpoint position P2. In any case, the information processing device 1 executes the process in S2 as an example of a camera position detection device that detects the position of the forward camera 17A. The information processing device 1 also executes the process in S2 as an example of an eye position detection device that detects the position of the driver's eyes. The information processing device 1 then converts the camera image from the forward camera 17A into an image in the driver's viewpoint coordinate system (hereinafter referred to as the driver's viewpoint image) (S3). That is, the information processing device 1 converts the camera image into an image of the driver's viewpoint as seen from the driver of vehicle C (hereinafter referred to as the driver's viewpoint image) based on the position of the forward camera 17A and the position of the driver's eyes. The driver's viewpoint image is used to generate a saliency map in the process in S4, so it can also be considered an example of an evaluation image.

[0038] Figure 4 illustrates the coordinate relationships when converting a camera image to an image in the driver's viewpoint coordinate system (driver's viewpoint image). In Figure 4, a camera coordinate system (X, Y, Z) with the center O of the lens of the front camera 17A as the origin is shown. Here, it is assumed that the optical axis of the front camera 17A points from the center O in the Z-axis direction (direction of travel of vehicle C). Furthermore, the coordinate system in the image captured by the front camera 17A (image coordinate system) is two-dimensional and is given by (px, py). Here, (px, py) corresponds to the pixels in the image.

[0039] The relationship between the image coordinate system (px, py) and the camera coordinate system (X, Y, Z) is defined by the following equation (1). Here, a dot (·) represents multiplication, and a slash ( / ) represents division. Also, f is a parameter (focal length) that defines the distance between the viewpoint position of the virtual camera that captures the projected image and the projection surface (the surface of the display) on which the image is formed. Equation (1) is a transformation equation called perspective projection.

number

[0040] Incidentally, the information processing device 1 sets the driver viewpoint coordinate system (X',Y',Z') as the coordinate system in which the Z' axis of the coordinate axis is oriented in the direction of the driver's line of sight (visual axis) at the driver's viewpoint position P2. Here, the driver viewpoint coordinate system (X',Y',Z') is a coordinate system obtained by rotating (gamma, theta, phi) and translating (dx,dy,dz) the camera coordinate system (X,Y,Z) around each axis. Let's assume that... However, the rotations (gamma, theta, phi) around each axis are as follows in Figure 4. These are illustrated using Greek letters.

[0041] In the assumption illustrated in Figure 4, the camera coordinate system (X,Y,Z) and the driver's viewpoint coordinate system (X',Y',Z') satisfy the following relationship (2).

[0042]

number

[0043]

number

[0044] Figure 5 illustrates an alternative method for converting pixels (px, py) in a camera image to pixels (px', py') in an image from the driver's perspective. In this example, the information processing device 1 maintains a pixel conversion table that converts each pixel position (px, py) in the camera image before conversion to each pixel position (px', py') in the image from the driver's perspective after conversion. In Figure 5, the table on the left illustrates the pixel conversion table. The conversion table is a data table in which the converted pixel position data is stored in cells corresponding to each pixel (position) in the camera image. The pixel value of each pixel (px, py) in the camera image is then converted to the pixel position (px', py') indicated by the pixel position data of the corresponding cell (x, y) in the conversion table, and a converted image is generated. To make the explanation easier to understand, in Figure 5, the table on the left shows the converted pixel position data and the pixel value of the camera image before conversion in each cell. Also, in Figure 5, the table on the right is an array illustrating the pixel positions (px', py') in the image from the driver's perspective after conversion, which are generated according to the pixel conversion table.

[0045] The pixel conversion table has the same pixel array (px,py) as the camera image before conversion. Each element of the pixel conversion table holds the converted pixel position and the pixel value at that pixel position before conversion. The information processing device 1 can generate each pixel (px',py') of the image from the driver's viewpoint after conversion by writing the pixel values ​​recorded in each element of the pixel conversion table to the converted pixel positions. For each combination of the position P1 of the front camera 17A and the driver's viewpoint position P2, a group of data tables is prepared in which the pixel position data before and after conversion is associated and stored. The data table corresponding to the detected position P1 of the front camera 17A and the driver's viewpoint position P2 is selected, and the value of the data in that table is set (registered) as the converted pixel position data in the pixel conversion table, thereby generating the desired pixel conversion table. The pixel values ​​in the pixel conversion table are the same as those in the camera image of the front camera 17A. Each pixel value in the image will be set (registered) in the corresponding cell in the table.

[0046] For example, the pixel value of pixel (1,1) in the camera image before conversion is P1-1. This pixel value becomes the pixel value of pixel (1,2) in the image from the driver's perspective after conversion, as specified in 1-2. Similarly, the pixel value of pixel (2,1) in the camera image before conversion is P2-1. This pixel value becomes the pixel value of pixel (3,1) in the image from the driver's perspective after conversion, as specified in 3-1.

[0047] Furthermore, if multiple pixels in the original image before conversion are converted to the same pixel after conversion, the information processing device 1 should write the average value of the multiple pixel values ​​in the original image to the destination. Conversely, for pixels in the converted image that do not have a corresponding pixel in the original image before conversion, the information processing device 1 should set the average value of the pixel values ​​in the surrounding image of the converted pixel to the converted pixel.

[0048] Next, the explanation returns to Figure 3. Once the coordinate transformation to the driver's viewpoint coordinate system is complete, the information processing device 1 creates a saliency map from the image at the driver's viewpoint after the transformation (S4). The image transformed to the driver's viewpoint coordinate system is an example of an evaluation image. Therefore, the process in S4 is an example of a process that creates a saliency map based on an evaluation image.

[0049] Various methods have been proposed for creating saliency maps (see, for example, Masatoshi Yoshida, Application of Saliency Maps to Visual Search Analysis, Journal of the Japanese Neural Network Society, Vol. 21, No. 1 (2014)). The information processing device 1 can receive camera image input at frame intervals and repeatedly execute processes S3 and S4.

[0050] For example, the information processing device 1 creates individual maps from an input image to individually evaluate various features of visual information, such as color, contrast, and motion. Then, the information processing device 1 adds up these maps, which represent different features such as color, contrast, and motion, using a linear sum. The linear sum can be a simple addition, or different weights can be assigned to each feature map before adding them up. By assigning weights, it becomes possible to reflect the importance of certain visual features when they are more important than others. Furthermore, if the scales of the feature maps are different, the information processing device 1 can normalize them to make their values ​​comparable on a common scale.

[0051] Meanwhile, the information processing device 1 acquires the driver's face image input from the driver camera 17B at a frame rate (S5). The information processing device 1 then detects the driver's gaze from the driver's face image (S6). The position of the driver's gaze can be identified as the intersection point where the driver's gaze intersects a virtual plane in front of the driver that is directly facing the driver. The virtual plane directly facing the driver is the plane on which the transformed image coordinate system (px',py') at the driver's viewpoint obtained in S3 is formed. Alternatively, for example, the virtual plane may be divided into sub-regions common to each region in the saliency map generated in S4.

[0052] In this embodiment, there are no limitations on the gaze detection method. Various methods have been proposed for detecting the driver's gaze. For example, active and passive methods are known. The active method uses the spot position of a laser irradiated onto the eyeball (cornea) as a reference point, photographs the eyeball with an infrared camera, and detects the gaze from the movement of the pupil. On the other hand, the passive method uses features such as the face and eyes in an image captured with visible light as reference points, and detects the gaze from the positional relationship with a moving point (e.g., the outer edge of the iris). As an example of a gaze detection device, the information processing device 1 executes the processes of S5 and S6 to detect the driver's gaze.

[0053] Next, the information processing device 1 uses the saliency map acquired in S4 and the line of sight acquired in S6. The detected position is compared with the saliency map (S7). However, the comparison is not limited to the saliency map. The information processing device 1 may, for example, compare the driver's gaze with individual maps that individually evaluate various features of visual information, such as color, contrast, and motion, before the linear sum is performed. Alternatively, the information processing device 1 may, for example, store data recording the gaze movements of other drivers when they travel the same route, create a heatmap based on the stored data, and compare the created heatmap with the driver's gaze.

[0054] The information processing device 1 then evaluates the proportion of time that the driver's gaze is located in attention-grabbing regions of the saliency map. For example, the information processing device 1 can evaluate the proportion of time the driver's gaze lingers in attention-grabbing regions of the saliency map.

[0055] For example, an array of n x m sub-rectangles is assumed to exist in the same virtual plane as the saliency map formed in front of the driver. The range of the virtual plane may be, for example, a circle corresponding to the driver's field of view. The information processing device 1 represents the sub-regions included in these ranges with an index of (i,j), and defines the time spent at the intersections through which the line of sight passes in the sub-regions specified by (i,j) as Ti,j.

[0056] For example, the information processing device 1 calculates the dwell time ratio Ri,j = Ti,j / (sum of dwell time in each region) for each region. The information processing device 1 only needs to evaluate the correlation between regions with relatively high dwell time ratios Ri,j and regions in the saliency map that are likely to attract attention.

[0057] Then, based on the comparison results in S7, the information processing device 1 determines whether the driver's gaze distribution is appropriate or whether the driver's driving is appropriate. If a correlation exists between the area where the driver's gaze dwell time is relatively high and the area in the saliency map that easily attracts attention, exceeding a predetermined threshold value, the information processing device 1 can determine that distracted driving is not occurring. On the other hand, if no significant correlation exists, the information processing device 1 can determine that distracted driving is occurring. Following these steps, the information processing device 1 detects distracted driving (S8). As an example of a driving evaluation device, the information processing device 1 determines whether the driver's driving is appropriate based on the evaluation image and the driver's gaze, as processed in S8.

[0058] In Figure 3, the information processing device 1 that performs the processes from S4 to the left of S6, excluding the processes of S7 and S8, can be considered an example of the image processing device G. The image processing device G generates an evaluation image to be used in an evaluation device that performs evaluation based on the gaze position, which is the position where the driver's gaze is directed in the evaluation image, according to the procedure in Figure 3. Furthermore, the information processing device 1 that performs the processes in Figure 3, including S7 and S8, can be considered an example of a driving evaluation device that determines the appropriateness of the driver's driving based on the evaluation image and the driver's gaze. In addition, the information processing device 1 can be considered an example of a driving evaluation system.

[0059] (Effects of the embodiment) As described above, in this embodiment, the information processing device 1 performs processing as an evaluation device that performs evaluation based on the gaze position, which is the position where the driver's gaze is directed in the saliency map used as an evaluation image. The information processing device 1 also includes a controller 10 that performs processing as an image processing device G that generates the saliency map used in this evaluation. The controller 10 acquires a camera image taken by a front camera 17A mounted on the vehicle C. The information processing device 1 also acquires the position of the front camera 17A. Furthermore, the information processing device 1 acquires the position of the driver's eyes. Then, based on the position of the front camera 17A and the position of the driver's eyes, the information processing device 1 converts the camera image into an image from the driver's viewpoint as seen from the driver of the vehicle C, and generates an evaluation image.

[0060] In this way, the information processing device 1 provides an evaluation that is very close to the image from the driver's viewpoint. An image for evaluation is generated. As a result, the information processing device 1 can accurately evaluate the driver's driving condition, etc., in the vehicle cabin environment by reducing the influence of the difference between the camera's mounting position and the driver's viewpoint during driving. Furthermore, it is expected that the performance of devices using evaluation images, particularly driver assistance systems that use the relationship between evaluation images and the driver's line of sight, will be improved.

[0061] Furthermore, the information processing device 1 determines the appropriateness of the driver's driving based on a saliency map that shows which parts of the visual information are most likely to attract a person's attention (i.e., "conspicuous") and the driver's gaze. Therefore, the information processing device 1 can objectively determine the appropriateness of the driver's driving.

[0062] Furthermore, since the information processing device 1 generates a saliency map based on evaluation images from the driver's perspective, it can generate a saliency map more accurately than before based on the scenery that is within the driver's field of view. In addition, the information processing device 1 can evaluate the driver's driving more accurately.

[0063] <Second Embodiment> The information processing device 1 according to the second embodiment will be described with reference to Figures 6 to 9. In the first embodiment described above, the information processing device 1 converted the camera image into an image from the driver's perspective (evaluation image) as seen from the driver of the vehicle C, based on the position of the front camera 17A and the position of the driver's eyes, and generated a saliency map. The information processing device 1 also determined the appropriateness of the driver's driving based on the saliency map and the driver's line of sight.

[0064] In this embodiment, in addition to the processing of the first embodiment described above, the information processing device 1 further generates an image of the interior equipment from the driver's viewpoint based on vehicle information that identifies the structure of vehicle C and the position of the driver's eyes, and superimposes (reflects) the image of the interior equipment onto the evaluation image. Alternatively, instead of an image of the interior equipment, the information processing device 1 may generate an image of the interior equipment area that shows the area of ​​the interior equipment from the driver's viewpoint, and superimpose (reflects) the image of the interior equipment area onto the evaluation image. Furthermore, the information processing device 1 may generate a reflection image SH2 that is reflected on the windshield GL of vehicle C based on vehicle information that identifies the structure of vehicle C, the position of the sun, and the position of the driver's eyes, and superimpose (reflects) the reflection image SH2 onto the evaluation image.

[0065] Aside from the addition of this processing, the configuration and operation of the information processing device 1 are the same as in the first embodiment. Therefore, the information processing device 1 of the first embodiment is applied directly to the second embodiment.

[0066] Figure 6 illustrates a camera image captured by the front camera 17A and an image converted from this camera image to the driver's viewpoint in the second embodiment. In Figure 6, the left side of the arrow shows an example of a camera image captured by the front camera 17A. In Figure 6, the right side of the arrow shows an example of an image converted to the driver's viewpoint.

[0067] Similar to the first embodiment, the image converted to the driver's viewpoint is an image that looks further away than the camera image captured by the front camera 17A. Furthermore, in this embodiment, the information processing device 1 generates images of the interior equipment from the driver's viewpoint based on vehicle information that identifies the structure of the vehicle C. For example, the vehicle information includes shape information that identifies the position and shape of the dashboard DS, pillar PL, etc., in three-dimensional space. Here, the three-dimensional space may be, for example, the coordinate system of the design information or the coordinate system of the vehicle C.

[0068] Furthermore, the shape information may be in the form of point cloud data (Xi, Yi, Zi) in 3D space, or polygon data (for example, a set of vertices representing a triangle). Polygon data may be converted to point cloud data using known methods such as point sampling on the polygon surface, voxelization of the polygon, or edge sampling where points are placed on the edges of the polygon.

[0069] The information processing device 1 can, for example, convert the coordinate system of the design information or the shape information in the coordinate system of the vehicle C to the driver's viewpoint coordinate system (X', Y', Z') using equation (2) of the first embodiment. Then, based on the position of the driver's eyes, the information processing device 1 can calculate the pixel values ​​and positions (px', py') of the image at the driver's viewpoint from the shape information (e.g., point cloud data) in the driver's viewpoint coordinate system (X', Y', Z') using equation (3) of the first embodiment. The image at the driver's viewpoint thus formed will be the outline of the interior equipment. Then, the information processing device 1 can superimpose (reflect) the formed image of the interior equipment outline onto the image converted to the driver's viewpoint (evaluation image). Here, interior equipment refers to, for example, the dashboard DS, pillars PL, and other wall surfaces on the interior side of the vehicle.

[0070] Furthermore, the information processing device 1 does not need to generate images of the in-vehicle equipment itself. That is, instead of generating images of the in-vehicle equipment, the information processing device 1 may generate an image of the in-vehicle equipment area that indicates the area of ​​the in-vehicle equipment. The information processing device 1 may then determine the position of the in-vehicle equipment area image in the same coordinate system (X',Y',Z') as the image converted to the driver's viewpoint, based on the position of the driver's eyes, and superimpose (reflect) it on the image converted to the driver's viewpoint. In this way, the information processing device 1 generates an evaluation image that is even closer to the driver's viewpoint image than in the first embodiment.

[0071] Figure 7 illustrates the reflection of interior equipment on the windshield GL. Figure 7 shows the view seen through the windshield GL of vehicle C traveling on road RD. However, Figure 7 also illustrates some interior equipment, such as the dashboard DS. Furthermore, Figure 7 illustrates the preceding vehicle C1, a mountain range labeled "Mountain," and the sun above the mountain range, as seen through the windshield GL.

[0072] Depending on the position of the sun (also called solar altitude or solar azimuth angle), the brightness inside the vehicle changes, which can cause reflections of interior equipment on the windshield GL of vehicle C. Reflections can also occur on the side and rear windows, so the following discussion will focus on reflections on vehicle glass. Specifically, reflections on glass occur when high-intensity light, such as sunlight, shines on interior equipment (whose surfaces have strong properties of diffuse reflection), increasing its brightness. The image of this bright interior equipment is then reflected by the vehicle glass and enters the driver's field of vision. The smaller the ratio between the brightness of the scenery transmitted through the windshield and the brightness of the image of the interior equipment, the more noticeable the reflection becomes, and the more difficult it is to see the scenery. Figure 7 shows an example of a reflected image SH2 on the windshield GL. In this example, the reflected image SH2 is an image created by light reflected from the front part of the dashboard DS.

[0073] Figure 8 illustrates the relationship between the reflected image SH2 on the windshield GL and sunlight. The reflected image SH2 visible to the driver can be described as a virtual image formed on the opposite side of the windshield GL from the dashboard and other interior equipment (towards the front of the vehicle), with the windshield GL as the plane of symmetry. Therefore, the greater the amount of light reflected by the dashboard and other interior equipment, the stronger the reflected image SH2 will appear.

[0074] The sun is considered as the light source for light reflected from the dashboard, etc. That is, the amount of light reflected from the dashboard, etc. changes depending on the position of the sun, and the appearance of the reflected image SH2 also changes. For example, if the sun is directly above vehicle C and sunlight enters the cabin at the angle of arrow LN1, the brightness around the dashboard DS will be relatively close to the brightness outside the vehicle. In such a case, the effect of reflections on the windshield GL on the driver's field of vision becomes strong enough not to be ignored. On the other hand, if the sun is in front of vehicle C and sunlight enters the cabin at the angle of arrow LN2 When entering the vehicle interior, the brightness around the dashboard DS is relatively lower than the brightness outside the vehicle. In such cases, reflections on the windshield GL are less likely to occur (however, the effect of direct sunlight entering the field of view may still occur). In the example in Figure 8, the angle of arrow LN1 is shown as an example of the sun's position being higher than the angle of arrow LN2. Also, for example, when the sun is directly above vehicle C and sunlight is incident at the angle of arrow LN1, the brightness around the dashboard DS is brighter than when sunlight is incident from the side of vehicle C through windows W1, W2, etc., at the angle of arrow LN3. Therefore, the effect of reflections is likely to be greater when sunlight is incident at the angle of arrow LN1 than when it is incident at the angle of arrow LN3.

[0075] In such a case, the information processing device 1 can estimate the brightness near the dashboard DS, and therefore the intensity of the reflected image SH2 (or the intensity of the influence that the reflected image SH2 has on the driver), based on the relationship between the direction of travel of the vehicle C and the position of the sun. For example, the information processing device 1 can obtain the current date and time from calendar information it has acquired from the operating system (OS), etc.

[0076] Furthermore, the information processing device 1 can obtain the position of the sun (solar altitude, solar azimuth angle) at the current date and time from an information provision site such as a weather information server 2 on the network N1. The information processing device 1 may also store the position of the sun (solar altitude, solar azimuth angle) at each date and time in a database such as the main memory unit 12 or the external memory unit 13. The information processing device 1 may also store the brightness outside the vehicle at each date and time in a database.

[0077] Furthermore, the information processing device 1 can obtain the direction of travel of vehicle C from the navigation device. In addition, the information processing device 1 can obtain real-time location information from GPS and GNSS and determine the direction of travel of vehicle C.

[0078] From the above, the information processing device 1 can determine the position of the sun relative to vehicle C at the current date and time, or the orientation of vehicle C relative to the position of the sun. The position of the sun relative to vehicle C is, for example, on the GL side of the windshield of vehicle C, on the window side of the left and right doors, or on the rear side of vehicle C.

[0079] Furthermore, when the sun is located on the GL side of the windshield of vehicle C, the information processing device 1 can determine the brightness near the dashboard DS based on the sun's position (solar altitude, solar azimuth angle). Similarly, when the sun is located on the window side of either the left or right door of vehicle C, the information processing device 1 can determine the brightness near the dashboard DS based on the sun's position (solar altitude, solar azimuth angle). The information processing device 1 then obtains the ambient brightness outside the vehicle for each date and time from the database and calculates the ratio of the brightness near the dashboard to the ambient brightness outside the vehicle. As an approximation, the information processing device 1 may calculate the brightness ratio assuming that the ambient brightness outside the vehicle during the day is constant regardless of the sun's position.

[0080] The information processing device 1 can determine that the intensity of the reflected image SH2, or the strength of the influence of the reflected image SH2 on the driver, is not negligible when the ratio of the brightness near the dashboard to the brightness outside the vehicle exceeds a predetermined limit. However, the information processing device 1 may store in a database the dates and times when the ratio of the brightness near the dashboard to the brightness outside the vehicle is not negligible. The information processing device 1 may also determine that the strength of the influence of the reflected image SH2 on the driver is not negligible based on the current date and time and the weather. In this case, the information processing device 1 can obtain the current weather from an information provision site such as a server 2 that provides weather information on the network N1. As described above, for example, vehicle information is in the coordinate system of the design information, or the coordinate system of the vehicle C, with the dashboard D The system has shape information (e.g., point cloud data) that identifies the position and shape of S, pillar PL, etc., in three-dimensional space. The information processing device 1 can then use the vehicle information to convert the shape information in the coordinate system of the design information, or in the coordinate system of the vehicle C, to the driver's viewpoint coordinate system (X', Y', Z') using, for example, the following conversion formula (2) in the first embodiment. The information processing device 1 can then form an image of the exterior shape of the interior equipment, similar to the procedure described in Figure 6. Furthermore, the information processing device 1 can generate a virtual image of the exterior shape of the interior equipment, such as the dashboard DS and pillar PL, in the driver's viewpoint coordinate system (X', Y', Z'), with the windshield GL as the plane of symmetry, and generate a reflected image SH2. The information processing device 1 may also change the shape (or size) of the reflected image SH2 based on the position of the sun (solar altitude, solar azimuth angle) in that case.

[0081] Figure 9 illustrates an image processing method and a driving evaluation method using the information processing device 1 of the second embodiment. In Figure 9, the processing from S1 to S8 is the same as in Figure 3 of the first embodiment. In Figure 9, processing from S1A to S1E is added as processing related to images of in-vehicle equipment and reflected images SH1 and SH2.

[0082] In this process, as in the first embodiment, the information processing device 1 receives a camera image input from the front camera 17A that captures the view in the direction of travel of the vehicle C (S1). In this embodiment, the information processing device 1 excludes reflected images SH1 in which in-vehicle equipment such as the dashboard is reflected in the image captured by the front camera 17A through the windshield GL. Reflected images SH1, like reflected images SH2 described in Figures 7 and 8, depend on the position of in-vehicle equipment such as the dashboard and the position of the sun. However, reflected images SH1 differ from reflected images SH2 as seen by the driver, as described in Figures 7 and 8, in that they are formed in the camera image of the front camera 17A.

[0083] Therefore, the information processing device 1 inputs the design information of vehicle C (shape of the vehicle structure). The information processing device 1 obtains the design information of vehicle C from, for example, the main memory unit 12, the external memory unit 13, or a database on the server 2 connected via the network N1 (S1A). The information processing device 1 also obtains the current solar altitude using the procedure described in Figure 8 and identifies the position of the sun in the direction of travel of vehicle C (S1B).

[0084] Then, the information processing device 1 generates the reflected image SH1 that is captured by the front camera 17A using the same procedure as described in Figures 7 and 8, and temporarily removes the reflected image SH1 from the image of the front camera 17A (S1C).

[0085] Next, following the same procedure as in Figure 3 of the first embodiment, the information processing device 1 acquires the positional relationship between the position P1 (the position of the center of the lens) of the front camera 17A and the driver's viewpoint position P2 (S2). Furthermore, the information processing device 1 converts the camera image from the front camera 17A into an image in the driver's viewpoint coordinate system (hereinafter referred to as the driver's viewpoint image) (S3). This procedure is explained in Figure 4 or Figure 5.

[0086] Next, the information processing device 1 calculates the shapes of vehicle structures (pillar PL, dashboard DS, etc.) that appear in the driver's viewpoint image (S1D). This process is as described in Figure 6. Specifically, the information processing device 1 executes the S1D process based on the design information of vehicle C (point cloud data representing the shape of vehicle structures) and the positional relationship between the position P1 of the front camera 17A (the position of the center of the lens) and the driver's viewpoint position P2. The S1D process is an example of generating images of in-vehicle equipment from the driver's viewpoint based on vehicle information that identifies the structure of vehicle C and the position of the eyes.

[0087] Furthermore, the information processing device 1, in the driver's viewpoint coordinate system, reflects the windshield GL. The shape of the reflected vehicle structure (such as the dashboard) is calculated, and a reflected image SH2 is generated (S1E). The S1E process is carried out according to the procedure described in Figures 7 and 8. That is, the information processing device 1 performs the S1E process based on the design information of the vehicle C (point cloud data representing the shape of the vehicle structure), the positional relationship between the front camera 17A and the driver's viewpoint, and the position of the sun relative to the vehicle C. The S1E process is an example of generating a reflected image SH2 reflected on the windshield GL of the vehicle C, based on vehicle information that identifies the structure of the vehicle C, the position of the sun, and the position of the eyes. However, if the ratio of the brightness near the dashboard DS to the brightness outside the vehicle at the current date and time is negligibly small, the information processing device 1 may omit the S1E process.

[0088] The information processing device 1 then superimposes the shape of the vehicle structure (exterior appearance of pillars PL, dashboard DS, etc.) and the reflected image SH2 reflected on the windshield GL onto the driver's view image (S1F). The S1F process is an example of superimposing images of interior equipment to create an evaluation image. The S1F process is also an example of superimposing the reflected image SH2 to create an evaluation image.

[0089] The processing from S4 to S8 is the same as in Figure 3. In Figure 9, the information processing device 1 that performs the processing from S4 and S6 to the left, excluding the processing of S7 and S8, can be considered an example of an image processing device G that generates evaluation images. Furthermore, the information processing device 1 that performs the processing in Figure 9, including S7 and S8, can be considered an example of a driving evaluation device that determines the appropriateness of the driver's driving based on the evaluation images and the driver's gaze. In addition, the information processing device 1 can be considered an example of a driving evaluation system.

[0090] (Effects of the embodiment) As described above, in this embodiment, the information processing device 1 generates images of the in-vehicle equipment from the driver's perspective based on vehicle information that identifies the structure of the vehicle C and the position of the driver's eyes. The information processing device 1 then superimposes the images of the in-vehicle equipment to create an evaluation image. Therefore, the information processing device 1 can include in the evaluation image those in the driver's field of view, such as pillars PL and dashboard DS, which are difficult to capture with the front camera 17A. Thus, the information processing device 1 can faithfully reproduce the scene that is in the driver's field of view to create a saliency map and evaluate the driver's driving.

[0091] Furthermore, the information processing device 1 generates a reflected image SH2 that is reflected on the windshield GL of vehicle C, based on vehicle information that identifies the structure of vehicle C, the position of the sun, and the position of the eyes. The information processing device 1 superimposes the reflected image SH2 to create an evaluation image. In other words, the information processing device 1 can include the reflected image SH2 from the driver's perspective, which is difficult to capture with the front camera 17A, in the evaluation image. Therefore, the information processing device 1 can faithfully reproduce the scene that enters the driver's field of view to create a saliency map and evaluate the driver's driving.

[0092] (modified version) In the process shown in Figure 9 above, as in S1D, the information processing device 1 calculates the shape of vehicle structures (such as pillars, dashboards, etc.) that are reflected in the driver's viewpoint image. However, the processing of the information processing device 1 is not limited to Figure 9. For example, the information processing device 1 does not have to generate the shape of vehicle structures (such as pillars, dashboards, etc.) that are reflected in the driver's viewpoint image. When generating a saliency map from the evaluation image, the shape of vehicle structures (such as pillars, dashboards, etc.) that are reflected in the driver's viewpoint image acts to obstruct the driver's view. Therefore, the information processing device 1 may generate the shape of the area that obstructs the driver's view.

[0093] For example, the information processing device 1 identifies vehicle information that identifies the structure of vehicle C and the position of the driver's eyes. Based on this, an interior equipment area image is generated that shows the area of ​​the interior equipment from the driver's perspective. The interior equipment area image is, for example, an image that shows the outline of the vehicle structure, with the inside of the outline filled in with a single color. The information processing device 1 can then superimpose the interior equipment area image to create an evaluation image. With this processing, the information processing device 1 can more easily create a saliency map corresponding to the scenery that enters the driver's field of view and evaluate the driver's driving. (Other variations) In a warning system for obstacles around a vehicle (objects that may cause collision and should be avoided), a function is employed that does not issue a warning for obstacles that the driver is aware of in order to prevent unnecessary warnings (to reduce annoyance). Such a warning system determines whether or not the driver is aware of an obstacle by matching the driver's line of sight with the position of the obstacle in the camera image. The image processing device G exemplified in the first embodiment, the second embodiment and its modifications described above can be applied to the correction of the camera image in such a warning system. [Explanation of Symbols]

[0094] 1. Information Processing Device 2 servers 3. Information and Communication Systems 10 Controllers 11 CPU 12 Main memory 13 External storage unit 15 Control section 16 Communications Department 17A Front Camera 17B Driver Camera

Claims

1. An image processing device for generating an evaluation image used in an evaluation device that performs evaluation based on the gaze position, which is the position where the vehicle driver's gaze is directed in the evaluation image, comprising a controller, The aforementioned controller, Camera images captured by the camera mounted on the aforementioned vehicle are acquired. The position of the aforementioned camera is obtained, The driver's eye position is obtained, Based on the position of the camera and the position of the eye, the camera image is converted into an image from the driver's viewpoint as seen by the driver to generate the evaluation image. Image processing device.

2. The aforementioned controller, Based on vehicle information that identifies the structure of the vehicle and the position of the eyes, an image of the interior equipment from the driver's viewpoint is generated. The image processing apparatus according to claim 1, which superimposes images of the in-vehicle equipment to obtain the evaluation image.

3. The aforementioned controller, Based on vehicle information that identifies the structure of the vehicle and the position of the eyes, an interior equipment area image is generated that shows the area of ​​the interior equipment from the driver's viewpoint. The image processing apparatus according to claim 1, wherein the image of the vehicle interior equipment area is superimposed to form the evaluation image.

4. The aforementioned controller, Based on vehicle information that identifies the structure of the vehicle, a reflected image is generated that is reflected in the windshield of the vehicle. The image processing apparatus according to claim 1, wherein the reflected image is superimposed to obtain the evaluation image.

5. The aforementioned controller, An image processing apparatus according to any one of claims 1 to 4, which generates a saliency map based on the evaluation image.

6. A driving evaluation device equipped with a controller that performs evaluation based on the gaze position, which is the position where the vehicle driver's gaze is directed in the evaluation image, The aforementioned controller, Camera images captured by a camera mounted on the vehicle are acquired. The position of the aforementioned camera is obtained, The driver's eye position is obtained, The driver's gaze is acquired, Based on the position of the camera and the position of the eye, the camera image is converted into an image from the driver's viewpoint as seen by the driver to generate an evaluation image. Based on the evaluation image and the driver's line of sight, the suitability of the driver's driving is determined. Operation evaluation device.

7. The aforementioned operation evaluation device is A saliency map is generated based on the aforementioned evaluation image. Claim 6 for determining distracted driving based on the saliency map and the driver's line of sight. The operation evaluation device described above.

8. The camera mounted on the vehicle, A camera position detection device for detecting the position of the aforementioned camera, A gaze detection device for detecting the driver's gaze of the vehicle, An eye position detection device for detecting the eye position of the driver, A driving evaluation system comprising a driving evaluation device for performing driving evaluations of the aforementioned driver, The aforementioned operation evaluation device is The camera image captured by the aforementioned camera is acquired, The camera position is obtained from the camera position detection device, The eye position of the driver is obtained from the eye position detection device. The driver's gaze is acquired from the gaze detection device. Based on the position of the camera and the position of the eye, the camera image is converted into an image from the driver's viewpoint as seen by the driver to generate an evaluation image. Based on the evaluation image and the driver's line of sight, Determining whether the driver is performing the operation appropriately. Driving evaluation system.

9. An image processing method for generating an evaluation image used in an evaluation device that performs evaluation based on the gaze position, which is the position where the vehicle driver's gaze is directed in the evaluation image, Computers Camera images captured by the camera mounted on the aforementioned vehicle are acquired. The position of the aforementioned camera is obtained, The driver's eye position is obtained, Based on the position of the camera and the position of the eye, the camera image is converted into an image from the driver's viewpoint as seen by the driver to generate the evaluation image. Image processing methods.

10. A program for causing a computer to generate an evaluation image used in an evaluation device that performs evaluation based on the gaze position, which is the position where the vehicle driver's gaze is directed in the evaluation image, To the aforementioned computer, Camera images captured by the camera mounted on the aforementioned vehicle are acquired. The position of the aforementioned camera is obtained, The driver's eye position is obtained, To generate the evaluation image by converting the camera image into an image from the driver's viewpoint as seen by the driver, based on the position of the camera and the position of the eye. program.

Images