Image creation device, image creation method, and recording medium

Abstract

Provided are: an image creation device capable of displaying, presenting, or recording an area in the line-of-sight direction of a driver operating a moving body or an occupant of a moving body; a display control method; and a recording medium.　The image creation device comprises: a line-of-sight direction identifying means that identifies the line-of-sight direction of a user from an image of the user captured by a first camera that captures an image of the user; and an image creation means that, on the basis of the line-of-sight direction of the user, creates an image obtained by superimposing and displaying a graphic along the line-of-sight direction on an image captured by a second camera that captures an image of an area outside of a moving body in which the user is riding.

Description

Image generating apparatus, image generating method, and recording medium

[0001] The present invention relates to an image generating apparatus, an image generating method, and a recording medium.

[0002] Patent Document 1 discloses a vehicle distance measuring device having means for measuring the line-of-sight direction of a driver and capable of displaying only the information required by the driver among the vehicle distance information of the vehicle ahead in the line-of-sight direction of the driver to the driver. According to this document, this vehicle distance measuring device obtains the vehicle distance and azimuth data of the preceding vehicle by a vehicle distance measuring unit. Then, the vehicle distance measuring device measures the line-of-sight direction of the driver by a line-of-sight measuring unit to obtain the coordinate data of the fixation point. Then, this vehicle distance measuring device specifies the "preceding vehicle being stared at by the driver" from the vehicles existing ahead in the line-of-sight direction of the driver, and describes that the vehicle distance information with the preceding vehicle is displayed on the windshield by HUD (Head Up Display).

[0003] Japanese Patent Application Laid-Open No. 7-055941

[0004] There are needs such as wanting to know in real time whether a driver driving a moving body is paying appropriate attention to oncoming vehicles or pedestrians, and wanting to record the evidence. In this regard, the vehicle distance measuring device of Patent Document 1 only displays the vehicle distance information of the preceding vehicle stared at by the driver.

[0005] An object of the present disclosure is to provide an image generating apparatus, an image generating method, and a recording medium capable of displaying, presenting, or recording the line-of-sight direction of a driver or a passenger of a moving body during operation of the moving body.

[0006] According to a first aspect, there is provided an image generating apparatus including a line-of-sight direction specifying means for specifying the line-of-sight direction of a user from an image of the user photographed by a first camera for photographing the user, and an image generating means for creating an image in which a figure is superimposed and displayed along the line-of-sight direction on an image photographed by a second camera for photographing the outside of the moving body on which the user is riding, based on the line-of-sight direction of the user.

[0007] From a second perspective, a display control method is provided which identifies the user's gaze direction from an image of the user captured by a first camera that photographs the user, and, based on the user's gaze direction, creates an image in which a figure is superimposed along the gaze direction on an image captured by a second camera that photographs the outward direction of the mobile body in which the user is riding.

[0008] From a third perspective, a recording medium is provided that contains a program that causes a computer to perform the following steps: a process of determining the direction of the user's gaze from an image of the user taken by a first camera that photographs the user; and a process of creating an image in which a figure is superimposed along the direction of the gaze on an image taken by a second camera that photographs the outward direction of the mobile body in which the user is riding, based on the direction of the user's gaze.

[0009] According to this disclosure, it is possible to display, present, or record the line of sight of a driver or passenger in a moving vehicle while the vehicle is in motion.

[0010] This is a diagram showing one configuration of the present disclosure. This is a flowchart showing the operation of the present disclosure. This is a diagram for explaining the operation of the present disclosure. This is a diagram for explaining the operation of the present disclosure. This is a diagram showing one configuration of the present disclosure. This is a diagram for explaining calibration when installing a camera in a vehicle. This is another diagram for explaining calibration when installing a camera in a vehicle. This is a flowchart for explaining the operation of the present disclosure. This is a diagram showing an example of a screen displayed on the display device of an in-vehicle device (out-of-view display). This is a diagram for explaining the operation of the present disclosure. This is a diagram showing an example of a screen displayed on the display device of an in-vehicle device. This is a diagram showing another configuration of the present disclosure. This is a diagram showing another example of a screen displayed on the display device of an in-vehicle device. This is a diagram showing another example of a screen displayed on the display device of an in-vehicle device. This is a diagram showing another configuration of the present disclosure. This is a diagram showing another configuration of the present disclosure. This is a diagram showing another configuration of the present disclosure. This is a diagram showing the configuration of the computer constituting the display control device of the present disclosure.

[0011] First, an overview of one embodiment of this disclosure will be described with reference to the drawings. In this disclosure, the drawings are associated with one or more embodiments. The reference numerals in the drawings appended to this overview are provided for convenience as examples to aid understanding and are not intended to limit this disclosure to the illustrated embodiments. In addition, the connecting lines between blocks in the drawings and other references referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows schematically indicate the flow of the main signal (data) and do not exclude bidirectionality. The program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and a display device as needed. This computer device is also configured to communicate with devices (including computers) inside or outside the device via the communication interface, whether wired or wireless. In addition, there are ports or interfaces at the input / output connection points of each block in the figures, but these are omitted from the illustration.

[0012] In one embodiment, this disclosure can be realized by an image creation device 10, as shown in Figure 1, which includes a gaze direction identifying means 11 and an image creation means 12. More specifically, the gaze direction identifying means 11 identifies the user's gaze direction from an image of the user captured by a first camera C1 that photographs the user. Based on the user's gaze direction, the image creation means 12 creates an image in which a figure is superimposed along the gaze direction on an image captured by a second camera C2 that photographs the outward direction of the mobile body on which the user is riding.

[0013] The image creation device 10 configured as described above operates as follows. First, the image creation device 10 identifies the direction of the user's gaze from the image of the user captured by the first camera C1 that photographs the user (step S01 in Figure 2). For example, the image creation device 10 identifies the direction of the user's gaze based on the image of the user's eyes in the image of the user captured by camera C1. Figure 3 shows the direction of gaze ED identified when the user is staring at the vehicle V1 in front of them. The dashed lines in Figure 3 indicate the contours of the user's head and shoulders.

[0014] Next, the image creation device 10 creates an image in which a figure is superimposed along the line of sight on an image captured by a second camera C2 that photographs the outward direction of the mobile vehicle in which the user is riding, based on the user's line of sight direction ED (step S02 in Figure 2). Figure 4 shows a state in which multiple figures DEV are superimposed along the identified line of sight direction ED on an image captured by the second camera C2. In the example in Figure 4, the presence of the figures DEV indicates that the user is looking at the vehicle V1 in front, not the vehicle V2 in the adjacent lane. As a mechanism to accurately represent the line of sight direction ED on the image from the second camera C2, a model plotting the line of sight direction can be created in a virtual space corresponding to the image from camera C2, and an image of that model as seen from the viewpoint of camera C2 can be created.

[0015] As explained above, this disclosure makes it possible to display, present, or record the user's line of sight, making it easier to understand what the user is focusing on. This is because it employs a configuration in which multiple shapes are arranged along the user's line of sight. In the example in Figure 4, five circles are drawn as multiple shapes, but the shape and number of shapes to be displayed are not limited to this. For example, a configuration in which multiple rectangles are superimposed along the user's line of sight is also possible. In the example in Figure 4, the multiple markers are displayed such that they become smaller as they move away from the user in the line of sight, but the size of the shapes is not limited to this. For example, multiple circles of the same size may be superimposed along the user's line of sight. The colors of the multiple shapes can also be changed as appropriate. For example, colors may be predetermined according to the distance from the user, and circles of different colors may be drawn according to that rule. For example, by changing the color of the shapes according to the distance, such as drawing a red circle at a distance of 5m from the user and a blue circle at a distance of 10m from the user, it becomes easier to understand the distance to an object in the user's line of sight.

[0016] [First Embodiment] Next, a first embodiment in which an in-vehicle terminal functions as an image creation device of the present disclosure will be described. Figure 5 is a diagram showing one configuration of the present disclosure. Referring to Figure 5, a configuration is shown that includes a camera C1 for photographing a driver U sitting in the driver's seat of a vehicle, a camera C2 for photographing the area in front of the vehicle, and an in-vehicle terminal 100 connected to these cameras C1 and C2. As camera C1, a camera that constitutes an in-cabin sensing system or a drive recorder camera capable of photographing the interior of a vehicle can be used. As camera C2, a drive recorder capable of photographing the area in front of the vehicle or a camera for checking the surroundings can be used.

[0017] The in-vehicle terminal 100 includes a gaze direction identification unit 101, an image creation unit 102, and a display unit 103. Such an in-vehicle terminal 100 can utilize a car navigation system or an in-vehicle computer equipped with a display device as the display unit 103. Furthermore, the in-vehicle terminal 100 corresponds to the image creation device 10 described above.

[0018] The gaze direction identification unit 101 identifies the gaze direction of the driver U from the image of the driver U's eyes captured by the camera C1, and sends the identified gaze direction information to the image creation unit 102. Therefore, the gaze direction identification unit 101 corresponds to the gaze direction identification means 11 described above.

[0019] The image creation unit 102 creates an image in which multiple shapes are superimposed along the direction of the driver U's line of sight on the image captured by the camera C2, and sends it to the display unit 103. Therefore, the image creation unit 102 corresponds to the image creation means 12 described above.

[0020] The display unit 103 displays the image created by the image creation unit 102. This display unit 103 can be a display device installed on the vehicle's instrument panel or center cluster. Alternatively, a head-up display (HUD) can be used as the display unit 103.

[0021] First, we will explain the calibration required for the in-vehicle terminal 100 to identify the user's gaze direction from the images of the driver U's eyes captured by cameras C1 and C2, and to superimpose and display a figure along that gaze direction.

[0022] <Calibration> Figure 6 is a diagram illustrating the calibration process when installing cameras in a vehicle. Figure 6 is a top-down view of the driver's seat of the vehicle, with the right side of the diagram being the front of the vehicle. Camera C1 is pointed towards the driver's seat and fixed in a position to capture the driver U. Camera C2 is fixed in a position to capture the front of the vehicle, such as the dashboard.

[0023] The information necessary to determine the direction of the driver U's gaze from the image of the driver U's eyes captured by camera C1 is the angle Deg1 between camera C1 and the forward direction from the driver U's eye position. This angle Deg1 can be calculated by measuring the horizontal position (c,d) of camera C1 with the driver U's eye position as the origin (0,0) (tan Deg1 = d / c). Alternatively, this angle Deg1 can also be calculated by fixing camera C1 in a predetermined position and then calculating the position of camera C1 with the driver U's eye position as the origin (0,0) from the front-to-back position of the driver's seat and the reclining angle.

[0024] Alternatively, this angle Deg1 can also be calculated by setting the camera C1 to a predetermined position and photographing the driver U facing forward. When performing this calculation, the in-vehicle terminal 100 may instruct the driver U to face forward and remain still for a certain period of time, either through a display on the display device or by voice.

[0025] Next, the horizontal position (x0, y0) of camera C2 is determined. The position (x0, y0) can be determined by measuring the distances x0 and y0 from the origin, respectively. Alternatively, the position (x0, y0) can be determined by measuring x0 and angle Deg2. Note that angle Deg2 is the angle between the position (x0, y0) and the forward direction from the driver U's eye position. The position (x0, y0) can also be determined by measuring y0 and angle Deg2. This is because angle Deg2, x0, and y0 are related by tan Deg2 = y0 / x0. Furthermore, the distance D2 between the horizontal position (x0, y0) of camera C2 and the driver U's eye position can be determined using at least two of x0, y0, and angle Deg2, along with trigonometric functions. Furthermore, if the distance D2 can be determined by measurement, the remaining values ​​may be calculated using the distance D2, x0, y0, angle Deg2, and trigonometric functions.

[0026] Note that Deg3 in Figure 6 represents the field of view of camera C2. This field of view can be determined from the focal length of the camera C2's lens, etc. Of course, Deg3 can also be determined by measuring it from the image captured by camera C2.

[0027] The above explanation described horizontal calibration, but the same calibration process is performed for vertical calibration. The principle is the same as for horizontal calibration, so the explanation is omitted.

[0028] <Calculation of Driver U's Gaze Position> Once the above calibration is complete, it becomes possible to calculate the position of the object that Driver U is looking at (gaze position) from the image of Driver U's eyes. Below, an example of how to calculate the position of the object that Driver U is looking at (gaze position) will be explained with reference to Figure 7.

[0029] As mentioned above, the following parameters are known: Deg3: Camera field of view x0, y0: Position of camera C2 with driver U's eye position as the origin (0,0) Deg1: Angle between the forward direction from driver U's eye position and camera C1

[0030] First, we calculate the angle Deg4 of the driver U's line of sight relative to the forward direction from the user's eye position. As mentioned above, the angle Deg1 between camera C1 and the forward direction from the driver U's eye position is known, so we can calculate Deg4 by finding the angle between the forward direction from the driver U's eye position and the driver U's line of sight.

[0031] Once Deg4 is determined, the horizontal coordinates of positions T1, T2, and T in the line of sight can be found using the following equations. Here, x1 and x2 are the predetermined distances in the forward direction from the tip of camera C2. Also, x0 is obtained when calculating the distance D mentioned above. Coordinates of T1 = (x0 + x1, (x0 + x1) × tan Deg4) Coordinates of T2 = (x0 + x1 + x2, (x0 + x1 + x2) × tan Deg4)

[0032] To convert the coordinates of T1 and T2 obtained in this way to the coordinates of camera C2's viewpoint, we can convert them to coordinates based on the aforementioned x0 and y0. For example, if the position of T1 is (tx1, ty1), the coordinates of T1 from camera C2's viewpoint will be (tx1 - x0, ty1 - y0). Similarly, if the position of T2 is (tx2, ty2), the coordinates of T2 from camera C2's viewpoint will be (tx2 - x0, ty2 - y0). Note that if ty1 - y0 and ty2 - y0 are negative values, it means the user's line of sight is directed to the right of the front of camera C2.

[0033] As described above, the in-vehicle terminal 100 calculates the driver U's line of sight position from the image from camera C1. The line connecting this line of sight position and the driver U's eyes represents the direction of the line of sight. This direction of line of sight can be said to specify the direction of the line of sight in real space, starting from the position of the driver U's eyes. The in-vehicle terminal 100 then overlays a figure onto the image from camera C2 to make it easier to understand this direction of line of sight in real space.

[0034] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 8 is a flowchart illustrating the operation of this disclosure. Referring to Figure 8, first, the in-vehicle terminal 100 acquires the image captured by the camera C1 (step S001).

[0035] Next, the in-vehicle terminal 100 identifies the driver U's gaze position and direction from the image captured by camera C1 using the method described above (step S002). Next, the in-vehicle terminal 100 determines whether or not the driver U's gaze position is within the image of camera C2 (step S003). Whether or not the driver U's gaze position is within the image of camera C2 can be determined by whether or not the gaze position is within the field of view Deg3 of camera C2 in Figure 7.

[0036] If it is determined that the driver U's gaze position is not on the image of camera C2, or if the driver U's gaze position cannot be identified (No. in step S003), the in-vehicle terminal 100 creates and displays an image with a gaze-off indicator on the image of camera C2 (step S004). Figure 9 shows an example of a gaze-off indicator on the image of camera C2. In the example in Figure 9, since the driver U's gaze position is off to the left of the image of camera C2, a marker M indicating the left side is displayed. By displaying such information, it is possible to understand which direction outside the image the driver U is looking. In the example in Figure 9, a triangular marker M is used to indicate the direction in which the driver U's gaze position is off, but the format of representing the direction in which the driver U's gaze position is off is not limited to this. For example, the direction in which the driver U's gaze position is off can be indicated by displaying text or a band at the edge of the screen in that direction. Similarly, if the driver U's gaze position cannot be identified, a message indicating that the gaze position could not be identified may be displayed.

[0037] On the other hand, if it is determined that the driver U's line of sight is on the image of camera C2 (Yes in step S003), the in-vehicle terminal 100 places a marker indicating the line of sight along the direction of the line of sight in a virtual space corresponding to the area captured by camera C2 (step S005).

[0038] Next, the in-vehicle terminal 100 calculates the projection position and projection size of the markers placed above from the viewpoint of camera C2 (step S006). Finally, the in-vehicle terminal 100 creates and displays an image in which the markers are superimposed on the image from camera C2 according to the calculation results (step S007).

[0039] <Drawing a figure indicating the direction of gaze> Here, we will explain how to superimpose a marker of an appropriate size onto the image from camera C2. Figure 10 is a diagram showing the driver's seat of a vehicle viewed from above, with a marker indicating the direction of gaze, the screen SCR of the display unit 103, etc.

[0040] Once the driver U's line of sight position and direction are determined in step S004 described above, markers T1 to T3 can be placed along the line of sight from the tip of camera C2. At this time, the markers are placed at positions that are a predetermined distance x1, x2, x3 from the tip of camera C2 in the direction in front of camera C2. Next, the projection position and projection size of markers T1 to T3 from the viewpoint of camera C2 are calculated. For example, if marker T3 is the furthest marker, marker T3 will be projected at the same size as the specified marker. On the other hand, markers T1 and T2 are located closer to camera C2 than marker T3. Therefore, if they are projected at the same position as marker T3, they will appear larger in size because they are closer to camera C2. For example, if distance x3 is three times distance x1 and distance x2 is twice distance x1, marker T1 will be three times the size of marker T3, and marker T2 will be twice the size of marker T1. In other words, markers T1 to T3 are displayed so that they become smaller the further they are from the user. By placing these markers, with their calculated positions and sizes, on the screen SCR of the display unit 103 that displays the image from camera C2, a perspective-based display can be achieved.

[0041] The above explanation described how to draw shapes horizontally, but the same process is applied to the vertical direction. Since the principle is the same as for horizontal drawing, the explanation will be omitted.

[0042] Figure 11 shows an example of a screen displayed on the display unit 103 of the in-vehicle terminal 100. In the example of Figure 11, based on the calculation results described above, markers T1 to T5 are displayed in the direction of the vehicle V1 that the driver U is looking at, with the size decreasing as the vehicle moves away from the camera C2. By showing an image in which such markers T1 to T5 are superimposed, it becomes easier to understand what the driver U is looking at. Also, as mentioned above, since the size of the markers T1 to T5 decreases as the vehicle moves away from the camera C2, it is also easier to understand the direction of the gaze.

[0043] In a more desirable configuration, a standard size for markers T1 to T5 can be determined, and the size of the markers can be changed according to the distance from the tip of camera C2 in the forward direction. In addition, the size of the markers on the image from camera C2 can be set to a predetermined standard size (specified size). For example, in Figure 11, markers T1 to T5 are circles with a radius of 50 cm in virtual space, and their size is adjusted according to the distance from the tip of camera C2 in the forward direction so that the markers appear as objects of the specified size. In this way, it becomes possible to estimate the size of 1 m from the diameter of the circles (markers T1 to T5) on the image. This makes it possible for the driver U to grasp the width of the object they are looking at, as well as the objects in front of and behind it, and the lane width. For example, in the example in Figure 11, by comparing the width of vehicle V1 on the image with the diameter of the circle, it becomes possible to estimate that the width of vehicle V1 that driver U is looking at is approximately 2 m. Similarly, by comparing the width of the lane on the image with the diameter of the circle displayed at approximately the same distance as its position, it becomes possible to estimate that the width of the lane is approximately 4 m.

[0044] As described above, according to the present embodiment, the line of sight of the driver U can be displayed on the image of the camera C2 that captures the traveling direction of the vehicle. When the driver U is appropriately paying attention to the vehicles and pedestrians in the traveling direction, the graphic in the traveling direction is constantly in a moving state. Conversely, when the movement of the driver U's line of sight is small, it is suspected that the driver U is paying too much attention to a specific object or is driving absent-mindedly. Such display of the line of sight can also be used for guidance in a driving school or the like and guidance for drivers of various transportation agencies.

[0045] Also, as described above, by recording an image with the line-of-sight direction superimposed, the line-of-sight direction of the driver U during driving can be recorded. For example, when using the camera of a drive recorder as the camera C2, the data recorded in this way can be used as evidence to prove the line-of-sight direction of the driver U during driving. And when the driver U encounters an accident or the like, it can be used as evidence to show that the driver U was paying due attention at that time.

[0046] In the above-described embodiment, the camera C2 has been described as using the camera of a drive recorder capable of capturing the front of the vehicle. However, as the camera C2, a camera that captures the outer direction (surrounding direction) of other vehicles can also be used. For example, when using a camera that captures the side of the vehicle as the camera C2, the line of sight in the side direction of the vehicle of the driver U can be displayed or recorded.

[0047] [Second Embodiment] Next, a second embodiment in which a computing function for calculating the eye-opening rate or the eye-closing rate is added to the in-vehicle terminal 100a will be described. FIG. 12 is a diagram showing another configuration of the present disclosure. The difference from the first embodiment shown in FIG. 5 is that an eye-opening rate calculation unit 104 is added to the in-vehicle terminal 100a, and the image creation unit 102a changes the display of the graphic using the eye-opening rate calculated by the eye-opening rate calculation unit 104. Since other configurations are substantially the same as those of the first embodiment, the following description will focus on the differences in their operations.

[0048] The eye-opening rate calculation unit 104 calculates an eye-opening rate indicating the degree of opening of the driver U's eyes based on an image of the driver U captured by the camera C1 or the like, and outputs it to the image creation unit 102a. This eye-opening rate can be represented, for example, by values or percentages in three levels: high, medium, and low. Such an eye-opening rate can be estimated, for example, based on an image of the driver U's eye portion. For example, when the driver U is prone to closing their eyes, the eye-opening rate is "low". Also, for example, when the driver U's eyes are open for a certain period, the eye-opening rate is "high". Note that the eye-opening rate calculation unit 104 may calculate an eye-opening rate indicating the degree of opening of the driver U's eyes based on images of the camera C1 in a past certain period. Also, instead of the eye-opening rate, similar indicators such as a closing rate indicating the degree of closing of the eyes or an arousal level indicating the degree of arousal of the driver U can be used.

[0049] Similar to the first embodiment, the image creation unit 102a creates an image in which a plurality of figures are superimposed along the line-of-sight direction on an image captured by the camera C2 based on the line-of-sight direction of the driver U. At that time, the image creation unit 102a changes the shape of the plurality of figures using the eye-opening rate calculated by the eye-opening rate calculation unit 104.

[0050] FIG. 13 is a diagram showing an example of a screen displayed on the display unit 103 of the in-vehicle terminal 100a. For example, the image creation unit 102a displays a figure that is vertically compressed and deformed according to the eye-opening rate. In the example of FIG. 13, the shapes of the markers T1 to T5 are not perfect circles but vertically compressed elliptical shapes. This indicates that the eye-opening rate of the driver U is, for example, 70% or "medium".

[0051] As the eye-opening rate decreases, the image creation unit 102a displays the figure with a stronger degree of compression. In the example of FIG. 14, the shapes of the markers T1 to T5 are elliptical shapes that are vertically more greatly compressed than in FIG. 13. This indicates that the eye-opening rate of the driver U is, for example, 50% or "low".

[0052] As described above, according to this embodiment, in addition to the driver U's line of sight, the driver U's eye-opening rate can be displayed on the image from the camera C2 that photographs the direction of travel of the vehicle. This makes it easier to estimate the driver U's condition and the cause when the driver U's eye movement is small. For example, as shown in Figure 14, if the shape of markers T1 to T5 is greatly compressed vertically, it can be seen that the driver's eye-opening rate has decreased. This makes it easier to understand not only the driver U's line of sight but also the driver U's condition in instruction at driving schools and for bus crew training.

[0053] Conversely, if the markers T1 to T5 are displayed in a state close to a perfect circle on the in-vehicle terminal 100a of this embodiment, it indicates that the driver U had a high rate of eye opening. The image obtained in this way may be saved as evidence. The data saved in this way can be used, for example, as evidence that, if the driver U is involved in an accident, the driver U was paying appropriate attention at that time and also had a high rate of eye opening.

[0054] [Third Embodiment] Next, a second embodiment will be described in which the server 200, which receives images from the in-vehicle terminal 100b, is equipped with a display control device function. Figure 15 is a diagram showing another configuration of the present disclosure. Referring to Figure 15, a configuration is shown in which the in-vehicle terminal 100b and the server 200 are connected via a network. Cameras C1 and C2 are the same as in the first and second embodiments, so their description will be omitted.

[0055] The in-vehicle terminal 100b is equipped with a transmission unit 105. The transmission unit 105 transmits images from cameras C1 and C2 to the server 200 via the network.

[0056] The server 200 includes a gaze direction identification unit 201 and an image creation unit 202. The server 200 is also connected to the display device 300. A server 200 with this configuration corresponds to the image creation device described above.

[0057] The gaze direction identification unit 201 identifies the gaze direction of the driver U from the image of the driver U's eyes received from the in-vehicle terminal 100b, and sends the identified gaze direction information to the image creation unit 202.

[0058] The image creation unit 202 creates an image on the image from the camera C2 received from the in-vehicle terminal 100b, based on the driver U's line of sight, by superimposing multiple shapes along the line of sight, and displays it on the display device 300.

[0059] The operation of the server 200 described above is the same as in the first embodiment. The server 200 identifies the driver U's line of sight direction from the image of camera C1 and displays multiple shapes superimposed on the image of camera C2 along this line of sight direction.

[0060] As described above, the display control device of this disclosure can be implemented using a server 200 that receives images from cameras C1 and C2 from an in-vehicle terminal 100b. Such a server 200 can be located in a control center that controls buses and trains. By using the in-vehicle terminals of these buses and trains as the in-vehicle terminal 100b, it becomes possible to display the gaze direction of the driver U operating these vehicles on the display device 300 of the control center. Such a display of the gaze direction can be used to support and guide the driver U from the control center.

[0061] [Fourth Embodiment] In the third embodiment described above, the server 200 was described as receiving images from cameras C1 and C2 directly from the in-vehicle terminal 100b, but the images from cameras C1 and C2 may be stored in storage or the like. Figure 16 is a diagram showing another configuration of the present disclosure. Referring to Figure 16, a configuration is shown in which storage 400 is placed between the in-vehicle terminal 100c and the server 200a.

[0062] The storage 400 includes a first recording unit (first recording means) 401 for recording images from camera C1 and a second recording unit (second recording means) 402 for recording images from camera C2. Such storage may be the same type used in cloud-based drive recorder services or telematics services.

[0063] The in-vehicle terminal 100c transmits images from cameras C1 and C2 to the storage 400. The storage 400 stores the images received from the in-vehicle terminal 100c in the first recording unit (first recording means) 401 and the second recording unit (second recording means) 402, respectively.

[0064] The gaze direction identification unit 201a of the server 200a acquires an image of the driver U's eyes from the first recording unit (first recording means) 401 of the storage 400, identifies the driver U's gaze direction, and sends the identified gaze direction information to the image creation unit 202a.

[0065] The image creation unit 202a of the server 200a acquires the image from camera C2 retrieved from the second recording unit (second recording means) 402 of the storage 400. Then, based on the driver U's line of sight direction, the image creation unit 202a creates an image in which multiple shapes are superimposed on the image from camera C2 along the line of sight direction, and displays it on the display device 300.

[0066] As described above, according to this embodiment, evidence can be created to prove the direction of driver U's gaze while driving, as needed. For example, if driver U is involved in an accident, evidence can be created to show that driver U was paying appropriate attention at the time, or conversely, that driver U was looking away.

[0067] [Fifth Embodiment] In the third embodiment described above, the driver U's gaze direction was displayed on the display device 300, but the image created by the image creation unit 202a may be saved. Figure 17 is a diagram showing another configuration of the present disclosure. Referring to Figure 17, a configuration is shown in which an in-vehicle terminal 100b and a server 200d are connected via a network. The difference from the third embodiment is that the server 200d is equipped with a gaze transition recording unit 203, and the image creation unit 202d saves the image created in the gaze transition recording unit 203. The other configurations are the same as in the third embodiment, so their description is omitted.

[0068] The image creation unit 202d creates data for superimposing multiple shapes along the direction of the driver U's gaze onto the image from the camera C2 received from the in-vehicle terminal 100b, based on the direction of the driver U's gaze, and stores it in the gaze transition recording unit 203.

[0069] The gaze transition recording unit 203 stores data (gaze transition data) for superimposing multiple shapes onto the image from camera C2 along the direction of gaze. The data may be stored in the form of a second image in which multiple shapes are drawn along the direction of gaze on the image from camera C2. Alternatively, the data may be stored in the form of data for drawing multiple shapes along the direction of gaze on the image from camera C2 in synchronization with the image from camera C2 (layer data for drawing the multiple shapes).

[0070] According to this embodiment, the driver U's gaze direction while driving can be recorded. The data accumulated in the gaze shift recording unit 203 in this way can be used as evidence to prove the driver U's gaze direction while driving. Furthermore, if driver U is involved in an accident or other incident, it can be used as evidence to show that driver U was paying appropriate attention at that time.

[0071] While the embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of this disclosure. For example, the network configurations, element configurations, and data representations shown in the drawings are examples to aid in understanding this disclosure and are not limited to the configurations shown in these drawings.

[0072] For example, a function equivalent to the eye-opening rate calculation unit 104 of the second embodiment may be added to the servers 200, 200a, and 200c of the third to fifth embodiments described above. By doing so, the eye-opening rate of the driver U can be displayed on the image of the camera C2 projected on the display device 300, in addition to the driver U's line of sight, and the data for this purpose can be stored.

[0073] Furthermore, in each of the embodiments described above, examples were given in which the direction of the driver U's gaze was determined from an image of the driver U's eyes. However, it is conceivable that an image of the eyes may not be obtainable, for example, if the driver U is wearing sunglasses. In this case, the direction of the gaze may be determined from the orientation of the driver U's face or the orientation of the sunglasses.

[0074] Furthermore, in the embodiments described above, the target for displaying, presenting, or recording the direction of gaze was the driver U. However, the direction of gaze of other passengers on the moving vehicle can also be displayed, presented, or recorded. In this case, calibration would be performed based on the assumption of the target passenger to determine their direction of gaze.

[0075] (Regarding Hardware Configuration) In each embodiment of this disclosure, each component of each device represents a functional unit block. Some or all of each component of each device is realized by any combination of an information processing device 900 and a program, for example, as shown in Figure 18. Figure 18 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes each component of each device. The information processing device 900 includes, as an example, the following configuration: ・CPU (Central Processing Unit) 901 ・ROM (Read Only Memory) 902 ・RAM (Random Access Memory) 903 ・Program 904 loaded into RAM 903 ・Storage device 905 that stores the program 904 ・Drive device 907 that reads and writes to the recording medium 906 ・Communication interface 908 that connects to a communication network 909 ・Input / output interface 910 that performs data input and output ・Bus 911 that connects each component

[0076] Each component of each device in each embodiment is realized by the CPU 901 acquiring and executing a program 904 that realizes these functions. That is, the CPU 901 in Figure 18 executes a gaze direction identification program and an image creation program, and performs update processing of each calculation parameter held in RAM 903, storage device 905, etc. The program 904 that realizes the functions of each component of each device is, for example, stored in storage device 905 or ROM 902 in advance and read by the CPU 901 as needed. The program 904 may be supplied to the CPU 901 via a communication network 909, or it may be stored in a recording medium 906 in advance, and the drive device 907 may read the program and supply it to the CPU 901.

[0077] There are various variations in how each device is implemented. For example, each device may be implemented by any combination of a separate information processing device 900 and a program for each component. Alternatively, multiple components of each device may be implemented by any combination of a single information processing device 900 and a program. That is, each part (processing means, function) of the image creation device described above can be implemented by a computer program that causes a processor mounted on the device to execute the above-described processes using its hardware.

[0078] Furthermore, some or all of the components of each device are realized by other general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may consist of a single chip or multiple chips connected via a bus.

[0079] Some or all of the components of each device may be realized by a combination of the circuits and programs described above.

[0080] When some or all of the components of each device are implemented by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form in which each is connected via a communication network, such as a client-and-server system or a cloud computing system.

[0081] The embodiments described above are preferred embodiments of this disclosure and do not limit the scope of this disclosure to these embodiments alone. That is, a person skilled in the art can modify or substitute the embodiments described above to construct various modified forms without departing from the gist of this disclosure.

[0082] Some or all of the above embodiments may also be described as follows, but are not limited to these.

[0083] [Note 1] An image creation device comprising: a gaze direction identifying means for identifying the direction of the user's gaze from an image of the user captured by a first camera that photographs the user; and an image creation means for creating an image in which a figure is superimposed along the gaze direction on an image captured by a second camera that photographs the outward direction of a mobile body in which the user is riding, based on the direction of the user's gaze. [Note 2] The gaze direction identified by the image creation device described above is the gaze direction in real space starting from the position of the user's eyes, and the image creation means can be configured to create an image in which a figure is superimposed along the gaze direction in real space. [Note 3] The figure superimposed by the image creation device described above is composed of a plurality of markers displayed along the gaze direction, and the image can be configured to create an image in which the plurality of markers are displayed such that the markers become smaller as they move away from the user in the gaze direction. [Note 4] The image creation means of the image creation device described above can be configured to create an image in which the size of the plurality of markers is adjusted so that they correspond to the size of an object of a predetermined size located at a predetermined distance from the user's position. [Note 5] The image creation means of the image creation device described above can be configured to create an image on the image captured by the second camera that indicates the user's gaze direction is outside the screen if the gaze direction identification means cannot identify the user's gaze direction. [Note 6] The image creation means of the image creation device described above can be configured to create an image on the image captured by the second camera that indicates the user's gaze direction is outside the screen if it determines that the user's gaze direction is not on the image captured by the second camera. [Note 7] The image creation device described above can access a first recording means for recording an image of the user captured by the first camera, and a second recording means for recording an image captured by the second camera. The gaze direction identification means and the image creation means can be configured to retrieve images from the first recording means and the second recording means, respectively, and create an image on the image captured by the second camera in which a figure is superimposed along the gaze direction.[Note 8] The image creation means of the image creation apparatus described above may be configured to create data for superimposing a figure along the line of sight onto an image captured by the second camera and to store it in a predetermined storage device. [Note 9] The image creation apparatus described above may further include a calculation means for calculating the user's eye-opening rate or eye-closing rate from the user's image, and the image creation means may be configured to create an image in which the shape of the figure is changed based on the eye-opening rate or eye-closing rate. [Note 10] An image creation method for determining the user's line of sight from an image of the user captured by a first camera that photographs the user, and creating an image in which a figure is superimposed along the line of sight onto an image captured by a second camera that photographs the outward direction of a mobile body on which the user is riding, based on the user's line of sight. [Note 11] A recording medium that records a program causing a computer to perform the following: a process of identifying the direction of the user's gaze from an image of the user taken by a first camera that photographs the user; and a process of creating an image in which a figure is superimposed along the direction of the gaze on an image taken by a second camera that photographs the outward direction of the mobile body on which the user is riding, based on the direction of the user's gaze. The forms described in each of the above notes can be combined with each other after making the necessary modifications. For example, a configuration that combines the contents of Note 2 and the contents of Note 3 is also included in the scope of disclosure of this specification. Specifically, this image creation device identifies the direction of the gaze in real space starting from the position of the user's eyes, and displays a plurality of markers such that the markers become smaller as the distance from the user in this direction of gaze increases. The forms described in Notes 10 to 11 can be expanded into the forms described in Notes 2 to 9, similar to Note 1.

[0084] Furthermore, each disclosure in the above-mentioned patent documents is incorporated into this document by reference and may be used as the basis or part of this disclosure as necessary. Within the framework of this disclosure (including the claims), further modifications and adjustments to the embodiments or examples are possible based on their fundamental technical concept. Also, within the framework of this disclosure, various combinations or selections (including partial deletions) of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, this disclosure naturally includes the entire disclosure, including the claims, and various modifications and alterations that a person skilled in the art could make in accordance with the technical concept. In particular, with respect to the numerical ranges described in this document, any numerical value or sub-range included within that range should be interpreted as being specifically described, even if not otherwise stated. Furthermore, each disclosure item of the above-mentioned cited documents may, as necessary, be used in part or in whole as part of this disclosure, in accordance with the spirit of this disclosure, and this is also considered to be included in the disclosure items of this application.

[0085] 10 Image creation device 11 Gaze direction identification means 12 Image creation means 100, 100a to 100c In-vehicle terminal 101, 201, 201a Gaze direction identification unit 102, 102a, 202, 202a, 202d Image creation unit 103 Display unit 104 Eye opening rate calculation unit 105 Transmission unit 200, 200a, 200c Server 203 Gaze transition recording unit 300 Display device 400 Storage 401 First recording unit (first recording means) 402 Second recording unit (second recording means) 900 Information processing device 901 CPU (Central Processing Unit) 902 ROM (Read Only Memory) 903 RAM (Random Access Memory) 904 Program 905 Storage device 906 Recording medium 907 Drive device 908 Communication interface 909 Communication network 910 Input / output interface 911 Bus C1 First camera (camera) C2 Second camera (camera) DEV Figure ED Line of sight M, T1-T5 Marker U Driver V1 Vehicle

Claims

1. An image creation device comprising: a gaze direction identifying means for identifying the direction of the user's gaze from an image of the user captured by a first camera that photographs the user; and an image creation means for creating an image in which a figure is superimposed along the gaze direction on an image captured by a second camera that photographs the outward direction of a mobile body in which the user is riding, based on the direction of the user's gaze.

2. The image creation device according to claim 1, wherein the line of sight direction is the line of sight direction in real space starting from the position of the user's eyes, and the image creation means creates an image in which figures are superimposed along the line of sight direction in real space.

3. The image creation device according to claim 1 or 2, wherein the figure is composed of a plurality of markers displayed along the line of sight, and the image creation means creates an image in which the plurality of markers are displayed such that the markers become smaller as they move away from the user in the line of sight.

4. The image creation device according to claim 3, wherein the image creation means creates an image in which the size of the plurality of markers is adjusted so that the size of the markers corresponds to the size of an object of a predetermined size located at a predetermined distance from the user's position.

5. If the gaze direction identification means fails to identify the user's gaze direction, the image creation means creates an image on the image captured by the second camera indicating that the user's gaze direction is outside the screen, an image creation device according to any one of claims 1 to 4.

6. If the image creation means determines that the user's line of sight is not on the image captured by the second camera, the image creation means creates an image on the image captured by the second camera indicating that the user's line of sight is off-screen, according to any one of claims 1 to 5.

7. An image creation device according to any one of claims 1 to 6, further comprising: a first recording means for recording an image of the user captured by the first camera; and a second recording means for recording an image captured by the second camera, wherein the gaze direction identifying means and the image creation means retrieve images from the first recording means and the second recording means, respectively, and create an image in which a figure is superimposed on the image captured by the second camera along the gaze direction.

8. An image creation device according to any one of claims 1 to 7, wherein the image creation means creates data for superimposing a figure on an image captured by the second camera along the line of sight and stores it in a predetermined storage device.

9. An image creation device comprising a calculation means for calculating the user's eye-opening rate or eye-closed rate from the user's image, wherein the image creation means creates an image in which the shape of the figure is changed based on the eye-opening rate or eye-closed rate, any one of claims 1 to 8.

10. An image creation method comprising: determining the direction of the user's gaze from an image of the user captured by a first camera that photographs the user; and creating an image in which a figure is superimposed along the direction of the gaze on an image captured by a second camera that photographs the outward direction of the mobile body in which the user is riding, based on the direction of the user's gaze.

11. A recording medium that records a program that causes a computer to perform the following steps: a process of determining the direction of the user's gaze from an image of the user taken by a first camera that photographs the user; and a process of creating an image in which a figure is superimposed along the direction of the gaze on an image taken by a second camera that photographs the outward direction of the mobile body in which the user is riding, based on the direction of the user's gaze.

Images