Gaze guidance device, gaze guidance method, gaze guidance program, and storage medium

Abstract

To guide the gaze.SOLUTION: A calculation unit 131 of a gaze guidance device 10 calculates a first order by arranging multiple regions within an image, determined based on the degree of gaze attractiveness, in descending order of peak values of the degree of gaze attractiveness. The determination unit 132 determines whether the specified second order is the same as the first order. The guidance unit 133 performs gaze guidance according to the second order when the determination unit 132 determines that the first order and the second order are different.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a visual line guidance device, a visual line guidance method, a visual line guidance program, and a storage medium. [Background technology]

[0002] Conventionally, there is known a technique for estimating information regarding the movement of a gaze point based on visual saliency calculated from an image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-77248 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have the problem that it can be difficult to guide the gaze.

[0005] For example, there are cases where the line of sight movement path estimated from an image showing the area ahead of the vehicle does not match the line of sight movement path required for driving the vehicle safely.

[0006] For example, in order for a vehicle driver to drive safely, it may be desirable for the driver to prioritize looking at an inconspicuous pedestrian over a building with an eye-catching color or shape.

[0007] The present invention has been made in view of the above, and aims to provide a visual line guidance device, a visual line guidance method, a visual line guidance program, and a storage medium that can guide the line of sight. [Means for solving the problem]

[0008] The gaze guidance device described in claim 1 is characterized by having: a calculation unit that calculates a first order in which a plurality of regions in an image determined based on the degree of gaze concentration is arranged in descending order of peak values ​​obtained by scanning a map in pixel units in which the degree of gaze concentration is allocated to the pixels of the image; a determination unit that determines whether a specified second order is the same as the first order; and a guidance unit that guides the gaze according to the second order when the determination unit determines that the first order and the second order are different.

[0009] The gaze guidance method described in claim 6 is a gaze guidance method executed by a computer, and is characterized by including: a calculation step of calculating a first order in which a plurality of regions in an image determined based on the degree of gaze attraction are arranged in descending order of peak values ​​obtained by scanning a map in pixel units in which the degree of gaze attraction is allocated to the pixels of the image; a determination step of determining whether a specified second order is the same as the first order; and a guidance step of guiding the gaze according to the second order when it is determined by the determination step that the first order and the second order are different.

[0010] The gaze guidance program described in claim 7 causes a computer to execute the following steps: a calculation step of calculating a first order in which a plurality of regions in an image determined based on the degree of gaze attraction is arranged in descending order of peak values ​​obtained by scanning a map in pixel units in which the degree of gaze attraction is allocated to the pixels of the image; a determination step of determining whether a specified second order is the same as the first order; and a guidance step of guiding the gaze according to the second order when it is determined by the determination step that the first order and the second order are different.

[0011] The storage medium described in claim 8 is characterized in that it stores a gaze guidance program for causing a computer to execute the following steps: a calculation step for calculating a first order in which multiple areas in an image determined based on the degree of gaze attraction are arranged in descending order of peak values ​​obtained by scanning a map in pixel units in which the degree of gaze attraction is allocated to the pixels of the image; a determination step for determining whether a specified second order is the same as the first order; and a guidance step for guiding the gaze in accordance with the second order when it is determined by the determination step that the first order and the second order are different. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an outline of the gaze guidance process. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of a visual guidance device. [Figure 3] FIG. 3 is a diagram illustrating visual saliency. [Figure 4] FIG. 4 is a diagram illustrating an example of calculation of the order of the lines of sight. [Figure 5] FIG. 5 is a diagram showing an example of a method for guiding the line of sight. [Figure 6] FIG. 6 is a diagram showing an example of a method for guiding the line of sight. [Figure 7] FIG. 7 is a diagram showing an example of a method for guiding the line of sight. [Figure 8] FIG. 8 is a flowchart showing the flow of processing by the visual guidance device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, a mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below. Furthermore, in the description of the drawings, the same parts are given the same reference numerals.

[0014] [First embodiment] The visual line guidance device according to the first embodiment calculates the order of movement of the visual line based on an image, and guides the movement order so that the order of movement matches a designated order.

[0015] 1 is a diagram showing an outline of the visual line guidance process. As shown in FIG.

[0016] The visual guidance device 10 may be an in-vehicle device such as a drive recorder or a car navigation system, or may be an information processing device such as a personal computer or a server device. Furthermore, the calculation unit 131 and the determination unit 132 may exist on a server, and the guidance unit 133 may exist on a terminal in the vehicle, and each block may be appropriately distributed, and is not necessarily limited to being integrated into one device.

[0017] As shown in FIG. 1, first, the visual guidance device 10 captures an image of the area ahead of the vehicle V (step S1).

[0018] Next, the visual guidance device 10 calculates the order of the gaze based on visual saliency from the image (step S2). The method of calculating the order of the gaze will be described later.

[0019] Then, if the order of the line of sight calculated in step S2 differs from the designated order, the line of sight guidance device 10 guides the line of sight in accordance with the designated order (step S3).

[0020] For example, the line-of-sight guidance device 10 can guide the line of sight by outputting an image or sound.

[0021] 2 is a diagram showing an example of the configuration of a visual line guidance device 10. As shown in FIG. 2, the visual line guidance device 10 includes an interface unit 11, a storage unit 12, and a control unit 13.

[0022] The interface unit 11 is an interface for inputting and outputting data, and may also be a communication module capable of data communication with other devices via a communication network such as the Internet.

[0023] 2, the interface unit 11 is connected to a camera 20 and an output device 30. The camera 20 is provided in a vehicle V and captures images of the surroundings or the interior of the vehicle V. The output device 30 is a display that displays images or a speaker that outputs sound.

[0024] The output device 30 may also be a see-through projection device equipped with a transparent screen and a projector. The transparent screen may be the front window of a vehicle. The output device 30 may also be AR (Augmented Reality) goggles that allow a real landscape and CG to be viewed simultaneously. The output device 30 may also be a device that can display an object superimposed on a landscape, such as a head-up display, an MR (Mixed Reality) device, or a mirror device with a see-through display.

[0025] The storage unit 12 stores various programs executed by the visual line guidance device 10, data required for executing processes, and the like.

[0026] The storage unit 12 stores model information 121. The model information 121 is parameters such as weights for constructing a neural network that calculates visual saliency.

[0027] The control unit 13 is realized by a controller such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in the memory unit 12, and controls the operation of the entire line-of-sight guidance device 10.

[0028] The control unit 13 is not limited to a CPU or an MPU, but may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0029] The control unit 13 includes a calculation unit 131 , a determination unit 132 , and a guidance unit 133 .

[0030] The calculation unit 131 calculates the order in which a plurality of areas in an image determined based on the degree of tendency to attract gazes are arranged in descending order of the peak value of the degree of tendency to attract gazes.

[0031] Here, the order calculated by calculation unit 131 is an example of the first order. Furthermore, the degree of likelihood of attracting gazes is, for example, visual saliency.

[0032] Visual saliency will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining visual saliency. As shown in Fig. 3, visual saliency is an index obtained by estimating the position of the driver's line of sight for an image showing the area ahead of the vehicle (see, for example, Patent Document 1).

[0033] Visual saliency may be calculated by inputting images into a neural network, for example, trained on a large number of images from a wide range of fields and the gaze information of multiple subjects who have actually viewed the images.

[0034] The visual saliency is, for example, an 8-bit (0 to 255) value given to each pixel of an image, and is expressed as a value that increases as the probability that the pixel is in the driver's line of sight increases.

[0035] Therefore, a saliency map can be obtained by mapping visual saliency to the corresponding pixels in an image, as shown in Figure 3. A saliency map can be thought of as data that maps the degree to which each pixel in an image attracts gaze.

[0036] Here, multiple regions (blobs) can be defined by dividing a saliency map, which is a plane corresponding to the image. In this case, each region has a point where the visual saliency reaches a peak value. The division method is not particularly limited, and may be based on an object region detected using, for example, an SSD (single shot multiple detector) or a clustering method such as k-means. Alternatively, the peak value point may be detected directly by scanning and comparing the saliency map pixel by pixel without dividing the region.

[0037] It is known that the gaze tends to move from areas with large peak values ​​to areas with small peak values ​​in units of calculated visual saliency (see, for example, Reference 1). Reference 1: Adachi et al., "Comparison of top-down and bottom-up factors in saliency maps using eye tracking," MIRU2017, PS2-19, 2017.

[0038] The peak value and the region can be defined by any method. For example, the gaze guidance device 10 identifies the point with the highest visual saliency in the entire saliency map. Then, the gaze guidance device 10 identifies a point with the highest visual saliency from an area excluding a predetermined area centered on the identified point. The gaze guidance device 10 repeats this process to consider the identified multiple points as points with peak values.

[0039] Fig. 4 is a diagram showing an example of calculation of the gaze order. As shown in Fig. 4, first, the calculation unit 131 calculates a saliency map 200a from an image 200. Here, a region 201, a region 202, a region 203, and a region 204 are defined in the saliency map 200a.

[0040] Area 201, area 202, area 203, and area 204 are areas of a predetermined size that include points where the visual saliency takes a peak value.

[0041] Here, if the regions are arranged in order of the magnitude of the peak value, they become region 201, region 202, region 203, and region 204. For example, region 201 is a region that includes a point that has the largest peak value in saliency map 200a.

[0042] At this time, the calculation unit 131 calculates the order of gaze in the image 200 as (A) → (B) → (C) → (D). (A), (B), (C), and (D) are points corresponding to the region 201, the region 202, the region 203, and the region 204, respectively, when the saliency map 200a is superimposed on the image 200.

[0043] The determination unit 132 determines whether the designated order is the same as the order calculated by the calculation unit 131 (hereinafter referred to as the calculation order). The designated order is an example of the second order. The designated order is the order of the driver's line of sight that further improves safety.

[0044] The designation order may be determined manually based on knowledge, or may be determined automatically by image recognition or the like.

[0045] Fig. 5 is a diagram showing an example of a method for guiding the line of sight. As shown in Fig. 5, it is assumed that the designation order for image 200 is (A) → (D) → (B) → (C). In this case, determination unit 132 determines that the calculation order and the designation order are different.

[0046] When the determination unit 132 determines that the calculation order and the designated order are different, the guidance unit 133 guides the line of sight in accordance with the designated order.

[0047] 5, it is considered that the calculation order can be made closer to the specified order by increasing the visual saliency of the area including (D). Therefore, the guiding unit 133 performs editing such as composition and transformation on (D) in the image 200.

[0048] In this way, the guiding unit 133 guides the gaze by outputting an image edited to increase the visual salience of the area (area including (D)) that has a higher ranking in the specified order than in the calculation order.

[0049] For example, the guiding unit 133 edits the image by changing pixel values ​​or by performing a synthesis process. Specifically, the guiding unit 133 performs editing by locally changing the luminance of the target area, changing the color temperature of the target area compared to the surrounding area, synthesizing another image with the target area, etc.

[0050] Furthermore, the guiding unit 133 causes the output device 30 to display an object for guiding the line of sight to an area whose order of designation is higher than its order of calculation, in a manner that allows the user to view the object simultaneously with the scenery.

[0051] In this case, the calculation unit 131 calculates the calculation order for the image of the scenery in the user's field of view. In addition, the user here is assumed to be the driver of the vehicle V.

[0052] In this case, the output device 30 is, for example, a display device having a transparent display. As shown in FIG.

[0053] This allows the user to simultaneously view the scenery 310 and the object 311 displayed by the guiding unit 133 via the output device 30.

[0054] Furthermore, the guiding unit 133 may display an image in which a landscape 310 is superimposed on an object 311. The object 311 is intended to raise the priority of the line of sight of a bicycle, which is a traffic participant that should be given priority attention.

[0055] As shown in Fig. 7, the output device 30 may be a mirror with an image display function. Fig. 7 is a diagram showing an example of a method for guiding the line of sight.

[0056] In the example of FIG. 7, the guiding unit 133 causes the output device 30 to display an object 321.

[0057] This allows the user to simultaneously view the scenery 320 and the object 321 displayed by the guiding unit 133 via the output device 30.

[0058] The object 321 is intended to raise the line of sight of the following vehicle, which is a traffic participant that should be given priority attention.

[0059] Fig. 8 is a flowchart showing the flow of processing by the visual guidance device. As shown in Fig. 5, first, the camera 20 provided on the vehicle V captures an image (step S101).

[0060] Next, the visual guidance device 10 calculates the order of the lines of sight based on visual saliency from the image (step S102).

[0061] Here, the visual guidance device 10 determines whether the calculated order matches the specified order (step S103).

[0062] If the calculated order does not match the specified order (No at step S103), the line-of-sight guidance device 10 guides the line of sight according to the specified order (step S104).

[0063] If the calculated order matches the specified order (step S103, Yes), the visual line guidance device 10 outputs the captured image as is (step S106). In this case, the visual line guidance device 10 does not need to output the image.

[0064] [Effects of the first embodiment] As explained above, the calculation unit 131 of the gaze guidance device 10 calculates a first order in which a plurality of regions in an image determined based on the degree of gaze concentration are arranged in descending order of the peak value of the degree of gaze concentration. The determination unit 132 determines whether the specified second order is the same as the first order. When the determination unit 132 determines that the first order and the second order are different, the guidance unit 133 guides the gaze according to the second order.

[0065] This makes it possible to guide the line of sight of the user, and also to guide the line of sight of a vehicle driver, for example, thereby improving safety.

[0066] The determination unit 132 determines whether the second order, which is specified so that a specific object has a higher rank, is the same as the first order. The guidance unit 133 guides the gaze by outputting an image edited so that an area having a higher rank in the second order than in the first order has greater visual salience.

[0067] This allows, for example, the driver's line of sight to be guided to a specific traffic participant, thereby improving safety.

[0068] The guidance unit 133 edits the image by changing pixel values ​​or by performing a synthesis process, thereby making it possible to easily guide the line of sight.

[0069] The calculation unit 131 calculates a first order for the image of the scenery in the user's field of view. The guidance unit 133 displays, on a predetermined display device, an object for guiding the line of sight to an area that has a higher order in the second order than the order in the first order, in a manner that allows the user to view the object simultaneously with the scenery.

[0070] This makes it possible, for example, to use a transparent display to simultaneously view a real landscape and a virtual object for visual guidance.

[0071] The determination unit 132 inputs the saliency map and the feature amounts output from the encoder into a CNN having an output layer corresponding to each class, and classifies the image into a class corresponding to the output obtained.

[0072] This reduces the influence of errors in saliency map estimation on class classification, thereby improving classification accuracy. [Explanation of symbols]

[0073] 10 Eye guidance device 11 Interface section 12 Storage section 13 Control Unit 20 Camera 30 Output Devices 121 Model Information 131 Calculation Department 132 Judgment section 133 Guidance part

Claims

1. a calculation unit that calculates a first order in which a plurality of regions in an image determined based on the degree of gaze concentration are arranged in descending order of peak values ​​obtained by scanning a map in which the degree of gaze concentration is allocated to pixels of the image, on a pixel-by-pixel basis; and a determination unit that determines whether a designated second order is the same as the first order; a guidance unit that guides the line of sight according to the second order when the determination unit determines that the first order and the second order are different; A visual guidance device comprising:

2. The visual guidance device according to claim 1 , wherein the determination unit determines whether a second order, in which a specific object is given a higher rank, is the same as the first order.

3. 3. The visual guidance device according to claim 1, wherein the guidance unit guides the gaze by outputting an image edited to increase visual salience of an area that has a higher rank in the second order than in the first order.

4. The visual line guidance device according to claim 3 , wherein the guidance unit edits the image by changing pixel values ​​or by performing a synthesis process.

5. the calculation unit calculates the first order for images of scenery in the user's field of view; 2. The visual guidance device according to claim 1, wherein the guidance unit displays, on a predetermined display device, an object for guiding the user's gaze to an area having a higher rank in the second order than in the first order, in a manner that allows the user to view the object simultaneously with the scenery.

6. A computer-implemented method for visual guidance, comprising: a calculation step of calculating a first order in which a plurality of regions in an image determined based on the degree of gaze attraction are arranged in descending order of peak values ​​obtained by scanning a map in which the degree of gaze attraction is allocated to the pixels of the image, on a pixel-by-pixel basis; a determining step of determining whether a specified second order is the same as the first order; a guiding step of guiding a line of sight according to the second order when it is determined that the first order and the second order are different by the determining step; A gaze guidance method comprising:

7. a calculation step of calculating a first order in which a plurality of regions in an image determined based on the degree of gaze attraction are arranged in descending order of peak values ​​obtained by scanning a map in which the degree of gaze attraction is allocated to the pixels of the image, on a pixel-by-pixel basis; a determining step of determining whether a specified second order is the same as the first order; a guiding step of guiding a line of sight according to the second order when it is determined that the first order and the second order are different by the determining step; A gaze guidance program that allows a computer to execute the above.

8. a calculation step of calculating a first order in which a plurality of regions in an image determined based on the degree of gaze attraction are arranged in descending order of peak values ​​obtained by scanning a map in which the degree of gaze attraction is allocated to the pixels of the image, on a pixel-by-pixel basis; a determining step of determining whether a specified second order is the same as the first order; a guiding step of guiding a line of sight according to the second order when it is determined that the first order and the second order are different by the determining step; A storage medium storing a gaze guidance program for causing a computer to execute the above.

Images