Driving support device and driving support method

Abstract

To execute appropriate driving support by displaying a vehicle image which is easy to be visually recognized by a driver in accordance with the preference of the driver.SOLUTION: A driving support method includes display processing (steps S518 and S522) for displaying a vehicle image, which is a virtual image, while superposing it on an actual scene so that it looks like, for a driver D1, a leading vehicle is disposed at a predetermined position ahead of a vehicle C1, and is configured to support driving of the vehicle C1 with respect to the driver D1. The driving support method includes: extraction processing (steps S516 and S520) for extracting preference information relating to the preference of the driver D1 based on at least one of appearance information relating to the vehicle C1, sight information relating to an actual scene that the driver D1 visually recognizes and speech information of the driver D1; and determination processing (steps S517 and S521) for determining appearance elements constituting the vehicle image on the basis of the preference information.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to a driving assistance method and a driving assistance device that assist a driver in driving a vehicle by superimposing a vehicle image, which is a virtual image, on an actual scene. [Background technology]

[0002] Conventionally, there is a technology that uses a vehicle image, which is a virtual image, to assist a driver in driving a vehicle. For example, a technology has been proposed in which a display position of a vehicle image (a virtual image displayed by superimposing it on an actual scene outside the vehicle) is calculated based on map data, driving data, and a viewpoint position, and the vehicle image is displayed at the calculated display position (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-184356 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, it is assumed that there are many other vehicles ahead of the vehicle at a certain point such as an intersection. In such a case, even if a vehicle image is displayed ahead of the vehicle, it is assumed that it is difficult to find the vehicle image ahead of the vehicle. In this case, the burden on the driver is large, and it may be difficult to execute appropriate driving assistance.

[0005] The present invention has an object to display a vehicle image that is easily visible to the driver in accordance with the driver's preferences, and to execute appropriate driving assistance. [Means for solving the problem]

[0006] One aspect of the present invention is a driving assistance method that includes a display process for displaying a vehicle image, which is a virtual image, superimposed on an actual scene so that the driver sees a leading vehicle positioned at a predetermined position in front of the vehicle, and that assists the driver in driving the vehicle. This driving assistance method includes an extraction process for extracting preference information related to the driver's preferences based on at least one of appearance information related to the vehicle, visual information related to the actual scene visually recognized by the driver, and speech information of the driver, and a determination process for determining appearance elements that constitute the vehicle image based on the preference information. Effect of the Invention

[0007] According to the present invention, a vehicle image that is easily visible to the driver can be displayed in accordance with the driver's preferences, and appropriate driving assistance can be performed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of display control on the front window. [Diagram 3] FIG. 3 is a diagram showing a display example in which a vehicle image is displayed on the front window. [Figure 4] FIG. 4 is a diagram showing the contents stored in the appearance element information DB in a simplified manner. [Diagram 5] FIG. 5 is a simplified diagram showing the contents stored in the preference information DB. [Figure 6] FIG. 6 is a diagram showing an example of customizing a base vehicle image. [Figure 7] FIG. 7 is a diagram showing a display example in which a vehicle image is displayed on the front window. [Figure 8] FIG. 8 is a top view showing a vehicle traveling on a road. [Figure 9] FIG. 9 is a top view showing a vehicle traveling on a three-lane road. [Figure 10] FIG. 10 shows an actual view seen through the front windshield and an image of the vehicle. [Figure 11] FIG. 11 is a flowchart showing an example of the vehicle image display process. [Figure 12] FIG. 12 is a flowchart showing an example of the vehicle image display process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] [Example of information processing system configuration] FIG. 1 is a diagram showing an example of a system configuration of an information processing system 100 installed in a vehicle C1.

[0011] The information processing system 100 includes an image acquisition unit 101, a sound acquisition unit 102, sensors 103, an information processing device 110, and a display device 200. The information processing device 110 includes a display control unit 126 that displays a vehicle image, which is a virtual image, superimposed on a real scene so that the driver D1 sees a leading vehicle positioned at a predetermined position in front of the vehicle C1, and is an example of a driving support device that supports the driver D1 in driving the vehicle C1. The vehicle image is an example of driving support information that presents information that affects the driving of the vehicle C1 to the driver D1 in advance. The driving support information is, for example, route guidance information, information about a moving object that is difficult to see, traffic environment information with poor visibility, etc.

[0012] The information processing device 110 and the display device 200 are connected by a communication method using wired communication or wireless communication. The information processing device 110 is connected to the network 20 by a communication method using wireless communication. The network 20 is a network such as a public line network or the Internet. The display device 200 may also be connected to the network 20 by a communication method using wireless communication. Although FIG. 1 shows an example in which the information processing device 110 and the display device 200 are configured as separate entities, the information processing device 110 and the display device 200 may also be configured as an integrated device.

[0013] The image acquisition unit 101 captures an image of a subject to generate an image (image data), and outputs image information related to the generated image to the information processing device 110. The image acquisition unit 101 is configured, for example, with an imaging element (image sensor) to which light from the subject collected by a lens is incident, and an image processing unit that performs predetermined image processing on the image data generated by the imaging element. For example, a CCD (Charge Coupled Device) type or CMOS (Complementary Metal Oxide Semiconductor) type imaging element can be used as the imaging element.

[0014] The image acquisition unit 101 is provided at least inside the vehicle C1, and captures an image of a subject inside the vehicle C1 to generate an image (image data). The image acquisition unit 101 is, for example, configured with one or more camera devices or image sensors capable of capturing an image of a subject. In addition, in Figs. 2 and 3, an example is shown in which the image acquisition unit 101 is installed on the dashboard 2 inside the vehicle C1. In addition, in Figs. 2 and 3, only one image acquisition unit 101 is illustrated, but the installation location of the image acquisition unit 101 is not limited to the example shown in Figs. 2 and 3, and can be changed as appropriate. For example, the image acquisition unit 101 may be provided at another position inside the vehicle C1 (for example, above the front window 4). In addition, for example, two or more image acquisition units 101 may be provided, and all or some of the images of these image acquisition units 101 may be used. Also, for example, one image acquisition unit 101 may be provided in front of the vehicle C1 to capture an image of a subject from in front of the vehicle C1 to generate an image (image data), and another image acquisition unit 101 may be provided behind the vehicle C1 to capture an image of a subject behind the vehicle C1 to generate an image (image data). Also, one or more devices capable of capturing subjects present in all directions of the vehicle C1 and subjects inside the vehicle C1, for example, a 360-degree camera, may be used.

[0015] The sound acquisition unit 102 is provided inside the vehicle C1, acquires sounds inside the vehicle C1, and outputs sound information related to the acquired sounds to the information processing device 110. As the sound acquisition unit 102, for example, one or more microphones or sound acquisition sensors can be used.

[0016] The sensors 103 are various sensors installed in the vehicle C1, and output detection information acquired by each sensor to the information processing device 110. The sensors are, for example, LiDAR (Light Detection And Ranging), RADAR (Radio Detection And Ranging), Sonar, a vehicle speed sensor, an acceleration sensor, a steering sensor (steering force angle meter), an accelerator position sensor, a position information acquisition sensor (position information acquisition unit), and the like. For each of these sensors, a known sensor can be used. Note that LiDAR, RADAR, Sonar, and the like are examples of sensors that detect the situation around the vehicle C1. Also, the vehicle speed sensor, the acceleration sensor, the steering sensor, the accelerator position sensor, and the like are examples of sensors that detect the situation of the driving operation of the driver D1. Note that these are only examples, and other sensors may be used. Also, only some of these sensors may be used.

[0017] The position information acquisition unit acquires position information on the position where the vehicle C1 is located, and outputs the acquired position information to the information processing device 110. For example, it can be realized by a GNSS receiver that acquires position information using a Global Navigation Satellite System (GNSS). The position information includes data on the position such as latitude, longitude, and altitude at the time of receiving the GNSS signal. The position information may be acquired by other methods of acquiring position information. For example, the position information may be derived using information from an access point or base station present in the vicinity. The position information may be acquired using a beacon. The absolute position of the vehicle C1 (i.e., position, attitude, speed, etc. relative to a certain reference point) may be measured and acquired using odometry. In this way, the position information acquisition unit is composed of a sensor that measures the absolute position of the vehicle C1.

[0018] The information processing device 110 includes a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices based on the control of the control unit 120 by using wired communication or wireless communication.

[0019] The control unit 120 controls each unit based on various programs stored in the storage unit 130. The control unit 120 is realized by a processing device such as a central processing unit (CPU) or a graphics processing unit (GPU). Note that the vehicle ECU (electronic control unit) of the vehicle C1 may be used as the control unit 120, or a processing device different from the vehicle ECU may be provided as the control unit 120.

[0020] The control unit 120 executes various controls based on the information output from the image acquisition unit 101, the sound acquisition unit 102, the sensors 103, the communication unit 140, etc. For example, the control unit 120 executes a control process for controlling the operation state of the display device 200. Specifically, the control unit 120 includes a route setting unit 121, a moving object detection unit 122, a determination unit 123, an extraction unit 124, a determination unit 125, and a display control unit 126.

[0021] When a destination of the vehicle C1 is specified based on a user operation by a user who gets in the vehicle C1, the route setting unit 121 searches for and sets a travel route from the current location of the vehicle C1 to the destination using the map information DB 170 in the storage unit 130. Then, the route setting unit 121 outputs the destination and the travel route to the determination unit 123, the extraction unit 124, etc. It is possible to use a known setting method for setting the travel route. Also, when a navigation device or the like is installed in the vehicle C1, the travel route to the destination set by the navigation device may be acquired and used.

[0022] The moving object detection unit 122 detects an object existing around the vehicle C1 based on each piece of information output from the image acquisition unit 101, the sound acquisition unit 102, the sensors 103, the communication unit 140, etc. Then, the moving object detection unit 122 outputs object detection information related to the detection result to the determination unit 123, the extraction unit 124, the display control unit 126, etc. The method of detecting a moving object will be described in detail with reference to FIG. 11 etc.

[0023] The determination unit 123 determines whether or not there is a vacant position for displaying a vehicle image in the road space in front of the vehicle C1 based on each piece of information output from the image acquisition unit 101, the sound acquisition unit 102, the sensors 103, the communication unit 140, etc., and the object detection information output from the moving body detection unit 122. Then, the determination unit 123 outputs vacant position determination information related to the determination result to the extraction unit 124, the decision unit 125, the display control unit 126, etc. Note that the vacant position is, for example, a position where there is no object (especially a moving body) detected by the moving body detection unit 122 and where a predetermined area for displaying a vehicle image can be secured. Note that this determination method will be described in detail with reference to FIG. 10, FIG. 12, etc.

[0024] The extraction unit 124 extracts preference information related to the preferences of the driver D1 based on each piece of information output from the image acquisition unit 101, the sound acquisition unit 102, the sensors 103, the communication unit 140, etc. Then, the extraction unit 124 stores the preference information related to the extraction result in the preference information DB 160 (see FIG. 5). Specifically, the extraction unit 124 extracts the preference information of the driver D1 based on at least one of the appearance information related to the vehicle C1, the visual information related to the actual scene visually recognized by the driver D1, and the speech information of the driver D1. Then, as necessary, the extraction unit 124 outputs the preference information stored in the preference information DB 160 to the determination unit 125, etc. In this way, the extraction unit 124 functions as a preference extraction unit of the driver D1 that extracts design elements of the exterior that match the preferences of the driver D1. The method of extracting the preference information will be described in detail with reference to FIG. 5, FIG. 6, etc.

[0025] The determination unit 125 determines the appearance elements constituting the vehicle image to be displayed on the display device 200 based on the preference information output from the extraction unit 124, and generates a vehicle image that will be the appearance elements. The determination unit 125 then outputs image information related to the generated vehicle image to the display control unit 126. In this manner, the determination unit 125 can customize the vehicle image based on the appearance elements that match the preferences of the driver D1. The method of determining the appearance elements and the method of generating the vehicle image will be described in detail with reference to Figs. 4 to 6, Fig. 12, etc.

[0026] The display control unit 126 controls the display state of the display device 200 based on each piece of information output from the image acquisition unit 101, the sound acquisition unit 102, the sensors 103, the communication unit 140, etc. Specifically, the display control unit 126 executes a display process for displaying a vehicle image, which is a virtual image, superimposed on a real scene so that the driver D1 can see that a leading vehicle is placed at a predetermined position in front of the vehicle C1. In this case, the display control unit 126 can display the vehicle image on the condition that the vehicle C1 approaches a position where guidance is required to support the driving of the vehicle C1. Note that examples of the display of the vehicle image are shown in Figs. 3, 7, and 10. In addition, a known display method can be used as a display method for displaying the vehicle image at a predetermined position in front of the vehicle C1. In this way, the display control unit 126 functions as a virtual vehicle presentation unit.

[0027] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (for example, a control program, an appearance element information DB 150, a preference information DB 160, and a map information DB 170) that are necessary for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 140. As the storage unit 130, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), an SRAM (Static Random Access Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a combination of these can be used.

[0028] Various types of information for managing vehicle information related to a plurality of vehicles including the vehicle C1 are stored in the appearance element information DB 150. The various types of information stored in the appearance element information DB 150 will be described in detail with reference to FIG.

[0029] Various types of information for managing preference information related to the preferences of the driver D1 who drives the vehicle C1 are stored in the preference information DB 160. The various types of information stored in the preference information DB 160 will be described in detail with reference to FIG.

[0030] The display device 200 displays a HUD image in a display area R (see FIG. 2) of the windshield 4 based on an instruction from the information processing device 110.

[0031] [Example of display system configuration] Fig. 2 is a simplified diagram showing a configuration example for realizing display control on the windshield 4. Fig. 2 is a simplified diagram showing a cross section of the inside of the vehicle C1 as viewed from the left side in the left-right direction, and components other than the dashboard 2, the steering wheel 3, the windshield 4, the image acquisition unit 101, and the display device 200 are omitted from the illustration.

[0032] An image acquisition unit 101 that acquires an image including the face of the driver D1 is provided on the upper surface of the dashboard 2. In addition, on the upper surface of the dashboard 2, near the boundary with the windshield 4, a display device 200 for realizing a HUD (Head Up Display) of the vehicle C1 is provided.

[0033] The windshield 4 is a shield (for example, a windshield) provided at the front of the vehicle C1, and functions as a display medium for the HUD of the vehicle C1.

[0034] The display area R is a HUD display area that displays an AR (Augmented Reality)-HUD image. The AR-HUD image refers to an image (virtual image) that is displayed as a virtual image on the windshield 4 using AR technology. As the AR-HUD image, various information related to the road and the vehicle is displayed. FIG. 3 shows an example of the display of this virtual image (AR-HUD image). In this embodiment, an example is shown in which a vehicle image 300 (see FIG. 3) is displayed as the virtual image in the display area R to guide the vehicle C1 in a predetermined direction at a predetermined position (for example, near a place where a lane change is required, near an intersection).

[0035] As shown in FIG. 2, the display device 200 is a display device, for example, a projector or an optical system, for realizing an AR-HUD display in which light P11 is projected onto a display area R of the windshield 4 and the reflected light P12 is used to show a virtual image to the driver D1. That is, the light P11 projected from the display device 200 onto the display area R of the windshield 4 is reflected by the windshield 4, and the reflected light P12 is directed toward the eyes of the driver D1. The reflected light P12 projected onto the display area R and entered the eyes of the driver D1 is perceived as an image (virtual image) that is superimposed on an actual object seen through the windshield 4 and seen on the opposite side of the windshield 4 (the outside of the vehicle). In this way, the display device 200 realizes a HUD display by displaying a virtual image using the windshield 4.

[0036] In this way, various information can be displayed in the display area R of the windshield 4 by using the display device 200 fixedly installed on the dashboard 2. That is, the windshield 4 and the display device 200 function as a display image projection means.

[0037] [Example of vehicle image display] Fig. 3 is a diagram showing a display example in which a vehicle image 300 is displayed on the front window 4 of a vehicle C1. Fig. 3 shows a simplified view of the interior of the vehicle C1, and does not show anything other than the dashboard 2, steering wheel 3, front window 4, rearview mirror 5, and image acquisition unit 101.

[0038] Here, the vehicle image 300 is a virtual image of a leading vehicle that is rendered as if it were traveling on a road ahead of the vehicle C1 at a predetermined travel trajectory and speed. This virtual image is a perspective projection image (or a deformed version of the perspective projection image) that fits the coordinate system of the actual scene, and is, for example, a semi-transparent or opaque image rendered from a simple polygon model defined based on three-dimensional CAD (Computer-Aided Design) data of the vehicle C1.

[0039] As shown in FIG. 3, a driver D1 driving a vehicle C1 sees a virtual image of a vehicle image 300 being displayed on the windshield 4 along with a real view of a road R1, so that the driver D1 sees the vehicle image 300 as if it were traveling along the road R1.

[0040] Here, the vehicle image 300 is virtually displayed so that the driver D1 of the vehicle C1 can see it, and it can be displayed without being affected by objects outside the vehicle C1. However, it is assumed that many other vehicles are present in front of the vehicle C1 at a certain point such as an intersection. In this way, when many other vehicles are present in the forward direction of the vehicle C1, the other vehicles and the vehicle image 300 may be close to each other or overlap with each other, which may make it difficult to visually recognize the vehicle image 300. In such a case, it is also assumed that it may be difficult to find the vehicle image 300 in the road space in front of the vehicle C1. In this case, the burden on the driver D1 may be large, and it may be difficult to perform appropriate driving assistance.

[0041] In this embodiment, the vehicle image 300 is customized according to the preferences of the driver D1 and displayed to appropriately assist the driver D1 in driving. That is, in this embodiment, a vehicle image that is easily visible to the driver D1 is displayed according to the preferences of the driver D1, and appropriate driving assistance is performed.

[0042] [Example of appearance element information DB configuration] FIG. 4 is a diagram showing in simplified form the contents stored in the appearance element information DB 150 stored in the storage unit 130. As shown in FIG.

[0043] The appearance element information DB150 is a database for managing various information related to a plurality of vehicles. Specifically, vehicle identification information 152, keyword information 153, and appearance element information 154 are stored in the appearance element information DB150 in association with vehicle type information 151.

[0044] Vehicle type information 151 is identification information for identifying each vehicle. For example, "own vehicle" in vehicle type information 151 corresponds to vehicle C1 (see FIG. 2). Also, "vehicle A", "vehicle B" and the like in vehicle type information 151 correspond to vehicles other than vehicle C1. Note that the vehicle types shown in this embodiment may be types of vehicles that are actually sold, or may be types of virtual vehicles. In the case of virtual vehicle types, it is preferable to set vehicles for each of a plurality of categories. For example, it is preferable to set vehicle types that can be adapted to the tastes of a plurality of people, such as sporty cars and solid cars.

[0045] The vehicle identification information 152 is information for identifying each vehicle whose vehicle type information is stored in the vehicle type information 151. In this embodiment, the vehicle identification information 152 is used to identify a vehicle that matches the preferences of the driver D1. For example, it is possible to identify each vehicle traveling in front of the vehicle C1 based on an image in front of the vehicle C1 acquired by the image acquisition unit 101. In this case, image identification information is stored in the vehicle identification information 152. This image identification information is identification information for identifying the vehicle based on an image of another vehicle included in the image in front of the vehicle C1 acquired by the image acquisition unit 101. It is possible to use a known image recognition technology as a method for identifying the vehicle included in the image. For example, it is possible to detect an object present in the line of sight of the driver D1 based on an image of the driver D1 acquired by the image acquisition unit 101 installed inside the vehicle C1 and an image in front of the vehicle C1 acquired by the image acquisition unit 101 installed outside the vehicle C1. Then, it is possible to determine whether or not the object present in the line of sight of the driver D1 is a vehicle whose vehicle type information is stored in the vehicle type information 151 based on the vehicle identification information 152. In addition, in order to facilitate the explanation, FIG. 4 shows an example in which image information showing the exterior appearance of the vehicle is stored in the vehicle identification information 152.

[0046] The keyword information 153 is information indicating keywords related to the characteristics of each vehicle. In this embodiment, the keyword information 153 is used to identify a vehicle that matches the preferences of the driver D1. For example, based on the voice of the driver D1 acquired by the sound acquisition unit 102, it is possible to identify a predetermined keyword contained in the voice. For example, it is possible to extract characteristic keywords from each word contained in the voice of the driver D1. It is possible to use a known voice recognition technology as a method for extracting characteristic keywords. Then, it is possible to identify a vehicle that matches the preferences of the driver D1 based on the degree of agreement between the keyword extracted from the voice of the driver D1 and each keyword stored in the keyword information 153. When the keyword "open" is extracted from the voice of the driver D1, it matches the keyword "open, open" stored in the keyword information 153, so it is possible to identify "car B" as a vehicle that matches the preferences of the driver D1. It is to be noted that in FIG. 4, an example in which simplified keywords are stored in the keyword information 153 is shown for ease of explanation.

[0047] The appearance element information 154 is information about each element constituting the appearance of each vehicle whose vehicle model information is stored in the vehicle model information 151. This information can be grasped as, for example, parameters of design elements constituting the vehicle's appearance. Elements constituting the vehicle's appearance (appearance design elements) include, for example, each pillar, grille, rear window, vehicle height, vehicle width, vehicle color, etc. In addition, the height, length, inclination, angle, shape, etc. of each element can also be an appearance design element. Each of these pieces of information is used as a parameter when generating a vehicle image according to the preferences of the driver D1. An example of this generation will be described in detail with reference to FIG. 6. The appearance element information 154 may store the specification values ​​of each vehicle. The specification values ​​refer to various parameters related to the vehicle, such as the basic structure of the vehicle (for example, overall length, vehicle width, vehicle height), performance, etc., and are also referred to as vehicle specifications.

[0048] [Example of preference information DB configuration] FIG. 5 is a simplified diagram showing the contents stored in the preference information DB 160 stored in the storage unit 130. As shown in FIG.

[0049] The preference information DB 160 is a database for managing preference information related to the preferences of the driver D1. For example, the preference information DB 160 stores first preference information 161 based on visual information related to the actual scene visually recognized by the driver D1 and second preference information 165 based on speech information of the driver D1.

[0050] The first preference information 161 is preference information related to a vehicle gazed upon by a driver D1 in the vehicle C1. For example, vehicle information 162 related to the type of vehicle gazed upon by the driver D1 in the vehicle C1 and gaze time information 163 related to the time the driver D1 gazed upon the vehicle are stored in the first preference information 161 in association with each other.

[0051] The second preference information 165 is preference information related to the speech content of the driver D1 who is in the vehicle C1. For example, keyword information 166 related to a keyword included in the speech content of the driver D1 who is in the vehicle C1 and score information 167 related to the number of times the keyword is uttered are stored in the second preference information 165 in association with each other.

[0052] [Example of driver preference extraction] Here, an example of extracting the preferences of the driver D1 will be described. Information related to the preferences of the driver D1 will be referred to as preference information. Each extraction process shown below is executed by the extraction unit 124.

[0053] For example, it is possible to extract preference information of the driver D1 based on the appearance elements of the vehicle C1 in which the driver D1 is riding. For example, it is possible to initially set each appearance element constituting the vehicle image based on the specification values ​​of the vehicle C1. For example, it is possible to display the initially set vehicle image 300 based on the appearance element information 154 corresponding to the vehicle type information 151 "own vehicle" in the appearance element information DB 150 (see FIG. 4).

[0054] Also, for example, it is possible to detect the line of sight of the driver D1 and extract the exterior elements of the vehicle that the driver D1 is gazing at among the vehicles existing around the vehicle C1. For example, based on an image acquired by the image acquisition unit 101 (in-vehicle camera) installed inside the vehicle C1, the line of sight direction of the driver D1 can be detected based on the position of the face, the position of the eyes, and the position of the black part of the eyes of the driver D1 included in the image. Also, based on an image in front of the vehicle C1 acquired by the image acquisition unit 101 (outside vehicle camera) installed outside the vehicle C1, the orientation and position (direction and position based on the vehicle C1) of another vehicle included in the image can be detected. Therefore, when the line of sight direction of the driver D1 and the direction of another vehicle in front of the vehicle C1 match for a predetermined time or more, it is possible to determine that the driver D1 is gazing at that vehicle. In this case, it is possible to determine whether or not the vehicle determined to be gazed at by the driver D1 is a vehicle whose vehicle model information is stored in the vehicle model information 151 based on the vehicle identification information 152. Then, if the vehicle determined to be gazed upon by the driver D1 is a vehicle whose vehicle model information is stored in the vehicle model information 151, the time during which the driver D1 gazed upon the vehicle is measured, the vehicle model of the vehicle is stored in the vehicle information 162, and the measured time is stored in the gaze time information 163. The measured time stored in the gaze time information 163 may be stored each time the driver D1 gazes upon the vehicle, and the total value of the measured time for a plurality of gazes may be stored for the same vehicle model. On the other hand, if the vehicle determined to be gazed upon by the driver D1 is not a vehicle whose vehicle model information is stored in the vehicle model information 151, the vehicle most similar to the vehicle is determined based on the vehicle identification information 152. In this case, the similarity of the images can be determined using a known image recognition process. Then, the time during which the driver D1 gazed upon the vehicle most similar to the vehicle is measured, the vehicle model of the similar vehicle is stored in the vehicle information 162, and the measured time is stored in the gaze time information 163.

[0055] Also, for example, it is possible to detect the voice of the driver D1 and extract the preferences of the driver D1 based on the voice. For example, it is possible to detect the voice of the driver D1 based on the sound inside the vehicle C1 acquired by the sound acquisition unit 102 installed inside the vehicle C1. Note that, inside the vehicle C1, the voice of the driver D1 can be detected by identifying the direction of the person who has made the voice using a known voice recognition technology (for example, a speaker identification technology). For example, it is possible to detect the voices of the people sitting in the driver's seat, passenger seat, back seat, etc. inside the vehicle C1 by identifying each person.

[0056] For example, assume that the driver D1 and one or more passengers are driving in the vehicle C1. In this case, it is assumed that the driver D1 and the passengers will have a conversation during the drive, and it is possible to identify the contents of the driver D1's utterance in the conversation. In addition, for example, if an agent device or the like that supports the driving of the vehicle C1 is installed in the vehicle C1, it is assumed that the driver D1 will speak in an exchange with the agent device. In this case, it is also possible to identify the contents of the driver D1's utterance in the exchange.

[0057] Then, based on the contents of the driver D1's speech, it is possible to extract the preferences of the driver D1. In this case, for example, the values, topics, etc. of the driver D1 can be extracted as the preferences of the driver D1. For example, it is possible to convert the contents of the driver D1's speech into text information (character information) using a known voice recognition technology, and extract the preferences of the driver D1 based on the text information. For example, the preferences of the driver D1 can be estimated by applying natural language processing (for example, topic analysis method, sentiment analysis method, positive / negative analysis method) to the text information. Note that the topic analysis method is an analysis method that estimates the topic (i.e., rough meaning) contained in the text information. For example, topics, fields, etc. are analyzed as the topics. Also, the sentiment analysis method is a method of analyzing the emotions felt by the person who made the speech based on the text information. Also, the positive / negative analysis method is a method of analyzing whether the contents contained in the text information are positive, negative, or neutral. For example, sentences containing keywords such as fun or happy are classified as positive, sentences containing keywords such as dislike or don't want to see are classified as negative, and sentences that are difficult to judge as either are classified as neutral.

[0058] For example, by applying topic analysis, it is possible to determine what field the driver D1 is talking about. Also, by applying positive-negative analysis, it is possible to determine how positive he is towards what topic. In this way, it is possible to determine what topic the driver D1 is positive towards using text information. Also, it is possible to calculate a score indicating the degree of positivity towards a certain topic. Then, keyword information corresponding to the positive topic is stored in keyword information 166, and a score indicating the degree of positivity towards that keyword is stored in score information 167. Note that instead of the score, other information indicating the level of the driver D1's preference, such as information on the number of times the keyword was uttered, may be stored in score information 167.

[0059] For example, if the positive topic is sports, "sporty" is stored in the keyword information 166, and if the positive topic is the latest technology, "advanced" is stored in the keyword information 166. Also, for example, if the positive topic is aggressive, "proactive" is stored in the keyword information 166, and if the positive topic is family, "domestic" is stored in the keyword information 166. In this way, emotional words can be stored and used as keyword information.

[0060] The information stored in the preference information DB160 can be constantly updated and used every time the same driver gets into the vehicle C1. However, it is also assumed that a different driver gets into the vehicle C1. Therefore, the information in the preference information DB160 may be deleted when the driver D1 gets off the vehicle C1, and the contents of the preference information DB160 may be updated based on new information acquired from the time the driver D1 gets into the vehicle C1 next time. This makes it possible to display a vehicle image that matches the preferences of the driver D1 based on the latest preferences of the driver D1.

[0061] In this embodiment, the appearance element information DB150 and the preference information DB160 are stored in the vehicle C1, but at least one of the appearance element information DB150 and the preference information DB160 may be stored in an external device and used. For example, the appearance element information DB150 may be managed in an external device (e.g., a server), and necessary information may be obtained from the external device via the network 20 and used.

[0062] [Example of vehicle image customization] Fig. 6 is a diagram showing an example of customization when customizing a base vehicle image 300. Fig. 6 shows an example in which a vehicle image 300 having substantially the same appearance configuration as the appearance of a vehicle C1 is set as the initial setting. Note that the vehicle image 300 can be generated based on the appearance element information 154 of the appearance element information DB 150 (see Fig. 4).

[0063] Here, it is assumed that many other vehicles are present ahead of the vehicle at a given point such as an intersection. In such a case, even if the vehicle image 300 is displayed ahead of the vehicle, it is assumed that it is difficult to find the vehicle image ahead of the vehicle. In this case, the burden on the driver is increased, and it may be difficult to perform appropriate driving assistance.

[0064] Therefore, in this embodiment, a vehicle image that is easily visible to the driver D1 is displayed according to the preferences of the driver D1. Specifically, the vehicle image 300 is customized based on the preference information of the driver D1 extracted by the above-mentioned extraction process. This customization process is executed by the determination unit 125.

[0065] First, a case where the vehicle image 300 is customized based on the first preference information 161 of the preference information DB 160 (see FIG. 5) will be described. It is possible to customize the vehicle image 300 using the appearance element information 154 (see FIG. 4) of the vehicle whose model is stored in the vehicle information 162 and whose value of the gaze time information 163 is the largest. For example, it is possible to generate a vehicle image composed of each element stored in the appearance element information 154 of the vehicle. Note that the vehicle image 300 may be customized using the appearance element information 154 of a plurality of vehicles whose models are stored in the vehicle information 162. For example, it is also possible to generate a vehicle image by synthesizing each element using the appearance element information 154 of a predetermined number of vehicles whose values ​​of the gaze time information 163 are the largest. In this case, it is possible to set the ratio of each element to be synthesized according to the value of the gaze time information 163.

[0066] Next, a case where the vehicle image 300 is customized based on the second preference information 165 of the preference information DB 160 (see FIG. 5) will be described. It is possible to customize the vehicle image 300 using the keyword information 166 with the maximum value of the score information 167. For example, it is possible to generate a vehicle image composed of each element stored in the appearance element information 154 corresponding to the keyword information 153 (see FIG. 4) that is the same as or similar to the keyword information. For example, the keyword information 166 with the maximum value of the score information 167 can be recognized as the most positive topic for the driver D1. For example, when the positive topic is "sporty", the vehicle type information 151 "Car C" corresponding to the keyword information 153 "sport" similar to "sporty" is selected. In this way, when the most positive topic for the driver D1 is "sporty", it is possible to extract the parameters of the sporty exterior design element and display the vehicle image 310 of a sports car or the like. Also, if the most positive topic for driver D1 is the latest technology, it is possible to extract parameters of cutting-edge exterior design elements and display a vehicle image of a cutting-edge vehicle.

[0067] In addition, the vehicle image 300 may be customized using a plurality of pieces of keyword information 166. For example, it is possible to use the appearance element information 154 corresponding to the keyword information 153 that is the same as or similar to a predetermined number of pieces of keyword information having the highest values ​​of the score information 167.

[0068] In addition, the vehicle image may be generated using all of the elements stored in the vehicle appearance element information 154 that matches the preferences of the driver D1, or the vehicle image may be generated using only some of the elements. For example, it is possible to change the rear design of the vehicle image by applying parameters of only some of the elements. For example, it is possible to change the rear design of the vehicle, such as the vehicle height, vehicle width, the arrangement of lamps at the rear of the vehicle, and the shape of the rear window of the vehicle. In addition, it is possible to change the rear design of the vehicle, for example, by changing the inclination and length of each pillar, the height and width of the rear bumper, or the shape and width of the tires. In addition, it is possible to change the rear design of the vehicle, for example, by changing the outline, shape, size, etc. of the vehicle's exterior to a station wagon, sedan, coupe, etc. In addition, it is possible to change the rear design of the vehicle, for example, by changing the color of the vehicle's exterior.

[0069] [Example of vehicle image customization using AI] The vehicle image can be customized using artificial intelligence (AI). For example, preference information regarding the preferences of the driver who gets into the vehicle C1 and the vehicle image that the driver finds easy to see can be learned in advance, and the above-mentioned customization process (decision process) can be executed using this learning data. A specific example of customizing a vehicle image using AI will be described below.

[0070] [Examples of using image recognition technology such as semantic segmentation] For example, by using an image recognition technique such as semantic segmentation, it is possible to extract by image processing each area of ​​another vehicle in front of the vehicle C1 acquired by an image acquisition unit 101 (exterior vehicle camera) installed in front of the vehicle C1. In this image processing, it is possible to extract what shape of C-pillar is installed on which side of the vehicle in front, what shape of rear bumper is installed on which side, etc. By applying various image processing such as morphing so that the angle of the C-pillar thus extracted is changed to an angle preferred by the driver D1, or morphing so that the height of the rear bumper is changed to a height preferred by the driver D1, it is possible to customize the vehicle image.

[0071] [Examples of using AI models such as conditional GAN] For example, it is possible to deform a vehicle image using an AI model (e.g., conditional GAN ​​(Generative Adversarial Networks) or the like) that has learned the correspondence between changes in the appearance elements of the vehicle image and changes in the vehicle image. When deforming a vehicle image using this AI model, it is possible to hold the correspondence between changes in the appearance elements of the vehicle image and changes in the vehicle image as a data set in advance, and to output a vehicle image corresponding to the parameters of the appearance elements preferred by the driver by learning the correspondence using a conditional GAN ​​or the like. For example, it is possible to hold a large number of vehicle images corresponding to each parameter when the inclination of the C-pillar is small, medium, and large as a data set, and to output a vehicle image corresponding to the inclination of the C-pillar preferred by the driver by performing machine learning on the data set in advance.

[0072] [Example of using image generation methods such as diffusion models] For example, it is possible to customize a vehicle image using a generative AI by using words of positive topics that express the preferences of the driver D1. In this case, by learning a large amount of correspondence between words and images in advance using an image generation method such as a diffusion model, it is possible to output a vehicle image that is arranged using words for a base vehicle image (original image). For example, if the positive topic is sports, it is possible to change the base vehicle image to a sporty vehicle image by arranging it with the word sporty.

[0073] [Example of using generation AI such as CLIP] For example, it is possible to customize the vehicle image using a generation AI such as CLIP (Contrastive Language-Image Pre-training) by using words such as values ​​and topics that express the preferences of the driver D1. CLIP is a model that combines natural language processing and image processing.

[0074] In this way, it is possible to learn vehicle images, keywords, changes in the appearance elements of the vehicle images, etc. in advance, and use this learning data in the customization process of the vehicle images. For example, it is possible to generate learning data by defining and learning each vehicle image stepwise, including sporty vehicle images and non-sporty vehicle images. That is, it is possible to generate learning data by learning changes in the vehicle images based on keywords such as sportiness. In addition, it is possible to give a score indicating the reliability of the changes in the vehicle images by learning changes in the vehicle images based on keywords such as sportiness. It is possible to generate vehicle images using this score. An example of using this score is shown below.

[0075] [Example of using two types of AI models] For example, it is possible to customize vehicle images using multiple types of AI models. Here, an example is shown in which two types of generating AIs capable of outputting predetermined information using images and text information are used. For example, an example is shown in which a first generating AI (e.g., Stable Diffusion) and a second generating AI (e.g., Deep CLIP) are used.

[0076] For example, when a base vehicle image (e.g., the vehicle itself) and text information that matches the preferences of driver D1 are input to the first generation AI, vehicle images that exhibit a text-like appearance corresponding to the text information are output as multiple candidates.

[0077] When the multiple candidates (vehicle images) output in this way and text information that matches the preferences of the driver D1 are input to the second generation AI, a score of the vehicle image indicating the text-likeness corresponding to the text information is output for each vehicle image. This score is a value indicating the reliability of the vehicle image output by the first generation AI.

[0078] In this way, based on the output results output from the first generation AI and the second generation AI, it is possible to filter the multiple candidates (vehicle images) output from the first generation AI to determine the customized vehicle image. For example, from the multiple candidates (vehicle images) output from the first generation AI, the candidate (vehicle image) with the maximum score output from the second generation AI is selected, and this candidate (vehicle image) is determined as the customized vehicle image.

[0079] In this way, by using the generation AI, it is possible to appropriately modify the vehicle image according to the preferences of the driver D1, even if each part of the vehicle image is not set in detail. In other words, when customizing the vehicle image according to the preferences of the driver D1, it is possible to customize the vehicle image by applying image processing and machine learning processing based on the exterior design elements that match the preferences of the driver D1.

[0080] The left side of Fig. 6 shows a vehicle image 310 generated based on "sporty" stored in the keyword information 166 (see Fig. 5). The center of Fig. 6 shows a vehicle image 320 generated based on "open feeling" stored in the keyword information 166. The right side of Fig. 6 shows a vehicle image 330 generated based on "homely" stored in the keyword information 166.

[0081] [Example of vehicle image display customized based on driver preference information] Fig. 7 is a diagram showing a display example in which a vehicle image 310 is displayed on the front window 4 of a vehicle C1. Fig. 7 shows a display example in which a vehicle image 310 is displayed instead of the vehicle image 300 shown in Fig. 3, and is the same as Fig. 3 except that the vehicle image 310 is displayed.

[0082] 7, by displaying a vehicle image 310 that is easy for the driver D1 to view in front of the vehicle C1 according to the driver D1's preference, the driver D1 can easily find the vehicle image 310 in the road space ahead. In this case, the driving burden on the driver D1 can be reduced, and appropriate driving assistance can be performed.

[0083] [Example of timing for displaying vehicle images] Next, the timing of displaying the vehicle image will be described. The vehicle image may be constantly displayed in front of the vehicle C1, or may be displayed only when a predetermined condition is met. The timing when the predetermined condition is met may be, for example, when the destination of the vehicle C1 is set by the route setting unit 121 and the travel route of the vehicle C1 to the destination is determined, and the vehicle C1 approaches a guidance point on the travel route. Examples of this are shown in Figs. 8 and 9.

[0084] For example, when the driver D1 sets the destination of the vehicle C1, the route setting unit 121 can generate a travel route of the vehicle C1 based on the map information DB 170. In this way, when the travel route is set by the route setting unit 121, it is possible to grasp in advance a position (for example, a change point) where some driving support is required. For example, on a straight road with good visibility, it is often not necessary to display a vehicle image in front of the vehicle C1. In contrast, when it is necessary to turn right or left at an intersection, it is necessary to display a vehicle image in front of the vehicle C1.

[0085] In this embodiment, therefore, a position where the vehicle C1 needs to turn, stop, or change lanes will be referred to as a guidance point. That is, a guidance point means a position where guidance is required to support the driving of the vehicle C1, such as a branch point on the travel route of the vehicle C1, such as a position where the vehicle C1 needs to turn, a position where the vehicle C1 needs to stop, or a position where the vehicle C1 needs to change lanes. For example, a guidance point is a right / left turn intersection point, a lane change point, a railroad crossing position, etc.

[0086] For example, when the vehicle C1 needs to turn right at an intersection, a vehicle image is displayed in front of the vehicle C1 near the intersection, and the vehicle image is made to turn right. Also, for example, when the vehicle C1 approaches a railroad crossing, a vehicle image is displayed in front of the vehicle C1 near the railroad crossing, and the vehicle image is made to stop in front of the railroad crossing. Also, for example, when the vehicle C1 approaches a position where traffic regulations are in place (for example, a one-way traffic stop), a vehicle image is displayed in front of the vehicle C1 near the position, and the vehicle image is made to stop in front of the position.

[0087] Fig. 8 is a top view showing a vehicle C1 traveling on a road R11 that intersects with a road R12. Fig. 8 shows an example in which the vehicle C1 turns right at an intersection IS1. In this way, the intersection IS1 where the vehicle turns right or left can be set as a guidance point GP1. Note that Fig. 8 shows an example in which the center position of the intersection IS1 is set as the guidance point GP1, but another position near the intersection IS1 may also be set as the guidance point GP1.

[0088] 8 shows an example in which a vehicle image 300 is displayed on the condition that a vehicle C1 traveling on a road R11 approaches a guide point GP1. For example, when the distance between the vehicle C1 and the guide point GP1 becomes equal to or less than a threshold L1, the vehicle image 300 can be displayed at a predetermined position in front of the vehicle C1 (for example, a position a distance L2 away from the vehicle C1). The threshold L1 and the distance L2 can be set appropriately based on an experiment, a simulation, or the like.

[0089] Fig. 9 is a top view showing a vehicle C1 traveling on a three-lane road R21. Fig. 9 shows an example of a case where a lane change is made on road R21. In this way, it is possible to set the position where the lane change is made as a guidance point GP2.

[0090] 9 shows an example in which a vehicle image 300 is displayed on the condition that a vehicle C1 traveling on a road R21 approaches a guide point GP2. For example, when the distance between the vehicle C1 and the guide point GP2 becomes equal to or less than a threshold L11, the vehicle image 300 can be displayed at a predetermined position in front of the vehicle C1 (for example, a position distance L12 away from the vehicle C1). The threshold L11 and the distance L12 can be set appropriately based on an experiment, a simulation, or the like.

[0091] 8 and 9 show an example in which the vehicle image 300 is displayed, but the display timing can also be set in the same manner when vehicle images 310, 320, etc., which are customized from the vehicle image 300, are displayed.

[0092] Here, the vehicle image to be displayed will be described. For example, assume that the contents of the preference information DB160 are erased when the driver D1 gets off the vehicle C1. In this case, the vehicle image 300 as the base is displayed from when the vehicle C1 starts traveling until the preference information DB160 is updated. Then, when the preference information DB160 is updated, vehicle images in which the vehicle image 300 is customized based on the contents of the preference information DB160 are displayed sequentially. Also, every time the preference information DB160 is updated, vehicle images in which the current vehicle image is customized based on the contents of the preference information DB160 are displayed sequentially. In this way, when the latest preference information of the driver D1 is acquired based on the line of sight of the driver D1, the conversation of the driver D1, etc., it is possible to display a vehicle image according to the latest preference information.

[0093] Next, a case where an appearance element of a certain vehicle image is changed to an appearance element of another vehicle image will be described. Here, an example where the vehicle image 300 is changed to the vehicle image 310 will be shown. For example, if the preference information DB 160 has been updated at the timing of displaying the vehicle image, it is possible to display only the customized vehicle image 310 without displaying the vehicle image 300. Also, for example, even if the preference information DB 160 has been updated at the timing of displaying the vehicle image, the vehicle image 300 may be displayed temporarily, and then the customized vehicle image 310 may be displayed. In this case, the vehicle image 300 may be transformed into the vehicle image 310 instantaneously by one image processing, or the vehicle image 300 may be transformed into the vehicle image 310 over a predetermined time while complementing an image that has been subjected to a predetermined image processing during the transformation from the vehicle image 300 to the vehicle image 310. For example, it is possible to perform image processing (for example, morphing) that naturally transforms the vehicle image 300 into the vehicle image 310. This allows the driver D1 to easily recognize the identity of the vehicle image when the vehicle image 300 transforms into the vehicle image 310.

[0094] [Example of displaying a vehicle image overlaid on the vehicle in front] In the above, an example has been shown in which a vehicle image is displayed in a vacant position in the road space ahead of the vehicle C1 when the vacant position is available in which the image vehicle can be placed. Here, when there are multiple vehicles ahead of the vehicle C1, it is assumed that there is no space in the road space ahead of the vehicle C1 to place the image vehicle. Therefore, in the following, an example will be shown in which a vehicle image is displayed superimposed on another vehicle in front of the vehicle C1 when there is no space in the road space ahead of the vehicle C1 to place the image vehicle.

[0095] Fig. 10 is a simplified diagram showing an actual view seen from the front window 4 of a vehicle C1 traveling on a road R31 intersecting with a road R32, and a vehicle image 310. Fig. 10 shows an example in which the vehicle C1 turns right at an intersection IS11. Also, an example in which another vehicle 400 traveling ahead of the vehicle C1 is present near the intersection IS11 and turns right at the intersection IS11 is shown.

[0096] As shown in FIG. 10(A), when the vehicle C1 approaches the intersection IS11, a vehicle image 310 for guiding the vehicle C1 to turn right at the intersection IS11 is displayed as shown in FIG. 10(B). However, as shown in FIG. 10(A), since a vehicle 400 exists in front of the vehicle C1, there is no vacant position for displaying a vehicle image in the road space in front of the vehicle C1. The vehicle 400 in front also turns right at the intersection IS11 like the vehicle C1. In such a case, as shown in FIG. 10(B), a vehicle image 310 for guiding the vehicle C1 to turn right at the intersection IS11 is displayed superimposed on the vehicle 400 in front. In this case, the vehicle image 310 may be displayed so as to be superimposed on the entire vehicle 400 in front, or the vehicle image 310 may be displayed so as not to be superimposed on a part of the vehicle 400 in front. It is possible to determine whether the vehicle 400 in front will turn right or left based on the direction indicator of the vehicle 400. The light emission state of the direction indicator can be acquired based on an image of the area ahead of the vehicle C1 acquired by the image acquisition unit 101.

[0097] 10B, by displaying a vehicle image 310 that is easy for the driver D1 to view on the vehicle 400 in front of the vehicle C1 in accordance with the driver D1's preference, the driver D1 can easily find the vehicle image 310 in the road space ahead. In this case, the driving burden on the driver D1 can be reduced, and appropriate driving assistance can be performed.

[0098] Here, when there are many other vehicles in the real environment around the vehicle C1, there is a risk that occlusion occurs and it becomes difficult to view the vehicle image. In contrast, in this embodiment, by displaying the vehicle image 310 according to the preference of the driver D1 superimposed on the vehicle 400, it is possible to prevent the occurrence of occlusion and make it easy to view the vehicle image even when there are many other vehicles in the real environment around the vehicle C1. This makes it possible to reduce the driving burden on the driver D1 and to perform appropriate driving assistance.

[0099] [Example of operation of information processing device] Fig. 11 is a flowchart showing an example of a vehicle image display process in the information processing device 110. This vehicle image display process is executed by the control unit 120 (see Fig. 1) based on a program stored in the storage unit 130 (see Fig. 1). This vehicle image display process is constantly executed for each control cycle. This vehicle image display process will be described with reference to Figs. 1 to 10 as appropriate.

[0100] Also, FIG. 11 shows an example in which a destination of the vehicle C1 is set by the route setting unit 121, and a travel route of the vehicle C1 to the destination is set, and a vehicle image is used to guide the vehicle C1 along the travel route.

[0101] In step S501, the moving body detection unit 122 executes an object detection process to detect an object existing around the vehicle C1 based on each piece of information output from at least one of the image acquisition unit 101, the sound acquisition unit 102, and the sensors 103. For example, the moving body detection unit 122 detects the position, attitude, size, speed, etc. of an object (e.g., a vehicle, a motorcycle, a pedestrian, an obstacle, etc.) existing around the vehicle C1 using a sensor (e.g., a laser radar, a millimeter wave radar, a camera, etc.) that detects various objects. In addition, for example, the moving body detection unit 122 generates object detection information as a detection result that can express the two-dimensional position, attitude, size, speed, etc. of the detected object in a zenith diagram of the vehicle C1 viewed from above.

[0102] In step S502, the moving object detection unit 122 calculates the most reasonable object position that minimizes the error of the object position, etc., based on the object detection information obtained in the object detection process (i.e., the position, posture, size, speed, etc. of the detected object) while also taking into consideration the error characteristics of each sensor, etc., and outputs one piece of object detection information (e.g., two-dimensional position, posture, size, speed, etc.) for each object. Furthermore, the moving object detection unit 122 performs object identity verification (association) between the object detection information (e.g., the position, posture, size, speed, etc. of the detected object) output at different times, and estimates the speed information of the object based on the association. In this way, the moving object detection unit 122 functions as an object detection unit and an object detection integration / tracking unit.

[0103] In step S503, the determination unit 123 executes a vehicle position information acquisition process to acquire the position information of the vehicle C1 based on the information output from the sensors 103 (position information acquisition unit). Here, an example is shown in which the absolute position of the vehicle C1 (i.e., the position, attitude, speed, etc. relative to a certain reference point) is used as the position information of the vehicle C1.

[0104] In step S504, the determination unit 123 executes a map information acquisition process to acquire map information from the map information DB 170. In this map information acquisition process, map information such as absolute lane positions, lane connection relationships, and relative position relationships is acquired from the map information DB 170.

[0105] In step S505, the determination unit 123 executes an in-map vehicle position estimation process to estimate the position of the vehicle C1 on the map based on the absolute position of the vehicle C1 acquired in step S503 and the map information acquired in step S504. That is, in this in-map vehicle position estimation process, information on which road and which lane the vehicle C1 is traveling on is acquired. In this way, the determination unit 123 functions as a vehicle position estimation unit, a map information acquisition unit, and an in-map vehicle position estimation unit.

[0106] In step S510, the control unit 120 executes a vehicle image display process for displaying a vehicle image on the display device 200. This vehicle image display process will be described in detail with reference to FIG.

[0107] [Example of vehicle image display processing] Fig. 12 is a flowchart showing an example of the vehicle image display process (step S510 shown in Fig. 11) in the information processing device 110. This vehicle image display process is executed by the control unit 120 (see Fig. 1) based on a program stored in the storage unit 130 (see Fig. 1). This vehicle image display process will be described with appropriate reference to Figs. 1 to 11.

[0108] In step S511, the determination unit 123 executes a guidance point extraction process to extract a position (guidance point) where guidance is required to assist the driving of the vehicle C1. For example, the determination unit 123 extracts latitude and longitude corresponding to a guidance point existing ahead of the vehicle C1 from the map information based on the absolute position of the vehicle C1 acquired in step S503 (see FIG. 11) and the map information acquired in step S504 (see FIG. 11). In this case, the determination unit 123 acquires the travel route of the vehicle C1 set by the route setting unit 121, and extracts latitude and longitude corresponding to the guidance point closest to the vehicle C1 among the guidance points existing in the traveling direction of the vehicle C1 on the travel route. For example, as shown in FIG. 8 and FIG. 9, the extraction target is a guidance point whose distance from the vehicle C1 is equal to or less than a threshold value (for example, threshold value L1, L11).

[0109] Also, for example, the determination unit 123 extracts a guidance point present ahead of the vehicle C1 based on the absolute position of the vehicle C1 acquired in step S503 (see FIG. 11) and the traffic regulation information and construction information acquired via the communication unit 140. This guidance point means a position where a traffic regulation is in place. Note that the position where the traffic regulation is in place may be extracted by applying a known image recognition technique using the image information acquired by the image acquisition unit 101.

[0110] In step S512, the determination unit 123 determines whether or not a guidance point was extracted in step S511. If a guidance point was extracted, the process proceeds to step S513. On the other hand, if a guidance point was not extracted, the operation of the vehicle image display process ends. Thus, the example shown in FIG. 12 shows an example in which a vehicle image is displayed when the vehicle C1 approaches a guidance point. Note that the vehicle image may be constantly displayed to perform driving assistance regardless of the relationship between the vehicle C1 and the guidance point.

[0111] In step S513, the determination unit 123 calculates the relative position of the guidance point with respect to the vehicle C1 based on the absolute position of the vehicle C1 acquired in step S503, the map information acquired in step S504, and the guidance point extracted in step S511. For example, the determination unit 123 acquires the latitude and longitude corresponding to the guidance point and the latitude and longitude corresponding to the current location of the vehicle C1 based on the map information, and calculates the relative position of the guidance point with respect to the vehicle C1 based on these latitude and longitude. Note that, although an example of calculating the relative position of the guidance point is shown here, the relative position of the guidance point may be calculated using map coordinates such as Tokyo 9 series instead of latitude and longitude.

[0112] In step S514, the moving object detection unit 122 detects moving objects around the vehicle C1 based on the object detection information detected in the object detection and tracking process (step S502). The moving objects shown here are, for example, vehicles, motorcycles, bicycles, pedestrians, etc.

[0113] In step S515, the determination unit 123 determines whether or not there is a vacant position for displaying a vehicle image in a position where no moving object exists in the road space ahead of the vehicle C1. For example, when there is no other vehicle ahead of the vehicle C1, it is determined that there is a vacant position for displaying a vehicle image in the road space ahead of the vehicle C1. On the other hand, when there are many other vehicles ahead of the vehicle C1 and there is no space between the vehicles to arrange a vehicle image, it is determined that there is no vacant position for displaying a vehicle image in the road space ahead of the vehicle C1. Even when there is only one other vehicle ahead of the vehicle C1, if the road space ahead of the vehicle C1 is narrow, it is determined that there is no vacant position for displaying a vehicle image in the road space ahead of the vehicle C1. If there is a vacant position for displaying a vehicle image in the road space ahead of the vehicle C1, the process proceeds to step S516. On the other hand, if there is no vacant position for displaying a vehicle image in the road space ahead of the vehicle C1, the process proceeds to step S519.

[0114] In step S516, the extraction unit 124 extracts preference information related to the preferences of the driver D1. Specifically, the extraction unit 124 acquires at least one of the first preference information 161 and the second preference information 165 stored in the preference information DB 160.

[0115] In step S517, the determination unit 125 determines the exterior elements of the vehicle image to be displayed in front of the vehicle C1 based on the preference information extracted in step S516, and generates a vehicle image that will be the exterior elements. That is, the determination unit 125 generates a vehicle image customized according to the preference information of the driver D1. For example, as shown in Figs. 4 to 6, the exterior elements of the vehicle image can be determined using the preference information DB160 (see Fig. 5). It is assumed that no information is stored in the preference information DB160 immediately after the driver D1 gets into the vehicle C1. In this case, the exterior elements of the vehicle image can be determined using each piece of information related to the vehicle type information 151 "own vehicle" in the appearance element information DB150 (see Fig. 4).

[0116] In step S518, the display control unit 126 displays a vehicle image including the exterior elements determined in step S517 in the road space in front of the vehicle C1. For example, as shown in Fig. 7, a vehicle image 310 customized according to the preferences of the driver D1 is displayed.

[0117] In step S519, the determination unit 123 selects a moving body (first moving body) that satisfies a predetermined condition from among the moving bodies in front of the vehicle C1. In this case, the moving body in front of the vehicle C1 is selected from among the moving bodies around the vehicle C1. For example, in a case where many other vehicles exist in front of the vehicle C1, it is assumed that even if a vehicle image is displayed in front of the vehicle C1, the vehicle image may be difficult for the driver D1 to see. Therefore, in a case where many other vehicles exist in front of the vehicle C1 and there is no free space between the vehicles to arrange the vehicle image, instead of displaying the vehicle image so as not to overlap each vehicle, a vehicle image having the same appearance elements as the vehicle is displayed superimposed on the actual vehicle visible to the driver D1. Then, the vehicle image displayed superimposed on the actual vehicle is gradually transformed (customized) on the AR, and a vehicle image (customized vehicle image) that matches the preferences of the driver D1 is displayed. In this case, a predetermined customization may be performed on the vehicle image displayed superimposed on the first moving body so that the vehicle image is easily noticed. For example, the vehicle image may be caused to blink for a predetermined period of time, or the color of the vehicle image may be changed according to the preference of the driver D1.

[0118] The first moving object may be, for example, another vehicle closest to the guide point extracted in step S511. For example, the determination unit 123 selects another vehicle closest to the guide point based on the relative position between the vehicle C1 and the guide point and the relative position between the vehicle C1 and another vehicle ahead. In this case, the other vehicle that is closer to the guide point than a predetermined threshold and is considered to be most visible from the driver D1's perspective may be selected. Note that the other vehicle that is most visible from the driver D1's perspective is, for example, the vehicle that is the largest in size in the image among multiple vehicles included in the image ahead of the vehicle C1 acquired by the image acquisition unit 101. In this way, the determination unit 123 functions as a moving object selection unit close to the point.

[0119] Steps S520 and S521 correspond to steps S516 and S517, and therefore description thereof will be omitted here. However, in step S521, the appearance elements of the vehicle image may be determined in consideration of the first moving object selected in step S519. For example, the appearance elements of the vehicle image may be determined to have the same color as or a color close to the color of the body of the first moving object.

[0120] In step S522, the display control unit 126 displays the vehicle image having the exterior elements determined in step S521 so as to be superimposed on the first moving object selected in step S519. For example, as shown in Fig. 8(A)(B), the display control unit 126 displays the vehicle image 310 so as to be superimposed on the vehicle 400 traveling ahead of the vehicle C1.

[0121] Here, it is assumed that there is a vacant position for displaying a vehicle image in the road space ahead of the vehicle C1, and after the vehicle image is displayed in that vacant position (steps S515 to S518), the vacant position for displaying the vehicle image in the road space ahead of the vehicle C1 may disappear due to the movement of the vehicle C1 and other vehicles, etc. In such a case, the vehicle image displayed in the vacant position in step S518 may be displayed superimposed on another vehicle in step S522. For example, in order to make it easier for the driver D1 to grasp the identity between the vehicle image (first vehicle image) displayed in step S518 and the vehicle image (second vehicle image) displayed superimposed on another vehicle in step S522, it is possible to display the second vehicle image so as to superimpose it on all or part of the other vehicle present in the position closest to the first vehicle image that has been displayed until now. In addition, for example, when the distance between the first vehicle image and the second vehicle image is large, the transition regarding the movement from the first vehicle image to the second vehicle image may be displayed as an animation image or the like so that the driver D1 can easily grasp the identity of the first vehicle image and the second vehicle image.

[0122] In the above, an example has been shown in which it is determined in step S515 whether or not there is a vacant position in the road space ahead of the vehicle C1 where a moving object is not present and a vehicle image is displayed, and the position where the vehicle image is displayed is changed according to the determination result. However, the determination process in step S515 may be omitted, and a vehicle image customized according to the preferences of the driver D1 may be displayed at a predetermined position in the road space ahead of the vehicle C1. Even in this case, the driver D1 can easily find the vehicle image ahead of the vehicle C1, the driving burden on the driver D1 is reduced, and appropriate driving assistance can be performed.

[0123] [Example of displaying vehicle images in locations such as roads with poor visibility] The above shows an example of extracting guidance points, but it is also necessary to provide guidance to warn drivers about poor visibility on roads, etc. Therefore, the following shows an example of providing guidance about points with poor visibility, etc.

[0124] For example, the determination unit 123 executes poor visibility point extraction processing to extract positions (poor visibility points) where guidance is required to assist driving of the vehicle C1. For example, the determination unit 123 extracts poor visibility points in front of the vehicle C1 based on the absolute position of the vehicle C1 acquired in step S503 (see FIG. 11) and the map information acquired in step S504 (see FIG. 11). The determination unit 123 may also extract poor visibility points in front of the vehicle C1 using traffic regulation information, construction information, image information acquired by the image acquisition unit 101, and the like acquired via the communication unit 140. The poor visibility points are, for example, T-junctions with many blind spots, crossroads or intersections, intersections without traffic lights, and curves with poor visibility.

[0125] Here, when a point with poor visibility is extracted, instead of displaying a vehicle image with the same traveling direction as the traveling direction of the vehicle C1, it is possible to display a vehicle image with a traveling direction opposite to the traveling direction of the vehicle C1. In this case, it is possible to display a display that allows the driver D1 to easily understand that there is a possibility of another moving object coming from the oncoming lane. In this way, when the vehicle C1 approaches a T-junction with many blind spots, a curve with poor visibility, etc., it is possible to display a vehicle image in a part that is difficult for the driver D1 to see, and to warn that there is a possibility of another moving object coming from the other side. It is also possible to display a vehicle image in front of the vehicle C1 and display some kind of warning information (for example, a warning in text information) on this vehicle image. In this way, in addition to displaying a vehicle image in front of the vehicle C1 to guide the vehicle C1, it is possible to display a vehicle image to inform the driver D1 that another moving object is coming from a position with poor visibility, a side road of the vehicle C1, etc. In this way, the determination unit 123 functions as a guidance point extraction unit, a poor visibility point extraction unit, and a point relative position calculation unit.

[0126] [Example of vehicle image display other than AR-HUD display] In this embodiment, an example of displaying a vehicle image by AR-HUD display has been shown, but the vehicle image may be displayed using other display media. For example, a glasses-type display device that can be worn by a driver D1 who gets into a vehicle C1 can be used. This glasses-type display device is a device that can recognize a real space using various sensors and provide a user with a representation that combines the real space with various information such as image information. For example, it is possible to display various images to a user who wears a glasses-type display device on his or her eyes, superimposed on a real scene seen through the display device. It is possible to use AR glasses and MR (Mixed Reality) glasses as glasses-type display devices. For example, it is possible for a driver D1 who gets into a vehicle C1 to wear AR glasses and display an image vehicle superimposed on a real scene seen through the AR glasses.

[0127] Also, for example, each display device installed in the vehicle C1 can be used. As this display device, an IVI (In-Vehicle Infotainment) or a navigation device can be used. If the navigation image displayed by the navigation device can be displayed on the HUD, the navigation image can be displayed on the HUD and the vehicle image can be displayed on the HUD display. In this way, it is possible to support driving by customizing the virtual vehicle image displayed in AR according to the preferences of the driver D1 using various display media.

[0128] [Example of executing processing on other devices or systems] In the above, an example has been shown in which the extraction process, the determination process, and the like are executed in the information processing device 110 (or the information processing system 100), but all or part of each process may be executed in other devices. In this case, the information processing system is configured by each device that executes part of each process. For example, at least part of each process can be executed using various driving support devices such as in-vehicle devices, devices that can be used by the user (e.g., smartphones, tablet terminals, personal computers, car navigation devices, IVI), and servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0129] In addition, a part (or the whole) of the information processing system capable of executing the functions of the information processing device 110 (or the information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0130] [Configuration Example and Effects of This Embodiment] The driving support method according to the present embodiment includes a display process (steps S518, S522) for displaying a vehicle image, which is a virtual image, superimposed on an actual scene so that the driver D1 sees a leading vehicle at a predetermined position in front of the vehicle C1, and is a driving support method for supporting the driver D1 in driving the vehicle C1. This driving support method includes an extraction process (steps S516, S520) for extracting preference information related to the preferences of the driver D1 based on at least one of appearance information related to the vehicle C1, visual information related to the actual scene visually recognized by the driver D1, and speech information of the driver D1, and a determination process (steps S517, S521) for determining appearance elements constituting the vehicle image based on the preference information. The program according to the present embodiment is a program for causing a computer to execute each of these processes. In other words, the program according to the present embodiment is a program for causing a computer to realize each function executable by the information processing device 110.

[0131] According to this configuration, by displaying the vehicle image according to the preference of the driver D1, even when there are many vehicles in the real environment, the vehicle image becomes easy to view, and the burden on the driver D1 in finding the vehicle image can be reduced. In other words, by displaying a vehicle image that is easy for the driver D1 to view according to the preference of the driver D1, it becomes possible to perform appropriate driving assistance.

[0132] The vehicle image determined in the determination process can be displayed after undergoing a predetermined image processing (e.g., morphing). In this way, by performing a predetermined image processing on the vehicle image displayed based on the determined appearance elements, it is possible to customize the vehicle image in detail according to each element constituting the vehicle image.

[0133] The driving assistance method according to this embodiment further includes a detection process (step S514) for detecting moving objects existing around the vehicle C1, and a determination process (step S515) for determining whether or not there is an empty position for displaying a vehicle image in a position where the detected moving object does not exist in the road space existing in front of the vehicle C1. In the display process (step S518), if an empty position exists, the vehicle image determined in the determination process (step S517) is displayed in the empty position. On the other hand, in the display process (step S522), if an empty position does not exist, the vehicle image determined in the determination process (step S521) is displayed so as to be superimposed on a predetermined moving object among the moving objects detected in the road space.

[0134] According to this configuration, even if there are many other vehicles in the real environment around the vehicle C1, the vehicle image according to the preference of the driver D1 can be displayed superimposed on the other real vehicles, thereby preventing the occurrence of occlusion and making it easy to view the vehicle image. This reduces the driving burden on the driver D1 and makes it possible to perform appropriate driving assistance.

[0135] In the driving assistance method according to this embodiment, in the determination process (steps S517, S521), if preference information is extracted based on the appearance information of the vehicle C1 in the extraction process (steps S516, S520), the appearance element is determined using the design information of the exterior of the vehicle C1 contained in the appearance information. For example, it is possible to determine the initial setting appearance element (vehicle image 300) based on the appearance element information 154 corresponding to the vehicle type information 151 "own vehicle" in the appearance element information DB 150 (see FIG. 4).

[0136] According to this configuration, the preference information of the driver D1 is extracted using the exterior design elements of the vehicle C1 in which the driver D1 is riding, so that the essential preferences of the driver D1 can be appropriately extracted.

[0137] In the driving support method according to the present embodiment, in the extraction process (steps S516, S520), when the preference information of the driver D1 is extracted based on the visual information of the driver D1, other vehicles present in the line of sight of the driver D1 are detected based on the line of sight of the driver D1, and vehicle type information (preference information) related to the vehicle type according to the preference of the driver D1 is extracted based on the detected other vehicles. In the determination process (steps S517, S521), when the vehicle type information (preference information) is extracted based on the visual information of the driver D1 in the extraction process (steps S516, S520), the vehicle type information is used to determine the appearance element. For example, it is possible to determine the appearance element that matches the preference of the driver D1 from the appearance element information 154 of the appearance element information DB 150 (see FIG. 4) using the vehicle information 162 of the preference information DB 160 (see FIG. 5).

[0138] According to this configuration, by extracting the preference information of the driver D1 using the exterior design elements of the surrounding vehicles that the driver D1 is gazing at, the preferences currently held by the driver D1 at that time can be appropriately extracted.

[0139] In the driving support method according to the present embodiment, in the extraction process (steps S516, S520), when the preference information of the driver D1 is extracted based on the speech information of the driver D1, keyword information (preference information) related to the keyword corresponding to the preference of the driver D1 is extracted based on the speech information of the driver D1. In the determination process (steps S517, S521), when keyword information (preference information) is extracted based on the speech information of the driver D1 in the extraction process (steps S516, S520), the appearance element is determined using the keyword information. For example, it is possible to determine the appearance element that matches the preference of the driver D1 from the appearance element information 154 of the appearance element information DB150 (see FIG. 4) using the keyword information 166 of the preference information DB160 (see FIG. 5).

[0140] According to this configuration, by extracting the preference information of the driver D1 using conversations with passengers or the like in the vehicle C1, it is possible to appropriately extract the latent preferences currently held by the driver D1 at that time.

[0141] In the driving assistance method according to this embodiment, in the determination process (steps S517, S521), the appearance elements are determined using a learning model (e.g., a machine learning model) that has learned the association between keyword information and changes in vehicle images.

[0142] According to this configuration, the vehicle image can be freely customized in accordance with keywords according to the preferences of the driver D1.

[0143] The appearance elements may be determined using a learning model (e.g., a machine learning model) that has learned the association between the appearance elements of the vehicle image and changes in the vehicle image. In this case, customization can be performed taking into account complex changes in the vehicle image.

[0144] The driving assistance method according to this embodiment further includes a vehicle position acquisition process (step S503) for acquiring vehicle position information indicating the current location of the vehicle C1, a guidance position acquisition process (step S511) for acquiring guidance position information relating to a position where guidance is required to assist the driving of the vehicle C1, and a determination process (step S511) for determining that the vehicle C1 has approached the position where guidance is required based on the vehicle position information and the guidance position information. In a display process (steps S518, S522), when the vehicle C1 approaches the position where guidance is required, the vehicle image determined in the determination process (steps S517, S521) is displayed.

[0145] According to this configuration, it is possible to display the vehicle image at the timing when guidance for assisting the driving of the vehicle C1 is required, thereby reducing the burden on the driver D1 of finding the vehicle image in a place other than the place where such guidance is required.

[0146] In the driving assistance method of this embodiment, locations where guidance is required to assist in driving of vehicle C1 are locations where vehicle C1 needs to turn right or left, locations where vehicle C1 needs to change lanes, locations where vehicle C1 needs to stop or slow down, locations with many blind spots from vehicle C1, locations with poor visibility from vehicle C1, or locations near intersections where no traffic lights are installed.

[0147] This configuration makes it easy to understand the route of the vehicle C1. It also makes it possible to reduce the risk of collision even in a moving body that is difficult for the driver D1 to see and in a traffic environment with poor visibility.

[0148] The information processing device 110 is a driving support device that includes a display control unit 126 that displays a vehicle image, which is a virtual image, superimposed on an actual scene so that the driver D1 sees a leading vehicle positioned at a predetermined position in front of the vehicle C1, and supports the driver D1 in driving the vehicle C1. The information processing device 110 includes an extraction unit 124 that extracts preference information related to the preferences of the driver D1 based on at least one of appearance information related to the vehicle C1, visual information related to the actual scene visually recognized by the driver D1, and speech information of the driver D1, and a determination unit 125 that determines appearance elements that constitute the vehicle image based on the preference information. The information processing device 110 may be a device built into the display device 200, or may be a device different from the display device 200. Instead of the information processing device 110, an information processing system configured of multiple devices capable of executing each process realized by the information processing device 110 may be used.

[0149] According to this configuration, by displaying the vehicle image according to the preference of the driver D1, even when there are many vehicles in the real environment, the vehicle image becomes easy to view, and the burden on the driver D1 in finding the vehicle image can be reduced. In other words, by displaying a vehicle image that is easy for the driver D1 to view according to the preference of the driver D1, it becomes possible to perform appropriate driving assistance.

[0150] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that allows the realization of this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0151] Each process of the present embodiment is executed based on a program for causing a computer to execute various processing procedures. The present embodiment can also be understood as an embodiment of a program for implementing the function of executing each process, and a recording medium for storing the program. For example, an update process for adding a new function to the driving assistance device can store the program in the storage device of the driving assistance device. This makes it possible to cause the updated driving assistance device to execute each process shown in the present embodiment.

[0152] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0153] 20 network, 100 information processing system, 101 image acquisition unit, 102 sound acquisition unit, 103 sensors, 110 information processing device, 120 control unit, 121 route setting unit, 122 moving object detection unit, 123 determination unit, 124 extraction unit, 125 decision unit, 126 display control unit, 130 storage unit, 140 communication unit, 150 appearance element information DB, 160 preference information DB, 170 map information DB, 200 display device

Claims

1. A driving assistance method for assisting a driver in driving a vehicle, the method comprising: displaying a vehicle image, which is a virtual image, superimposed on an actual scene so that the driver sees a leading vehicle positioned at a predetermined position in front of the vehicle; an extraction process for extracting preference information related to the driver's preferences based on at least one of appearance information related to the vehicle, visual information related to the actual scene visually recognized by the driver, and speech information of the driver; A determination process for determining appearance elements constituting the vehicle image based on the preference information. Driving assistance methods.

2. The driving assistance method according to claim 1, A detection process for detecting a moving object existing around the vehicle; and determining whether or not there is an empty position in a road space in front of the vehicle where the detected moving object is not present and in which the vehicle image is to be displayed, In the display process, when the vacant position exists, the vehicle image determined in the determination process is displayed in the vacant position, whereas, when the vacant position does not exist, the vehicle image determined in the determination process is displayed so as to be superimposed on a predetermined moving body among the moving bodies detected in the road space. Driving assistance methods.

3. The driving assistance method according to claim 1, In the determination process, when the preference information is extracted based on the appearance information in the extraction process, the appearance element is determined using design information of the exterior of the vehicle included in the appearance information. Driving assistance methods.

4. The driving assistance method according to claim 1, In the extraction process, when the preference information is extracted based on the visual information, another vehicle present in the line of sight of the driver is detected based on the line of sight of the driver, and vehicle type information related to a vehicle type according to the preference of the driver is extracted as the preference information based on the detected other vehicle. In the determination process, when the vehicle type information is extracted based on the visual information in the extraction process, the appearance element is determined using the vehicle type information. Driving assistance methods.

5. The driving assistance method according to claim 1, In the extraction process, when the preference information is extracted based on the speech information, keyword information related to a keyword corresponding to the driver's preference is extracted as the preference information based on the speech content of the driver; In the determination process, when the keyword information is extracted based on the speech information in the extraction process, the appearance element is determined using the keyword information. Driving assistance methods.

6. A driving assistance method according to claim 5, In the determination process, the appearance element is determined using a learning model that has learned the association between the keyword information and the change in the vehicle image. Driving assistance methods.

7. A driving assistance method according to any one of claims 1 to 6, a vehicle position acquisition process for acquiring vehicle position information indicating a current location of the vehicle; A guidance position acquisition process for acquiring guidance position information relating to a position where guidance for assisting driving of the vehicle is required; and determining whether the vehicle has approached a position where the guidance is required based on the vehicle position information and the guidance position information, In the display process, when the vehicle approaches a position where the guidance is required, the vehicle image determined in the determination process is displayed. Driving assistance methods.

8. A driving assistance method according to claim 7, The location where the guidance is required is a location where the vehicle needs to turn right or left, a location where the vehicle needs to change lanes, a location where the vehicle needs to stop or slow down, a location where there are many blind spots from the vehicle, a location where visibility from the vehicle is poor, or a location near an intersection where no traffic lights are installed. Driving assistance methods.

9. A driving assistance device that includes a display control unit that displays a vehicle image, which is a virtual image, superimposed on an actual scene so that the driver sees a leading vehicle positioned at a predetermined position in front of the vehicle, and assists the driver in driving the vehicle, an extraction unit that extracts preference information related to the driver's preferences based on at least one of appearance information related to the vehicle, visual information related to the actual scene visually recognized by the driver, and speech information of the driver; a determination unit that determines appearance elements constituting the vehicle image based on the preference information, Driving assistance device.

Images