Information recording support method, information recording support device, information recording support program, and information recording support system

Abstract

To provide a method, a device, a program, and a system for supporting information recording that support recording of information very naturally by a vehicle and the like without giving a sense of incongruity to a passenger.SOLUTION: An information recording support method is implemented in a vehicle 100 including a vehicle exterior camera 29 and an agent system 15 that is an information recording support device for supporting information recording at an arbitrary timing using the vehicle exterior camera 29 via an agent unit 23. With the information recording support method, motion and / or conversation of a passenger 10 is detected and an object of interest of the passenger 10 outside the vehicle 100 is specified based on the motion and / or conversation of the passenger 10. When the object of interest is specified, the agent system 15 notifies the passenger 10 that the object of interest has been specified, and the vehicle exterior camera 29 records the object of interest.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information recording support method, an information recording support device, an information recording support program, and an information recording support system that support information recording at any timing using an exterior camera that captures images outside the vehicle. [Background technology]

[0002] Patent Document 1 discloses an image and audio recording and playback device that is composed of a navigation device installed in a vehicle and a recording and playback device that is a computer installed in a home, etc. The image and audio recording and playback device in Patent Document 1 records video data and audio data inside the vehicle using the navigation device and transmits this data to the recording and playback device, thereby eliminating the need for passengers to take pictures themselves and enabling the recorded video data and audio data to be easily played back on a computer. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-105174 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, vehicles or devices mounted on vehicles (hereinafter referred to as vehicles, etc.) have sometimes been provided with a function to assist passengers in operating the devices or obtaining information based on, for example, interactive voice input. As one such assistance function, an information recording assistance function that automatically records videos, etc. as memories of travel, etc., is being considered.

[0005] However, if a vehicle, etc. automatically records videos, etc. and then provides the automatically recorded videos, etc. to passengers as memories of a trip, etc., the passengers may form a negative impression of the vehicle, etc. that automatically recorded the videos, etc. Specifically, if the vehicle, etc. does not show any reaction to the recording of information such as videos, but instead records the information arbitrarily, the passengers may feel uncomfortable even if the recorded information is later provided as a memory of the passenger or memories of the passenger.

[0006] The present invention aims to provide an information recording support method, an information recording support device, an information recording support program, and an information recording support system that naturally support the recording of information in a vehicle or the like without causing any discomfort to the occupants. [Means for solving the problem]

[0007] One aspect of the present invention is an information recording support method for a vehicle that includes an exterior camera that captures images outside the vehicle, and an information recording support device that has an agent unit that functions as an interface for the occupant and supports information recording at any timing using the exterior camera via the agent unit. In this information recording support method, the occupant's movements and / or conversations are detected, and an object of interest to the occupant that is outside the vehicle is identified based on the occupant's movements and / or conversations. When the object of interest is identified, the information recording support device notifies the occupant that the object of interest has been identified, Whether or not the passenger recognizes that the information recording assistance device has identified the subject of interest is determined based on the passenger's actions and / or conversation. When the passenger recognizes that the information recording assistance device has identified the subject of interest, Target of interest detected by exterior camera Recording begins will be done. [Effects of the Invention]

[0008] According to the present invention, recording of information can be assisted in a very natural manner in a vehicle or the like without causing any discomfort to the passengers. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram showing a schematic configuration of a vehicle and an information recording support system. [Figure 2]FIG. 2 is a block diagram showing the configuration of the agent controller. [Figure 3] FIG. 3 is a flowchart showing the process of starting automatic information recording. [Figure 4] FIG. 4 is a flowchart showing the process of ending automatic information recording. [Figure 5] FIG. 5 is a flowchart showing a specific example of determining loss of interest. [Figure 6] FIG. 6 is an explanatory diagram of a scene in which automatic information recording starts. [Figure 7] FIG. 7 is an explanatory diagram of a scene during automatic information recording. [Figure 8] FIG. 8 is an explanatory diagram of a scene where automatic information recording ends. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [First embodiment] FIG. 1 is an explanatory diagram showing the schematic configuration of a vehicle 100 and an information recording assistance system 101. The vehicle 100 uses an agent system 15 to receive, for example, interactive operation input from a passenger 10 of the vehicle 100 and / or provide various information to assist the passenger 10 in operating the vehicle 100 and acquiring information. In this embodiment, the vehicle 100 uses the agent system 15 to assist the passenger 10 in recording images, videos (videos), audio, or other information (hereinafter referred to as videos, etc.) at any timing. Therefore, the vehicle 100 including the agent system 15 constitutes the information recording assistance system 101. The passenger 10 is the driver and / or a passenger. In this embodiment, for simplicity, the passenger 10 is assumed to be the driver of the vehicle 100.

[0012] As shown in FIG. 1, the vehicle 100 includes an agent system 15, a driving operation unit 11, a car navigation system 12, a communication device 13, a speaker 14, a vehicle controller 16, and a sensor 17.

[0013] The driving operation unit 11 is an operation unit that is operated by the passenger 10 to drive the vehicle 100. The driving operation unit 11 is, for example, a steering wheel, an accelerator pedal, a brake pedal, and the like.

[0014] The car navigation system 12 is a guidance device that provides the passenger 10 with information such as a route to a destination. The car navigation system 12 has its own display and operation keys. The passenger 10 uses the operation keys of the car navigation system 12 to perform operations necessary for route guidance, such as setting a destination. The passenger 10 can also receive information such as route guidance on the display of the car navigation system 12 and / or through sounds or voices output from the speaker 14.

[0015] The car navigation system 12 includes map information 21, landmark information 22, and the like.

[0016] The map information 21 is a database containing maps and traffic information associated with the maps, etc. The map information 21 can be updated as needed by, for example, communicating with the outside of the vehicle 100 using the communication device 13.

[0017] The landmark information 22 is a database containing information about landmarks. Landmarks are generally famous and distinctive buildings (skyscrapers, bridges, etc.), structures (monuments, etc.), natural objects (mountains, etc.), and spaces or locations (parks, etc.). Examples of landmarks include famous structures such as Tokyo Tower (registered trademark) and Skytree (registered trademark), and the mountain Mount Fuji. In this embodiment, buildings and the like pre-registered in the landmark information 22 are treated as landmarks. Landmarks currently located around the vehicle 100 can be identified based on the position, orientation, etc. of the vehicle 100. In this embodiment, the area around the vehicle 100 refers to, for example, a range visible to the passenger 10, regardless of the distance from the vehicle 100, and landmarks located around the vehicle 100 refer to landmarks that the passenger 10 can see in part or in whole. The landmark information 22 can be updated as needed, for example, through information communication with the outside of the vehicle 100 using the communication device 13.

[0018] The map information 21 and landmark information 22 are not only used by the car navigation system 12 for route guidance and the like, but can also be used appropriately by each part of the vehicle 100, such as the agent system 15 and the vehicle controller 16.

[0019] The communicator 13 is a communication interface that wirelessly communicates with surrounding vehicles, transportation facilities, and / or any other systems, servers, etc. The communicator 13 connects to devices, etc. that are the communication targets, directly or via an internet line, etc., and transmits and receives various information. In this embodiment, the vehicle 100 is equipped with the communicator 13, but the communicator 13 may be provided as part of the car navigation system 12 or the agent system 15. Regardless of the affiliation of the communicator 13, each part of the vehicle 100, i.e., the car navigation system 12, the agent system 15, the vehicle controller 16, etc., can use the communicator 13 to appropriately communicate information with servers, etc., outside the vehicle 100.

[0020] The speaker 14 outputs sounds, voices, etc. to the interior of the vehicle 100. Each part of the vehicle 100 can access the speaker 14 as needed to output voices, etc. In this embodiment, the car navigation system 12 outputs sounds and voices from the speaker 14 for route guidance, etc. The agent system 15 also outputs sounds, voices, etc. from the speaker 14 for dialogue with the passenger 10 and for providing information to the passenger 10, etc.

[0021] The agent system 15 is a system that provides support for operating the vehicle 100 or various devices and systems installed in the vehicle 100, and for providing information that the vehicle 100 should perform. The agent system 15 provides the support through interactive communication, in which the agent system 15 exchanges information with the passenger 10. In particular, in this embodiment, the agent system 15 provides support for recording videos and the like that will become memories of a trip, etc., at any timing. In other words, the agent system 15 constitutes an information recording support device that supports information recording at any timing.

[0022] Specifically, the agent system 15 includes an agent unit 23, an agent controller 24, a display 25, and a recording device 26.

[0023] The agent section 23 is a section that functions as an interface for the passenger 10. In other words, the agent system 15 uses the agent section 23 to carry out interactive communication with the passenger 10.

[0024] The agent unit 23 is, for example, a mascot (also referred to as a mascot character or simply a character) or assistant that is represented in the form of a person, animal, plant, object, or other symbol. The agent unit 23 is configured physically using a mechanical mechanism such as a robot, or is configured virtually using computer graphics or the like. In this embodiment, the agent unit 23 is a virtual configuration that is displayed on the display 25.

[0025] The agent unit 23 is anthropomorphized to the extent that it can at least become a target of communication for the passenger 10. In other words, the agent unit 23 is configured to be able to express its own feelings and emotions, which are changes in feelings, through sounds or voices and / or changes in form. In this embodiment, the form of the agent unit 23 refers to the overall shape of the agent unit 23 (e.g., the direction and balance of the face and limbs), the partial shape (e.g., the shape, direction, or balance of parts such as the eyes and eyebrows), and the pattern or color (e.g., facial color). Therefore, a change in form refers to, for example, the movement of the face or body of the agent unit 23, a change in the line of sight of the agent unit 23, a change in facial expression of the agent unit 23, or a combination of these. Note that if the tone of voice output by the agent unit 23 is changeable, the change in tone of voice is also included in the above-mentioned change in form.

[0026] The agent unit 23 performs an attention target display operation to display an object to which the agent unit 23 directs its attention (hereinafter referred to as an attention target) or to display the presence of an attention target by changing its form or issuing a sound or other alert. In this embodiment, the agent unit 23 can execute a first operation, a second operation, a third operation, or a combination of these operations as the attention target display operation.

[0027] The first action is an action that displays the target of attention or indicates the presence of the target of attention by changing the form of moving at least a part of the agent unit 23 toward the target of attention. For example, an action of making a gesture according to the presence, position, or nature of the target of attention is the first action. More specifically, an action of directing the gaze toward the target of attention, an action of pointing at the target of attention, an action of turning the face or body toward the target of attention, or an action that combines these actions are the first action.

[0028] The second action is an action of displaying an object of attention of the agent unit 23 or displaying that there is an object of attention of the agent unit 23 by changing the form of the agent unit 23 to change the color and / or pattern, etc. of the agent unit 23. For example, an action of displaying an emotion or feeling according to the presence or absence, position, or nature, etc. of an object of attention is the second action. More specifically, an action of showing a facial expression according to the object of attention is the second action.

[0029] The third operation is an operation in which the agent unit 23 emits a sound or voice to display an object of attention of the agent unit 23 or to display that there is an object of attention of the agent unit 23. For example, an operation in which a sound or voice message is emitted according to the presence or absence, position, or nature of the object of attention is the third operation.

[0030] The agent unit 23 can perform three types of attention target display actions: a first action, a second action, and a third action. In this embodiment, the attention target is displayed or the presence of the attention target is indicated by a composite action that combines the first action, the second action, and the third action. However, the agent unit 23 can perform an attention target display action that is an arbitrary combination of one or more of the first action, the second action, and the third action.

[0031] The agent controller 24 is a control unit that controls the operation of the agent system 15. More specifically, the agent controller 24 is a computer that is programmed to operate the agent system 15 in response to the information provided by the vehicle 100 to the passenger 10 and the actions of the passenger 10, such as the action of the agent unit 23. The specific configuration of the agent controller 24 will be described in detail later. The program that causes the agent system 15 to operate to assist in recording videos and the like that will become memories of a trip or the like is an information recording assistance program. The information recording assistance program may be provided in the form of a recording medium that can be read by the agent system 15, the vehicle 100, or the information recording assistance system 101. Furthermore, the information recording assistance program may be partially or completely updated after implementation using a recording medium or by so-called OTA (Over The Air) technology.

[0032] The display 25 is a display unit that displays videos and various types of information. When the agent unit 23 is configured virtually, the agent unit 23 is displayed on the display 25. Furthermore, the display 25 can display, for example, videos captured by the vehicle 100. In this embodiment, the display 25 is a HUD (Head-Up Display), and displays the agent unit 23 in a part of the field of view of the passenger 10. Furthermore, the passenger 10 can visually confirm the capture range of the exterior camera 29 (described later) through the display 25, which is a HUD. In other words, the display 25 displays the capture range of the exterior camera 29.

[0033] In this embodiment, the display 25 is provided as a part of the agent system 15. However, instead of providing the display 25 in the agent system 15, or in addition to the display 25 of the agent system 15, a display may be provided in the vehicle 100 or the car navigation system 12. In this case, the agent system 15 can display the agent unit 23, etc. on the display provided in the vehicle 100 or the car navigation system 12.

[0034] The recording device 26 is a storage device that temporarily or permanently records videos and the like that become memories of travel, etc. The recording device 26 is configured by a memory, a hard disk drive, a solid state drive, cloud storage provided on the web, or the like.

[0035] In this embodiment, the agent system 15 configured as described above is configured substantially as one unit with the vehicle 100. Therefore, some or all of the functions of the agent controller 24 may be configured by the vehicle controller 16. In this case, the vehicle controller 16 functions as the agent controller 24. The agent system 15 may also be configured by a separate device independent of the vehicle 100. For example, when a device such as a smartphone owned by the passenger 10 can be linked with the vehicle 100, the agent system 15 may be implemented in the device owned by the passenger 10.

[0036] The vehicle controller 16 is a control unit that comprehensively controls the operation of the vehicle 100. The vehicle controller 16 controls the electric motor and / or engine that drive the vehicle 100, as well as the brakes, etc., based on input from the driving operation unit 11. The vehicle controller 16 is configured, for example, by one or more computers in which the operation of each unit that constitutes the vehicle 100 is programmed. The vehicle controller 16 can appropriately refer to signals output by the sensors 17, etc., in order to control the vehicle 100. Furthermore, the vehicle controller 16 acquires operations on the vehicle 100 and operations on various devices, etc., mounted on the vehicle 100. In addition, the vehicle controller 16 acquires parameters necessary for controlling the vehicle 100 or various devices, etc., mounted on the vehicle 100, by calculations using signals, etc., acquired from the sensors 17.

[0037] The sensor 17 is one or more sensors that measure the conditions of each part of the vehicle 100 and the conditions inside or outside the vehicle 100. The sensor 17 is, for example, a speed sensor, an acceleration sensor, an in-vehicle camera 27, an in-vehicle microphone 28, an outside-vehicle camera 29, an outside microphone, and / or a position sensor 30. In this embodiment, the sensor 17 includes at least the in-vehicle camera 27, the in-vehicle microphone 28, the outside-vehicle camera 29, and the position sensor 30.

[0038] The in-vehicle camera 27 is one or more imaging devices that capture images of the interior of the vehicle 100 and obtain still or video images thereof. In this embodiment, the in-vehicle camera 27 captures at least the passenger 10. Hereinafter, a still or video image of the passenger 10 captured by the in-vehicle camera 27 is referred to as an in-vehicle image 31 (see FIG. 2). The in-vehicle image 31 can detect the line of sight, position and orientation of the face, body, etc. of the passenger 10, as well as changes therein.

[0039] The in-vehicle camera 27 can appropriately acquire in-vehicle images 31 in response to a request from the agent system 15 and / or the vehicle controller 16. In this embodiment, the in-vehicle camera 27 acquires the in-vehicle images 31 intermittently or continuously in response to a request from the agent system 15. The in-vehicle images 31 are then temporarily stored and used by the agent system 15 to detect the line of sight, the position and orientation of the face, body, etc. of the passenger 10, as well as any changes therein.

[0040] The in-vehicle microphone 28 is one or more sound collection devices that capture sounds or voices generated inside the vehicle 100. In this embodiment, the in-vehicle microphone 28 captures at least the voices uttered by the passengers 10. Hereinafter, the voices captured by the in-vehicle microphone 28 will be referred to as in-vehicle voice 32 (see FIG. 2). The in-vehicle voice 32 can capture the murmurings (monologues) of the passengers 10, conversations between multiple passengers 10, and voices and the like resulting from dialogue between the passengers 10 and the agent unit 23 (hereinafter simply referred to as "conversations").

[0041] The in-vehicle microphone 28 can appropriately acquire the in-vehicle voice 32 in response to a request from the agent system 15 and / or the vehicle controller 16. In this embodiment, the in-vehicle microphone 28 acquires the in-vehicle voice 32 intermittently or continuously in response to a request from the agent system 15. The in-vehicle voice 32 is then temporarily stored and used by the agent system 15 to acquire or detect the conversation of the passenger 10.

[0042] The exterior camera 29 is one or more imaging devices that capture images of the outside of the vehicle 100 and obtain still images or videos thereof. In this embodiment, the exterior camera 29 captures images of at least the area ahead of the vehicle 100. Hereinafter, the still images or videos captured by the exterior camera 29 will be referred to as exterior image 33 (see FIG. 2). From the exterior image 33, it is possible to identify an object that the passenger 10 is interested in (hereinafter referred to as an object of interest) based on a change in the passenger 10's line of sight, etc.

[0043] The exterior camera 29 can appropriately acquire exterior images 33 in response to a request from the agent system 15 and / or the vehicle controller 16, etc. In this embodiment, the exterior camera 29 acquires exterior images 33 intermittently or continuously in response to a request from the agent system 15. The exterior images 33 acquired intermittently or continuously in this manner are, in principle, temporarily stored and used to identify objects of interest to the passenger 10, etc. The exterior images 33 may also be recorded in the recording device 26 to be provided to the passenger 10 as memories of the trip, etc. Whether the exterior images 33 acquired at any timing are recorded as memories of the trip, etc. is determined by the agent system 15.

[0044] The position sensor 30 is a GPS (Global Positioning System) sensor or other GNN (Global Navigation Satellite System) sensor, and measures the position of the vehicle 100. The position, orientation, and traveling direction of the vehicle 100 can be identified based on the position information and changes therein output by the position sensor 30. The position information output by the position sensor 30 is used by the car navigation system 12 for route guidance and the like. In addition, in this embodiment, the agent system 15 uses the position information output by the position sensor 30 when identifying landmarks in the vicinity of the vehicle 100 based on the position, orientation, etc. of the vehicle 100.

[0045] Fig. 2 is a block diagram showing the configuration of the agent controller 24. As shown in Fig. 2, the agent controller 24 includes a passenger detection unit 41, an interest target identification unit 42, an interest target recording unit 43, an interface control unit 44, an interest loss determination unit 45, and a shared recognition determination unit 46.

[0046] The passenger detection unit 41 detects the passenger 10 using the in-vehicle image 31 and / or the in-vehicle sound 32. The detection of the passenger 10 means detecting the partial or overall behavior of the passenger 10, and includes the movement of the passenger 10 itself as well as the sound emitted by the passenger 10.

[0047] Specifically, the passenger detection unit 41 includes a movement detection unit 51 and a conversation detection unit 52.

[0048] The movement detection unit 51 detects the movement of the passenger 10 using the in-vehicle image 31. That is, the passenger detection unit 41 detects the gaze of the passenger 10, the position and orientation of the face and / or other body parts of the passenger 10, the facial expression of the passenger 10, and changes therein. The gaze of the passenger 10 is detected based on the movement of the eyeballs of the passenger 10, for example. Furthermore, the facial expression of the passenger 10 is detected based on the movement of parts that make up the face of the passenger 10, such as the eyes, nose, mouth, eyebrows, and cheeks of the passenger 10. The movements (including gestures) of the face and other body parts of the passenger 10 are detected based on changes in their external appearance, etc.

[0049] The conversation detection unit 52 detects the conversation of the passenger 10 and its transitions (changes) using the in-vehicle voice 32. For example, the conversation detection unit 52 detects the conversation of the passenger 10 by detecting predetermined keywords and / or phrases, etc. from the voice (voice data) of the passenger 10 acquired as the in-vehicle voice 32. The conversation detection unit 52 also detects the topic of the conversation and its transitions based on the classification and frequency of appearance of the keywords and phrases. The topic here refers to the theme or subject of the conversation, etc.

[0050] The subject of interest identification unit 42 identifies a subject to which the passenger 10 is paying attention and / or showing interest (hereinafter referred to as a subject of interest) based on the outside-vehicle image 33 and the detection result of the passenger 10 by the passenger detection unit 41. That is, the subject of interest identification unit 42 identifies the subject of interest of the passenger 10 from landmarks, other structures, natural objects, etc. outside the vehicle 100 based on changes in the passenger's line of sight, conversation, etc. Furthermore, the subject of interest identification unit 42 acquires information, etc. related to the identified subject of interest.

[0051] Specifically, the subject of interest identification unit 42 includes an object detection unit 53, an interest level estimation unit 54, and a related information acquisition unit 55.

[0052] The object detection unit 53 detects one or more objects that may be of interest, such as landmarks, other structures, or natural objects, from the outside-vehicle image 33. The object detection unit 53 can detect various objects, including objects other than predetermined landmarks. However, for simplicity in this embodiment, the object detection unit 53 detects landmarks. Specifically, the object detection unit 53 acquires landmark information 22 based on the position information of the vehicle 100, and detects landmarks that are visible to the passenger 10 from the current position of the vehicle 100 as objects that may be of interest.

[0053] The interest level estimation unit 54 estimates the level of interest in the object detected by the object detection unit 53 based on the gaze, face or other body parts, and topic of conversation of the passenger 10, or by comprehensively combining these elements through a predetermined calculation or the like. The interest target identification unit 42 determines an object of interest of the passenger 10 from one or more detected objects based on the level of interest estimated by the interest level estimation unit 54. More specifically, when the level of interest in a certain object is greater than a predetermined threshold, the interest target identification unit 42 identifies the object as the interest target of the passenger 10. That is, in this embodiment, an object with a level of interest greater than the threshold is recognized as the interest target of the passenger 10.

[0054] In this embodiment, the interest level is an evaluation value representing the degree of interest of the passenger 10 in an object detected by the object detection unit 53. For example, the time the passenger 10's gaze shifts on a specific object or the time the passenger 10's face or other body parts are directed toward an object detected by the object detection unit 53 represents the interest level. Furthermore, the degree of agreement or relevance with a conversation topic represents the interest level. In principle, the interest level estimation unit 54 estimates the interest level in a specific object by performing a calculation using the time the passenger 10's gaze, face, etc. is directed, the relevance with a conversation topic, etc. For simplicity, in this embodiment, the interest level estimation unit 54 compares the time the passenger 10's gaze, or the direction of the passenger 10's face or body, continuously or intermittently shifts on a specific object with a predetermined time. The interest level estimation unit 54 then determines an object on which the passenger 10's gaze, etc. is directed for a predetermined period of time or longer as the passenger's target of interest.

[0055] When an object of interest is identified, the related information acquisition unit 55 acquires information related to the object of interest (hereinafter referred to as related information). For example, when the object of interest is a mountain, the related information includes the name and altitude of the mountain. Also, when the object of interest is a bridge, the related information includes the name, length, and year of construction of the bridge. The related information acquisition unit 55 acquires the related information from the Internet or the like using the communication device 13, for example.

[0056] When a subject of interest is identified, the subject of interest recording unit 43 executes information recording assistance control to notify the passenger 10 that the subject of interest has been identified by the agent unit 23, and to automatically record the vehicle exterior image 33 of the identified subject of interest taken by the exterior camera 29 in the recording device 26. That is, the subject of interest recording unit 43 automatically records the vehicle exterior image 33 in which the identified subject of interest is captured, in principle, but does not record it silently to the passenger 10, but notifies the passenger 10 by the agent unit 23 that the subject of interest has been identified. This allows the passenger 10 to know that the agent system 15 has identified the subject of interest and that the vehicle exterior image 33 in which the subject of interest is captured is being automatically recorded.

[0057] As described above, at least informing the passenger 10 that a subject of interest has been identified is the minimum condition for the subject of interest recording unit 43 to automatically record the outside-vehicle image 33 that shows the subject of interest. Furthermore, in this embodiment, the subject of interest recording unit 43 further sets the condition for starting recording as the passenger 10's recognition that the agent system 15 has identified the subject of interest and will automatically record it.

[0058] Furthermore, when the identified subject of interest is recorded as a video, the subject of interest recording unit 43 ends the recording when the passenger 10 loses interest in it. When the identified subject of interest is recorded as a still image, the recording ends almost simultaneously with the start of recording. Therefore, regardless of the passenger 10 losing interest as described above, the recording ends automatically.

[0059] In addition, the object of interest recording unit 43 can automatically transmit information such as the recorded outside-vehicle image 33 to an external information device linked to the vehicle 100 or the agent system 15 using the communication device 13. The external information device here refers to a device capable of displaying or playing back the outside-vehicle image 33, such as a device such as a smartphone owned by the passenger 10, or cloud storage accessible from a device owned by the passenger 10. When a device such as the car navigation system 12 installed in the vehicle 100 is capable of displaying or playing back the outside-vehicle image 33, such a device is also included in the external information device here. In order to enable the outside-vehicle image 33 to be displayed or played back without relying on the vehicle 100 or the agent system 15, the object of interest recording unit 43 transmits the recorded outside-vehicle image 33 to the external information device, for example, when or at the time the passenger 10 gets off the vehicle 100.

[0060] The interface control unit 44 controls the interface of the agent system 15 for the passenger 10, thereby providing the passenger 10 with the assistance function of the agent system 15 in a natural manner. Specifically, the interface control unit 44 comprehensively controls various operations to be performed by the agent unit 23, such as the display, movements, facial expressions, and voice output of the agent unit 23, as well as the display of the display 25. In particular, in this embodiment, the interface control unit 44 acquires information related to the position of the identified target of interest, for example, from the target of interest identification unit 42, in response to a command or notification from the target of interest recording unit 43. Then, the interface control unit 44 causes the agent unit 23 to perform a target of interest display operation, such as displaying the identified target of interest as the target of interest of the agent unit 23, and / or highlights the identified target of interest in the display 25.

[0061] The interface control unit 44 includes a first operation control unit 56, a second operation control unit 57, a third operation control unit 58, and a highlighting control unit 59.

[0062] The first operation control unit 56 causes the agent unit 23 to perform a target of interest display operation of displaying a target of interest of the agent unit 23, by a first operation that operates at least a part of the agent unit 23. In the present embodiment, the first operation control unit 56 causes the agent unit 23 to perform a first operation of displaying the identified target of interest as a target of interest of the agent unit 23. That is, when a target of interest is identified and the target of interest recording unit 43 automatically records the target of interest, the first operation control unit 56 causes the agent unit 23 to execute the first operation. As a result, the fact that the target of interest of the passenger 10 has been shared with the agent unit 23 is notified to the passenger 10, for example, by a gesture or the like of the agent unit 23.

[0063] The second operation control unit 57 causes the agent unit 23 to perform an attention target display operation of displaying an attention target of the agent unit 23 by a second operation of changing the color and / or pattern of the agent unit 23. In this embodiment, the second operation control unit 57 causes the agent unit 23 to perform a second operation of displaying the identified interest target as an attention target of the agent unit 23. That is, at least when an interest target is identified and the interest target recording unit 43 automatically records the interest target, the second operation control unit 57 causes the agent unit 23 to perform the second operation. As a result, the fact that the interest target of the passenger 10 has been shared with the agent unit 23 is notified to the passenger 10 by, for example, the facial expression of the agent unit 23.

[0064] The third operation control unit 58 causes the agent unit 23 to perform a target of interest display operation of displaying the target of interest of the agent unit 23 by a third operation of outputting sound, voice, or the like from the agent unit 23. In the present embodiment, the third operation control unit 58 causes the agent unit 23 to perform the third operation of displaying the identified target of interest as the target of interest of the agent unit 23. That is, at least when the target of interest is identified and the target of interest recording unit 43 automatically records the target of interest, the third operation control unit 58 causes the agent unit 23 to execute the third operation. As a result, the fact that the target of interest of the passenger 10 has been shared with the agent unit 23 is notified to the passenger 10, for example, by a direct or indirect (implicit) voice message or the like issued by the agent unit 23.

[0065] Furthermore, when the subject of interest recording unit 43 starts recording the subject of interest, the third operation control unit 58 can notify the passenger 10 of this by causing the agent unit 23 to perform the third operation. Similarly, when the subject of interest recording unit 43 finishes recording the subject of interest, the third operation control unit 58 can notify the passenger 10 of this by causing the agent unit 23 to perform the third operation. In this embodiment, when the subject of interest recording unit 43 starts and finishes recording the subject of interest, the third operation control unit 58 notifies the passenger 10 of this by causing the agent unit 23 to output a voice message.

[0066] In addition, after the subject of interest recording unit 43 starts recording the subject of interest, the third operation control unit 58 can cause the agent unit 23 to perform the third operation, thereby providing the passenger 10 with the related information acquired by the related information acquisition unit 55. In this embodiment, the third operation control unit 58 provides the related information as described above by the agent unit 23.

[0067] The highlighting control unit 59 adjusts the display format on the display 25 to highlight part or all of the display on the display 25. In this embodiment, the highlighting control unit 59 highlights the identified subject of interest on the display 25. That is, at least when a subject of interest is identified and the subject of interest recording unit 43 automatically records the subject of interest, the highlighting control unit 59 highlights the identified subject of interest, for example, by surrounding the subject of interest with a frame. As a result, the subject of interest of the passenger 10 identified by the agent unit 23 or the agent system 15 is notified to the passenger 10 by the display on the display 25.

[0068] In this embodiment, as described above, the interface control unit 44 executes the first, second, and third operations of the agent unit 23 in combination with highlighting on the display 25. However, the interface control unit 44 can display that the agent unit 23 or the agent system 15 has identified the subject of interest of the passenger 10 by performing one or any combination of these operations.

[0069] The interest loss determination unit 45 continuously monitors the detection result by the occupant detection unit 41 and / or the subject of interest identified by the subject of interest identification unit 42 to determine whether the occupant 10 has lost interest in the subject of interest identified by the subject of interest identification unit 42. In this embodiment, the result of the determination by the interest loss determination unit 45 (hereinafter referred to as interest loss determination) constitutes one of the triggers for the subject of interest recording unit 43 to end recording of the vehicle outside images 33 for the subject of interest. In other words, when the interest loss determination unit 45 determines that the occupant 10 has lost interest in the identified subject of interest, the subject of interest recording unit 43 ends continuous recording of the vehicle outside images 33 for that subject of interest.

[0070] The interest loss determination unit 45 can determine that the passenger 10 has lost interest in the identified object of interest, for example, when the topic (content) of the conversation by the passenger 10 shifts from a topic related to the object of interest to a topic unrelated to the object of interest. The interest loss determination unit 45 can also determine that the passenger 10 has lost interest in the previous object of interest when the object of interest of the passenger 10 changes, for example, when a new object of interest is identified. In this embodiment, it is determined that the passenger 10 has lost interest in the object of interest when the topic of the conversation shifts and the passenger 10 is no longer looking at the object of interest, the interest level decreases, and the object can no longer be considered an object of interest. As described above, the topic of the conversation by the passenger 10 is estimated, for example, by the conversation detection unit 52 based on keywords and phrases appearing in the conversation, their classification, frequency of appearance, etc.

[0071] Here, the subject of interest recording unit 43 automatically ends recording of the subject of interest based on the result of the interest loss determination, but this is not limited to this. For example, after starting recording of the identified subject of interest, the subject of interest recording unit 43 can end recording of the subject of interest when a predetermined period of time has elapsed without making a loss of interest determination, or regardless of the result of the interest loss determination. In this embodiment, the subject of interest recording unit 43 ends recording of the subject of interest when the passenger 10 loses interest in the subject of interest and a certain period of time has elapsed since the start of recording. Also, when the subject of interest recording unit 43 records the subject of interest using still images based on the settings of the passenger 10, the recording essentially ends automatically at the same time as it starts. In this embodiment, the subject of interest recording unit 43 records the identified subject of interest using video.

[0072] The shared recognition determination unit 46 monitors the detection result by the passenger detection unit 41 after the passenger 10 is notified that an object of interest has been identified by at least the agent unit 23 and / or the display 25. The shared recognition determination unit 46 then determines whether the passenger 10 recognizes that the agent unit 23 or the agent system 15 has identified the object of interest of the passenger 10. This determination by the shared recognition determination unit 46 (hereinafter referred to as shared recognition determination) is synonymous with determining whether the passenger 10 recognizes that the object of interest of the passenger 10 has been shared with the agent unit 23 or the agent system 15. Furthermore, the passenger 10 is usually aware of the information recording support function of the agent system 15. Therefore, the shared recognition determination is essentially synonymous with determining whether the passenger 10 agrees or disagrees with the agent system 15 automatically recording information about the object of interest.

[0073] In this embodiment, the result of the shared recognition determination constitutes one of the triggers for the object of interest recording unit 43 to start recording the outside-vehicle image 33 of the object of interest. That is, in this embodiment, when the object of interest is identified and the occupant 10 recognizes that the object of interest has been shared with the agent unit 23 or the agent system 15, the object of interest recording unit 43 starts recording the outside-vehicle image 33 of the identified object of interest.

[0074] The following describes the operation of automatic information recording support in the vehicle 100 equipped with the agent system 15 configured as described above. The program that realizes the processing for information recording support shown in the following Figures 3 to 5 is the information recording support program.

[0075] FIG. 3 is a flowchart for starting automatic information recording. As shown in FIG. 3, in step S101, the behavior of the occupant 10, i.e., the occupant's 10 gaze, facial or other body movements, facial expressions, and / or conversation, is detected. In step S102, an object that may be of interest to the occupant 10 is detected. In this embodiment, landmarks that the occupant 10 can see are detected as objects of potential interest. Then, in step S103, a level of interest in the detected object is estimated based on the detected behavior of the occupant 10, and the estimated level of interest is compared with a threshold. If the level of interest is greater than the threshold, the object is recognized as an object of interest to the occupant 10. Note that when an object of interest to the occupant 10 does not occur, the agent system 15 continues to monitor the behavior of the occupant 10 without arbitrarily recording the outside-vehicle image 33. On the other hand, when an object of interest to the occupant 10 is identified, the agent system 15 assists in recording the object of interest.

[0076] When an object of interest having a degree of interest greater than the threshold is identified, in step S104, the agent unit 23 performs an object of interest display operation, such as displaying the object of interest of the passenger 10 as an object of interest of the agent unit 23. In addition, in step S105, the identified object of interest is highlighted in the display 25. These operations notify the passenger 10 that the object of interest of the passenger 10 has been identified by the agent unit 23, etc. As a result, the passenger 10 can recognize that the object of interest has been shared with the agent unit 23, etc.

[0077] Thereafter, in step S107, a shared recognition determination is performed. In this embodiment, if the result of the shared recognition determination is that the passenger 10 does not recognize that the subject of interest is shared with the agent unit 23, etc., the recording of the outside-vehicle image 33 for that subject of interest is stopped. In this case, the agent system 15 does not start recording for that subject of interest, and ends the information recording support for that subject of interest.

[0078] On the other hand, if it is determined in the shared recognition determination that the passenger 10 recognizes that the subject of interest is shared with the agent unit 23, etc., recording of the outside-vehicle image 33 for the subject of interest is started in step S108. If the passenger 10 recognizes that the subject of interest is shared, it is considered that the passenger 10 has essentially recognized that the subject of interest will be automatically recorded, that is, that substantial consent to recording based on understanding of the functions of the agent system 15 has been obtained, and recording of the subject of interest is started.

[0079] When recording of the subject of interest starts, in step S109, the start of recording is interactively notified by a voice message from the agent section 23. Furthermore, if there is related information about the subject of interest, in step S110, the related information is provided by a voice message from the agent section 23.

[0080] Fig. 4 is a flowchart for terminating automatic information recording. As shown in Fig. 4, when recording of an interest subject is started, a determination of loss of interest is made in step S201. If the result of the determination of loss of interest shows that the passenger 10 has not lost interest in the interest subject, i.e., if interest in the recorded interest subject continues, then, in principle, recording of the interest subject continues, as shown in step S202.

[0081] On the other hand, when it is determined in the interest loss determination that the passenger 10 has lost interest in the subject of interest, it is determined in step S203 whether a certain time has passed since the start of recording of the subject of interest. Then, when the passenger 10 has lost interest in the recorded subject of interest and the certain time has passed, recording of the subject of interest is terminated in step S204. Furthermore, when recording of the subject of interest is terminated, the agent unit 23 interactively notifies the user of the end of recording by a voice message in step S205.

[0082] FIG. 5 is a flowchart showing a specific example of interest loss determination. As shown in FIG. 5, interest loss determination is performed, for example, in step S301, after recording of the subject of interest has started, based on the detection results of the conversation detection unit 52, which is continuously monitored, to determine whether the passenger 10 is having a conversation. If the passenger 10 is having a conversation, in step S302, it is further determined whether a conversation related to the subject of interest is continuing with respect to the topic of the conversation. If a topic related to the subject of interest is continuing, it can be assumed that the passenger 10 has not lost interest in the subject of interest for which recording has started, i.e., that the passenger 10 is maintaining interest in the subject of interest. Therefore, the process proceeds to step S202, and recording of the subject of interest continues.

[0083] On the other hand, if it is determined in step S301 that the passenger 10 is not having a conversation, it can be assumed that the passenger 10 has already lost interest in the subject of interest for which recording has started. Similarly, if the passenger 10 is having a conversation but it is determined in step S302 that the topic of the conversation has shifted to a topic unrelated to the subject of interest, it can be assumed that the passenger 10 has already lost interest in the subject of interest for which recording has started. Therefore, in these cases, it is further confirmed whether the passenger 10 has lost interest in the subject of interest for which recording has started, based on the detection result of the action detection unit 51 and / or the interest level estimated by the interest level estimation unit 54, etc.

[0084] Here, in step S303, after recording of the object of interest begins, it is determined whether the passenger 10 is continuously or intermittently looking at the object of interest based on changes in the passenger 10's line of sight, which are continuously monitored. Even if the passenger 10 is not having a conversation or the topic has already shifted to an unrelated topic, if the passenger 10 is continuously or intermittently gazing at the object of interest, it can be assumed that the passenger 10 has not lost interest in the object of interest for which recording began. Therefore, if it is determined in step S303 that the passenger 10 is gazing at the object of interest, the process proceeds to step S203, and after a certain period of time has elapsed, recording of the object of interest is terminated. On the other hand, if the passenger 10 is not having a conversation or the topic has already shifted to an unrelated topic, and it is further determined that the passenger 10 is not gazing at the object of interest, it can be assumed that the passenger 10 has lost interest in the object of interest for which recording began. Therefore, when it is determined in step S303 that the passenger 10 is not viewing the object of interest, the process proceeds to step S204, and the recording of the object of interest is terminated. In this way, the determination of loss of interest is made based on a comprehensive evaluation according to, for example, whether or not there is conversation, a change in topic, and whether or not the object of interest is viewed.

[0085] A more specific example of application of information recording support will be described below.

[0086] Fig. 6 is an explanatory diagram of a scene in which automatic information recording starts. Fig. 6(A) is an explanatory diagram of a scene S1 in which an object of interest of the passenger 10 is identified. Fig. 6(B) is an explanatory diagram of a scene S2 in which the agent unit 23 performs an object of interest display operation for the identified object of interest. Fig. 6(C) is an explanatory diagram of a scene S3 in which a shared recognition determination is made and recording of the object of interest actually starts. Fig. 6(D) is an explanatory diagram of a scene S4 in which the start of recording is notified.

[0087] As shown in FIG. 6(A), assume that a mountain 62, which is a landmark, is in the passenger's field of view 61. If the passenger 10 visually recognizes the mountain 62 for a predetermined period of time or longer, the agent system 15 identifies the mountain 62 as an object of interest to the passenger 10. In this case, as shown in FIG. 6(B), for example, the agent unit 23 faces the direction of the mountain 62, directs its gaze toward the mountain 62, and issues a voice message suggesting recognition of the mountain 62, such as "It's beautiful." In addition, a frame 63 or the like surrounding the mountain 62 is displayed in the display area of ​​the display 25 superimposed on the passenger's field of view, and the mountain 62 is emphasized as an identified object of interest. Therefore, the passenger 10 recognizes that the object of interest to the passenger 10 has been identified by these actions of the agent unit 23 and the highlighting on the display 25. The mountain 62 is then shared as a common object of interest between the passenger 10 and the agent unit 23. As a result, the mountain 62 becomes a memory shared by the passenger 10 and the agent section 23. Therefore, even if an outside image 33 showing the mountain 62 is later provided to the passenger 10, the passenger 10 will be able to naturally and comfortably accept it as a shared memory, without regarding it as a record made arbitrarily by the agent system 15.

[0088] When the passenger 10 responds to a voice message from the agent unit 23, such as "It's beautiful," by saying, for example, "Yeah, that's right," thereby indicating recognition of the shared interest, as shown in Fig. 6(C), recording (video recording) of the mountain 62, which has become the shared interest, begins. Then, as shown in Fig. 6(D), the agent unit 23 issues a voice message, such as "I'll remember," that directly or indirectly indicates the start of recording, thereby notifying the passenger 10 of the start of recording. This allows the passenger 10 to particularly reliably recognize that recording will be made of the mountain 62, which is the shared interest.

[0089] Figure 7 is an explanatory diagram of scenes during automatic information recording. Figure 7(A) is an explanatory diagram of scene S5 in which the agent unit 23 and the passenger 10 have a conversation. Figure 7(B) is an explanatory diagram of scene S6 in which related information is provided.

[0090] As shown in FIG. 7(A), after recording of mountain 62, which is a shared object of interest, begins, if passenger 10 says, for example, "I'd like to climb mountain 62," then, as shown in FIG. 7(B), agent unit 23 will interactively provide information related to mountain 62 by issuing a voice message, for example, "That mountain is called Mt. XX, and it's XXX meters above sea level." This leaves a strong impression on passenger 10 that mountain 62 is a shared object of interest. As a result, passenger 10 can naturally and comfortably accept outside-vehicle image 33, which includes mountain 62, as a shared memory.

[0091] Fig. 8 is an explanatory diagram of a scene in which automatic information recording ends. Scene S7 shown in Fig. 8 is a scene in which the agent section 23 notifies the passenger 10 that recording has ended.

[0092] As shown in Figure 8, when the passenger 10 loses interest in the object of interest, including cases where interest can be considered to have been lost passively, such as when the mountain 62, which is the object of interest, moves out of the field of view 61, recording is terminated. Furthermore, when recording ends, the agent unit 23 issues a voice message indicating the end of recording directly or indirectly, such as "I remembered," to notify the passenger 10 of the end of recording. This allows the passenger 10 to particularly reliably recognize that recording of the mountain 62, which is the shared object of interest, has ended.

[0093] As described above, the information recording assistance method according to this embodiment is an information recording assistance method using a vehicle 100 equipped with an exterior camera 29 that captures images outside the vehicle, and an agent system 15 that is an information recording assistance device that has an exterior camera 29 that captures images outside the vehicle, an agent unit 23 that functions as an interface for the passenger 10, and supports information recording at any timing using the exterior camera 29 via the agent unit 23. In this information recording assistance method, the movements and / or conversation of the passenger 10 are detected, and an object of interest of the passenger 10 that is outside the vehicle 100 is identified based on the movements and / or conversation of the passenger 10. Then, when the object of interest is identified, the agent system 15 notifies the passenger 10 that the object of interest has been identified, and the object of interest is recorded by the exterior camera 29.

[0094] As described above, in the information recording assistance method according to this embodiment, when the passenger 10 is assisted in recording information (such as a video) about an interest of the passenger 10 at any timing when the interest of the passenger 10 arises, the passenger 10 is notified that the interest of the passenger 10 has been identified. If the agent unit 23 records the information without any reaction when the interest of the passenger 10 arises, the passenger 10 may feel uncomfortable because he or she is not aware that the recorded information is being provided as a memory. In contrast, the information recording assistance method according to this embodiment notifies the passenger 10 that the interest of the passenger 10 has been identified, as described above, so that the passenger 10 can recognize that the interest is shared with the agent unit 23 or the agent system 15. Therefore, the passenger 10 can recognize the recorded information about the interest as a shared memory with the agent unit 23 or the agent system 15. As a result, the information recording assistance method according to this embodiment can naturally assist the passenger 10 in recording information without causing any discomfort to the passenger 10.

[0095] Furthermore, if the agent unit 23 records information about an object of interest of the passenger 10 without any reaction when the object of interest arises, the passenger 10's trust in the agent unit 23 or the agent system 15 may decrease. For example, if the agent unit 23 records an object of interest without any reaction and provides the record, the passenger 10 may perceive that the passenger's private information has been recorded without the passenger's knowledge or consent. In other words, unresponsive recording reduces transparency about the recording and the recorded information, and the agent unit 23 or the agent system 15, which is the target of interactive communication, may lose the passenger 10's trust. In contrast, in the information recording assistance method according to this embodiment, as described above, when the object of interest is recorded, the passenger 10 is notified that the object of interest has been identified, thereby ensuring transparency about the recording of information and the object of interest that is the target of the recording. Therefore, the information recording assistance method according to this embodiment also has the secondary effect of maintaining or improving the passenger 10's trust in the agent unit 23 or the agent system 15, which is the target of interactive communication.

[0096] The information recording assistance method according to this embodiment has the following advantageous effects in relation to a specific mode of notifying the passenger 10 that an object of interest has been identified.

[0097] In particular, the information recording assistance method according to this embodiment notifies the passenger 10 that the object of interest has been identified by performing an attention object display operation that indicates that the attention of the agent section 23 has been directed to the object of interest by changing the form of the agent section 23. This makes it possible to particularly effectively make the passenger 10 aware of and impress upon the passenger 10 that the object of interest has been shared between the agent section 23 or agent system 15 and the passenger 10. As a result, the information assistance method according to this embodiment can particularly naturally assist in recording the object of interest.

[0098] In the information recording assistance method according to this embodiment, the attention object display operation is performed by a first operation of operating at least a part of the agent unit 23 toward the object of interest. This makes it easier for the passenger 10 to almost unconsciously recognize that the object of interest has been shared between the agent unit 23 or agent system 15 and the passenger 10, leaving a strong impression. As a result, the information assistance method according to this embodiment can provide particularly natural assistance in recording the object of interest.

[0099] In the information recording assistance method according to this embodiment, the action of displaying the object of interest is performed by a second action of changing the color and / or pattern of the agent unit 23. This causes feelings or emotions to be shared between the agent unit 23 or agent system 15 and the passenger 10 regarding the sharing of the object of interest. Therefore, the information recording assistance method according to this embodiment is particularly likely to impress upon the passenger 10 that the object of interest has been shared. As a result, the information assistance method according to this embodiment can particularly naturally assist in recording the object of interest.

[0100] In this embodiment, the agent system 15, which is an information recording assistance device, includes a display 25 that displays the field of view of the exterior camera 29. In the information recording assistance method according to this embodiment, the passenger 10 is notified that the subject of interest has been identified by highlighting the subject of interest on the display 25. This notifies the passenger 10 in a particularly easy-to-understand and intuitive manner that the subject of interest of the agent unit 23 and the agent system 15 is the same as the subject of interest of the passenger 10. Therefore, the passenger 10 can almost certainly recognize that the agent unit 23 and the agent system 15 share the subject of interest, without oversight or the like. As a result, the information assistance method according to this embodiment can naturally assist the passenger 10 in recording the subject of interest, without the passenger 10 realizing that the subject of interest has been recorded.

[0101] The information recording assistance method according to this embodiment provides the following advantageous effects in terms of specific aspects for identifying the subject of interest of the passenger 10.

[0102] In this embodiment, the vehicle 100 is equipped with an in-vehicle camera 27 that captures images of the passenger 10, and in the information recording assistance method according to this embodiment, the movements of the passenger 10 are detected using the in-vehicle camera 27, and information recording assistance is performed based on the detected movements of the passenger 10. The movements of the passenger 10, including when the passenger 10 is unconscious, clearly indicate the interests of the passenger 10. Therefore, according to the information recording assistance method according to this embodiment, the interests of the passenger 10 are accurately recognized, and therefore recording of the interests can be supported in a timely and accurate manner.

[0103] In particular, in the information recording assistance method according to this embodiment, a change in the passenger's line of sight or a movement of the passenger's face or body is detected as the movement of the passenger 10. These movements of the passenger 10 tend to particularly clearly indicate the emergence of an object of interest. Therefore, according to the information recording assistance method according to this embodiment, the object of interest of the passenger 10 is particularly accurately recognized, and therefore recording of the object of interest can be supported in a timely and accurate manner.

[0104] Furthermore, in the information recording assistance method according to this embodiment, an object that the passenger 10's gaze or the passenger 10's face or body direction is directed toward for a predetermined period of time or more is identified as an object of interest. This prevents objects that are temporarily and accidentally viewed from the outside from being recognized as an object of interest, and increases the accuracy with which the agent system 15 recognizes objects of interest. Therefore, according to the information recording assistance method according to this embodiment, the object of interest of the passenger 10 is recognized particularly accurately, and recording of objects of interest can be supported in a timely and accurate manner.

[0105] Furthermore, in this embodiment, the vehicle 100 is equipped with an in-vehicle microphone 28 that captures audio inside the vehicle, and in the information recording assistance method according to this embodiment, the conversation of the passenger 10 is detected using the in-vehicle microphone 28, and information recording assistance is performed based on the detected conversation of the passenger 10. As a result, even in situations where the passenger 10 must keep an eye on traffic conditions, etc., the passenger's subject of interest can be accurately identified and information about that subject of interest can be recorded. In such cases, the need for information recording assistance is particularly high. Therefore, according to the information recording assistance method according to this embodiment, the passenger's subject of interest can be accurately recognized, and recording of the subject of interest can be supported in a timely and accurate manner.

[0106] In particular, the information recording assistance method according to this embodiment estimates the topic of conversation between the passengers 10 and identifies the subject of interest based on that topic. In this way, the topic of conversation between the passengers 10 clearly represents the current subject of interest of the passengers 10. Therefore, when an object based on the topic is identified as the subject of interest, the agent system 15 is more likely to recognize the subject of interest. Therefore, the information recording assistance method according to this embodiment can particularly accurately recognize the subject of interest of the passengers 10, thereby enabling timely and accurate support for recording the subject of interest.

[0107] In addition, in the information recording assistance method according to this embodiment, landmarks in the vicinity of the vehicle 100 are extracted based on the position and orientation of the vehicle 100, and an object of interest is identified from among the extracted landmarks based on the actions and / or conversations of the passenger 10. Landmarks are objects that are particularly likely to be objects of interest. Therefore, by selecting an object of interest from landmarks in the vicinity of the vehicle 100 as described above, the object of interest can be easily and accurately identified. Therefore, the information recording assistance method according to this embodiment can accurately assist in recording an object of interest.

[0108] Furthermore, in the information recording assistance method according to this embodiment, an interest level representing the degree of interest of the passenger 10 in an object that may be of interest is estimated based on the actions and / or conversation of the passenger 10. Then, when the interest level is greater than a predetermined threshold, the object is identified as an object of interest. In this way, when an object of interest is identified according to the interest level, which is a comprehensive evaluation based on the actions and / or conversation of the passenger 10, the object of interest of the passenger 10 is identified particularly accurately. As a result, the information recording assistance method according to this embodiment can accurately assist in recording objects of interest.

[0109] The information recording assistance method according to this embodiment has the following advantageous effects with respect to specific aspects of the recording start conditions.

[0110] In the information recording assistance method according to this embodiment, it is determined whether the passenger 10 recognizes that the agent system 15, which is an information recording assistance device, has identified a subject of interest, based on the passenger 10's actions and / or conversation. Then, when the passenger 10 recognizes that the agent system 15 has identified a subject of interest, recording of the subject of interest is initiated. That is, in the information recording assistance method according to this embodiment, when the result of the shared recognition determination indicates that the passenger 10 recognizes that the subject of interest of the passenger 10 has been shared with the agent unit 23 or the agent system 15 and there is a high probability that the passenger 10 has agreed to the recording, recording of the subject of interest is assisted. Therefore, recording of the subject of interest is not performed without the passenger 10's knowledge. As a result, the information assistance method according to this embodiment can assist in recording the subject of interest in a particularly natural way.

[0111] The information recording assistance method according to this embodiment has the following advantageous effects with respect to specific aspects of the recording termination condition.

[0112] In the information recording assistance method according to this embodiment, it is determined whether the passenger 10 has lost interest in an object of interest based on the passenger's actions and / or conversation. When the passenger 10 has lost interest in the object of interest, the recording of the object of interest is terminated. That is, in the information recording assistance method according to this embodiment, when the result of the interest loss determination indicates a high probability that the passenger 10 has lost interest in the object of interest, the recording of the object of interest is terminated. Therefore, for the purpose of recording memories shared with the agent unit 23, the unnaturalness of unnecessarily long recordings being made even after interest in the object has been lost does not occur. As a result, the information assistance method according to this embodiment can support particularly natural recording of objects of interest.

[0113] Specifically, in this embodiment, the vehicle 100 is equipped with an in-vehicle microphone 28 that captures audio inside the vehicle, and in the information recording assistance method according to this embodiment, the conversation of the passenger 10 is captured using the in-vehicle microphone 28. Then, the topic of the conversation of the passenger 10 is estimated, and when the estimated topic transitions from a topic related to the subject of interest to a topic unrelated to the subject of interest, it is determined that the passenger 10 has lost interest in the subject of interest, and recording of the subject of interest is terminated. In this way, recording is appropriately terminated in response to the topic transition. As a result, the information assistance method according to this embodiment can support particularly natural recording of the subject of interest.

[0114] Furthermore, in the information recording assistance method according to this embodiment, when the subject of interest changes, it is determined that the passenger 10 has lost interest in the subject of interest, and the recording of the subject of interest is terminated. As a result, when the subject of interest changes, the recording is appropriately terminated in accordance with the change in the subject of interest. As a result, the information assistance method according to this embodiment can assist in recording the subject of interest in a particularly natural way.

[0115] In addition, in the information recording assistance method according to this embodiment, the recording of an object of interest is terminated when a predetermined time has elapsed since the start of the recording of the object of interest. This prevents the inconvenience of lengthy recording of the object when, for example, a particular object is in the field of view 61 for a long time even if the object is not of interest, such as during a traffic jam. Therefore, the information assistance method according to this embodiment can assist in the natural recording of an object of interest.

[0116] The information recording assistance method according to this embodiment has the following advantages in terms of specific aspects of the operation of the agent section 23 and the like.

[0117] In the information recording assistance method according to this embodiment, when recording of an object of interest begins, the agent unit 23 notifies the passenger 10 that recording of the object of interest has begun. This increases transparency regarding the recording of the object of interest and the object of interest that is being recorded. This improves the passenger 10's trust in the agent unit 23 or agent system 15, which is the subject of interactive communication. As a result, the information recording assistance provided by the information recording assistance method according to this embodiment is likely to be accepted by the passenger 10 as something natural.

[0118] Similarly, in the information recording assistance method according to this embodiment, when recording of the subject of interest is completed, the agent unit 23 notifies the passenger 10 that recording of the subject of interest has been completed. This increases transparency regarding the fact that recording of the subject of interest has been completed. Therefore, the passenger 10's trust in the agent unit 23 or agent system 15, which is the subject of interactive communication, increases. As a result, the information recording assistance provided by the information recording assistance method according to this embodiment is likely to be accepted by the passenger 10 as something natural.

[0119] Furthermore, in the information recording assistance method according to this embodiment, after recording of an interest begins, information related to the interest is provided to the passenger 10 by the agent unit 23. This increases the shareability of the recorded interest between the agent unit 23 and the passenger 10. As a result, the information recording assistance provided by the information recording assistance method according to this embodiment is likely to be accepted by the passenger 10 as a natural experience.

[0120] The information recording assistance method according to this embodiment provides the following advantageous effects in relation to specific aspects of the recorded matter of interest.

[0121] In the information recording assistance method according to this embodiment, records of subjects of interest are sent to an external information device that cooperates with the vehicle 100 or the agent system 15, which is an information recording assistance device. This improves the convenience of records saved as memories, for example, by making it possible to display or play back records of subjects of interest without relying on the vehicle 100 or the agent system 15.

[0122] [Variations] In the above embodiment, the result of the shared recognition determination is considered to be an indication of the passenger's 10 consent or refusal to support for recording the object of interest. However, this is not limited to this. For example, recording of the object of interest may be started when the passenger 10 gives more explicit consent to recording the object of interest. In this case, the agent unit 23 clearly notifies the passenger 10 that the object of interest will be recorded by a voice message or the like when recording starts. Then, similar to the shared recognition determination in the above embodiment, the object of interest recording unit 43 determines the passenger's 10 consent or refusal to record the object of interest based on the passenger's 10's actions and / or conversational responses, and starts recording when consent is obtained. In this way, if recording of the object of interest is started when the passenger 10 gives explicit consent, the agent system 15 is prevented from recording the object of interest against the passenger's will when the passenger 10 is interested but does not need to record it.

[0123] Furthermore, in the above embodiment, recording of an object of interest is initiated when the shared awareness determination determines that the passenger 10 recognizes that the agent unit 23 or the agent system 15 has identified the object of interest of the passenger 10. However, this is not limited to this. For example, the object of interest recording unit 43 can start recording the object of interest when the object of interest is identified. In this case, if the shared awareness determination determines that the passenger 10 does not recognize that the agent unit 23 or the agent system 15 has identified the object of interest of the passenger 10, the object of interest recording unit 43 suspends recording of the object of interest and discards any previously recorded object of interest. In this way, if recording of the object of interest is initiated when the object of interest is identified before the shared awareness determination is made, the object of interest can be appropriately recorded without delay. Furthermore, if the passenger 10 does not recognize that the object of interest has been shared and will be recorded, the previously started recording is suspended and discarded. This prevents the object of interest from being recorded without the passenger 10's knowledge and the recording from being provided to the passenger 10 as a memory.

[0124] In the above embodiment, video of the vehicle exterior image 33 is saved as a record of the subject of interest, but this is not limited to this. In addition to video of the vehicle exterior image 33, still images of the vehicle exterior image 33, the vehicle interior image 31, the vehicle interior sound 32, conversation of the passenger 10 which is part or all of the vehicle interior sound 32, or a combination thereof, etc. may be recorded as information related to the subject of interest, i.e., memories. Furthermore, when the vehicle 100 is equipped with an exterior microphone that captures sounds or sounds outside the vehicle, the sound captured by the exterior microphone may also be recorded as information related to the subject of interest.

[0125] In the above embodiment, the object of interest recording unit 43 records objects of interest as memories of the passenger 10, etc. However, in addition to recording such memories, the object of interest recording unit 43 can also record the detection results of the passenger detection unit 41, the detection and estimation results of the object of interest identification unit 42, etc. That is, the object of interest recording unit 43 can record, as necessary, the actions and conversations of the passenger 10 related to the object of interest, the degree of interest in objects that may be objects of interest, the degree of interest in the object of interest, classifications related to the type of object of interest, related information about the object of interest, search history of the car navigation system 12, etc. as a history related to the object of interest of the passenger 10. Such history information reflects the personal preferences of the passenger 10, etc. Therefore, personal tendencies of the passenger 10, such as a tendency to be interested in mountains, can be extracted from the history information. Therefore, when the object of interest recording unit 43 records the above-mentioned history information, the object of interest identification unit 42 can detect objects and / or estimate the degree of interest based on the history information. For example, when the object detection unit 53 detects an object that may be of interest, the object of interest identification unit 42 can preferentially detect objects that are likely to interest the occupant 10 based on the history information. Furthermore, when the interest level estimation unit 54 estimates the level of interest, the object of interest identification unit 42 can adjust the estimated level of interest, for example, by weighting based on the history information, so that an object that the occupant 10 is likely to monitor is highly likely to be of interest. In this way, by recording and using the history information, a more appropriate object of interest can be identified based on the individual tendencies of the occupant 10.

[0126] The embodiments and modifications of the present invention have been described above, but the configurations of the above embodiments and modifications merely illustrate some of the application examples of the present invention and are not intended to limit the technical scope of the present invention. The configurations shown in the above embodiments and modifications can be implemented in any combination, in part or in whole. Furthermore, some of the configurations shown in the above embodiments and modifications can be omitted.

[0127] For example, in the above embodiment, the agent unit 23 outputs voice messages when performing an attention target display operation, announcing the start of recording, announcing the end of recording, etc. However, this is not limiting. Depending on the specific form of the agent unit 23, the above various operations may be realized by a cry or other sound, a facial expression such as a special smile, or a light such as the lighting of an indicator.

[0128] Furthermore, each component constituting the passenger detection unit 41 is an example of a configuration that realizes the function of the passenger detection unit 41. Therefore, to the extent that the passenger detection function described in the above embodiments is realized, the movement detection unit 51 and / or the conversation detection unit 52 can be replaced with other configurations, or one of them can be omitted. The same applies to the interest target identification unit 42 and the interface control unit 44. That is, to the extent that the interest target identification function is realized, each component constituting the interest target identification unit 42 can be replaced with other configurations instead of the configurations described in the above embodiments, or some of them can be omitted. To the extent that the function of each component constituting the interface control unit 44 is realized, each component constituting the interface control unit 44 can be replaced with other configurations instead of the configurations described in the above embodiments, or some of them can be omitted. In addition, the interest loss determination unit 45 and the shared recognition determination unit 46 can be replaced with other configurations or omitted to the extent that equivalent functions in the information recording assistance can be realized. [Explanation of symbols]

[0129] 10: Passenger, 11: Driving operation unit, 12: Car navigation system, 13: Communication device, 14: Speaker, 15: Agent system (information recording support device), 16: Vehicle controller, 17: Sensor, 21: Map information, 22: Landmark information, 23: Agent unit, 24: Agent controller, 25: Display, 26: Recording device, 27: In-vehicle camera, 28: In-vehicle microphone, 29: Outside-vehicle camera, 30: Position sensor, 31: In-vehicle image, 32: In-vehicle voice, 3 3: Outside vehicle image, 41: Occupant detection unit, 42: Interest target identification unit, 43: Interest target recording unit, 44: Agent control unit, 45: Interest loss determination unit, 46: Shared recognition determination unit, 51: Action detection unit, 52: Conversation detection unit, 53: Object detection unit, 54: Interest level estimation unit, 55: Related information acquisition unit, 56: First action control unit, 57: Second action control unit, 58: Third action control unit, 59: Highlighting control unit, 61: Field of view, 62: Mountain, 63: Frame line, 100: Vehicle, 101: Information recording support system

Claims

1. An information recording support method for a vehicle including an information recording support device having an exterior camera that captures images outside the vehicle and an agent unit that functions as an interface for a passenger, and that supports information recording at any timing using the exterior camera via the agent unit, comprising: Detecting the occupant's movements and / or speech; Identifying an object of interest to the occupant outside the vehicle based on the occupant's actions and / or conversations; When the subject of interest is identified, the information recording assistance device notifies the passenger that the subject of interest has been identified; determining whether the passenger has recognized that the information recording assistance device has identified the subject of interest based on the passenger's actions and / or conversations; When the passenger recognizes that the information recording assistance device has identified the subject of interest, recording of the subject of interest is started by the exterior camera. Information recording support methods.

2. 2. The information recording support method according to claim 1, notifying the passenger that the target of interest has been identified by executing an attention target display operation that displays that the attention of the agent unit has been directed to the target of interest by changing the form of the agent unit; Information recording support methods.

3. 3. The information recording support method according to claim 2, the attention target display operation is to move at least a part of the agent unit toward the target of interest; Information recording support methods.

4. 4. The information recording support method according to claim 2 or 3, the attention target display action changes the color and / or pattern of the agent part; Information recording support methods.

5. The information recording support method according to any one of claims 1 to 4, the information recording support device includes a display that displays the photographing range of the outside-vehicle camera, notifying the passenger that the object of interest has been identified by highlighting the object of interest on the display; Information recording support methods.

6. The information recording support method according to any one of claims 1 to 5, The vehicle is equipped with an in-vehicle camera that photographs the passengers, Using the in-vehicle camera, detect a change in the passenger's line of sight or a facial or body movement of the passenger; Identifying an object to which the passenger's gaze or the direction of the passenger's face or body is directed for a predetermined period of time or more as the object of interest; Information recording support methods.

7. The information recording support method according to any one of claims 1 to 5, the vehicle is equipped with an in-vehicle microphone for capturing audio inside the vehicle; Detecting conversations between the passengers using the in-vehicle microphone; Estimating a topic of conversation between the passengers; identifying the subject of interest according to the topic; Information recording support methods.

8. The information recording support method according to any one of claims 1 to 7, extracting landmarks around the vehicle based on the position and orientation of the vehicle; Identifying the object of interest from among the extracted landmarks based on the passenger's movements and / or conversations; Information recording support methods.

9. The information recording support method according to any one of claims 1 to 8, estimating an interest level representing a degree of interest of the passenger in the object of potential interest based on the passenger's actions and / or conversations; identifying the object as the subject of interest when the interest level is greater than a predetermined threshold; Information recording support methods.

10. 2. The information recording support method according to claim 1, determining whether the passenger has lost interest in the subject of interest based on the passenger's movements and / or conversations; and terminating the recording of the object of interest when the passenger loses interest in the object of interest. Information recording support methods.

11. 11. The information recording support method according to claim 10, the vehicle is equipped with an in-vehicle microphone for capturing audio inside the vehicle; Using the in-vehicle microphone, the conversation of the passenger is acquired; Estimating a topic of conversation between the passengers; determining that the passenger has lost interest in the subject of interest when the estimated topic has transitioned from a topic related to the subject of interest to a topic unrelated to the subject of interest; Information recording support methods.

12. An information recording support device is mounted on a vehicle having an exterior camera that captures images outside the vehicle, has an agent unit that functions as an interface for a passenger, and supports information recording at any timing using the exterior camera via the agent unit, an occupant detection unit that detects the movement and / or conversation of the occupant; an object of interest identification unit that identifies an object of interest of the occupant outside the vehicle based on the occupant's actions and / or conversations; an object of interest recording unit that, when the object of interest is identified, notifies the passenger that the object of interest has been identified and records the object of interest using the exterior camera; a shared recognition determination unit that determines whether the passenger recognizes that the information recording assistance device has identified the subject of interest based on the passenger's actions and / or conversation; Equipped with the interest object recording unit starts recording the interest object using the exterior camera when the passenger recognizes that the interest object has been identified. Information recording support device.

13. An information recording assistance program that is mounted on a vehicle having an exterior camera that captures images outside the vehicle, has an agent unit that functions as an interface for a passenger, and operates an information recording assistance device that supports information recording at any timing using the exterior camera via the agent unit, The information recording support device, an occupant detection unit that detects the movement and / or conversation of the occupant; an object of interest identification unit that identifies an object of interest to the occupant outside the vehicle based on the occupant's actions and / or conversations; an object of interest recording unit that, when the object of interest is identified, notifies the passenger that the object of interest has been identified and records the object of interest using the exterior camera; and a shared recognition determination unit that determines whether the passenger recognizes that the information recording assistance device has identified the subject of interest based on the passenger's actions and / or conversation; and When the passenger recognizes that the object of interest has been identified, the object of interest recording unit is caused to start recording the object of interest using the exterior camera. Information recording support program.

14. An information recording support system comprising: a vehicle having an exterior camera for capturing images of the exterior of the vehicle; and an information recording support device having an agent unit that functions as an interface for a passenger, and supporting information recording at any timing using the exterior camera via the agent unit, The information recording support device comprises: an occupant detection unit that detects the movement and / or conversation of the occupant; an object of interest identification unit that identifies an object of interest of the occupant outside the vehicle based on the occupant's actions and / or conversations; an object of interest recording unit that, when the object of interest is identified, notifies the passenger by the information recording assistance device that the object of interest has been identified and records the object of interest by the outside vehicle camera; a shared recognition determination unit that determines whether the passenger recognizes that the information recording assistance device has identified the subject of interest based on the passenger's actions and / or conversation; Equipped with the subject of interest recording unit starts recording the subject of interest using the outside camera when the passenger recognizes that the information recording assistance device has identified the subject of interest. Information recording support system.

Images