Control method and control device for agent equipment

Abstract

To enable an occupant to accept driving assistance appropriately to increase the effectiveness of the driving assistance when executing the driving assistance by using an agent device.SOLUTION: A control device 100 of an agent device 200 assists an occupant of a vehicle C1 in driving the vehicle. The control device 100 includes a detection unit 101 for detecting a mode of operation of an object that is inside or outside of the vehicle C1 and a control unit 103 that causes the agent device 200 to execute relevant operations related to the mode of operation of the object in a mode different from that of driving assistance when the mode of operation of the object is detected.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a control method and a control device for an agent device that supports the driving of a vehicle occupant.

Background Art

[0002] Conventionally, there is a technology that makes a robot imitate human manual work. For example, even when the work performed by a human and the work performed by a robot are different, a technology has been proposed for easily teaching the operation of a robot based on the human manual work (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described conventional technology, it is possible to make a robot imitate a user's operation in response to a user's instruction. However, it is assumed that the robot that imitates in response to a user's instruction does not feel autonomy and subjectivity, and thus cannot enhance the user's trust. In addition, it is assumed that the user does not have an attachment to the robot that imitates in response to a user's instruction.

[0005] <​​​​The present invention aims to improve the effectiveness of driver assistance provided by agent devices by ensuring that occupants appropriately accept the assistance. [Means for solving the problem]

[0007] One aspect of the present invention provides assistance to the driver of a vehicle. and includes a movable part of a rod-shaped body. This is a control method for agent equipment. This control method involves a detection process to detect the operating mode of an object located inside or outside the vehicle, and when the operating mode of the object is detected... ,thing Includes control processing that causes an agent device to perform related actions related to the manner of body movement. In the detection process, based on captured images of the interior or exterior of the vehicle, a similar shape is extracted from among the objects present inside or outside the vehicle that resembles at least a part of the appearance of the agent device (at least one of the following: a rod-shaped member mounted on the vehicle, a rod-shaped member present outside the vehicle, and the arm of an occupant riding in the vehicle). The movement of the extracted similar shape is detected as the motion of the object. In the control process, a mimicry operation is performed by the agent device as a related operation, causing the movable part to mimic the movement of the similar shape. . [Effects of the Invention]

[0008] According to the present invention, when driving assistance is performed using an agent device, the occupants can appropriately accept the driving assistance, thereby increasing the effectiveness of the driving assistance. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a simplified diagram showing an example of the interior layout of a vehicle. [Figure 2] Figure 2 is a simplified diagram showing an example of the interior layout of a vehicle. [Figure 3] Figure 3 is a simplified perspective view showing an example of the external configuration of an agent device. [Figure 4] Figure 4 shows an example of the system configuration of an information processing system installed in a vehicle. [Figure 5] Figure 5 shows an example of how a movable part operates in accordance with the movement of the wiper. [Figure 6] Figure 6 shows an example of how the first and second movable parts are operated in response to the opening and closing of the ETC or parking gate bar located in front of the vehicle. [Figure 7] Figure 7 shows an example of how a movable part is operated in response to the guiding actions of people surrounding the vehicle. [Figure 8] Figure 8 shows an example of how the eye portion of the agent device is illuminated in accordance with the illumination status of the headlights. [Figure 9] Figure 9 shows an example of how the eye unit of an agent device flashes in accordance with the flashing state of the turn signal. [Figure 10] Figure 10 shows an example of a case where a hat is placed on the head of the main unit in response to the opening and closing of the sunroof installed in the vehicle. [Figure 11] Figure 11 shows an example of how the main unit operates according to the condition of a vehicle traveling on a road that includes straight and curved sections. [Figure 12] Figure 12 is a flowchart showing an example of equipment control processing in a control device. [Figure 13] Figure 13 is a flowchart showing an example of equipment control processing in a control device. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the attached drawings.

[0011] [Example of agent device installation] Figures 1 and 2 are simplified diagrams showing an example of the interior configuration of vehicle C1. Figure 1 shows a simplified view of the interior of vehicle C1 from the right side in the left-right direction, and Figure 2 shows a simplified view of the interior of vehicle C1 from the rear side in the front-rear direction. In addition, for the sake of ease of explanation, in Figures 1 and 2, only the wipers W1 and W2, dashboard 2, steering wheel 3, windshield 4, and agent equipment 200 are shown.

[0012] The agent device 200 is a small robot installed on the dashboard 2 of the vehicle C1. In FIGS. 1 and 2, an example is shown in which a robot mimicking an animal such as a rabbit is used as the agent device 200. Note that, in FIGS. 1 and 2, an example of installing the agent device 200 on the dashboard 2 is shown, but it is not limited thereto. For example, the agent device 200 may be installed on the upper part of the windshield 4. Further, in FIGS. 1 and 2, an example is shown in which a robot mimicking an animal such as a rabbit is used as the agent device 200, but it is not limited thereto. For example, a robot mimicking another animal, a robot mimicking a virtual creature (e.g., the face of an anime character), or a robot mimicking another object (e.g., a TV-type device, a radio-type device) may be used as the agent device 200. Also, another device having a user interface such as a display unit (e.g., a smartphone, a tablet terminal, a car navigation device, an IVI (In-Vehicle Infotainment)) may be used as the agent device 200. In this case, it is possible to change the operation mode of the robot by displaying the above-described robot on the display unit and changing the display mode of the robot.

[0013] The agent device 200 executes various operations based on an instruction from the control device 100 (see FIG. 4). For example, the agent device 200 outputs various information related to driving support when the driver D1 performs a driving operation under the control of the control device 100. As this driving support, route guidance, proposal of a stopover point according to the driver D1's preference, notification of road information to be noted, etc. are assumed. For example, in the proposal of a stopover point, it is possible to output voice such as "There is a delicious pudding shop ahead". Also, as the notification of road information to be noted, it is possible to check whether the driver D1 is looking around, output voice such as "Be careful because the road ahead is narrow", or output voice such as "Be careful because there is a railroad crossing ahead". Thus, the agent device 200 executes support for the driving experience and support for the driving experience.

[0014] Also, for example, by changing the operation mode of the agent device 200, a change is made to the appearance of the agent device 200, allowing the driver to visually perceive the change and outputting information related to driving assistance. Specifically, information related to driving assistance can be output by voice output from the agent device 200, or by changes in the expression, color, or movement of the face of the agent device 200. Also, for example, by changing each part (e.g., eyes, mouth) on the surface of the agent device 200, the facial expression can be changed. Thus, for example, when the driver D1 performs an appropriate driving operation, the agent device 200 outputs an action or voice praising the driver D1, causing the driver D1 to be more motivated to perform better driving operations, leading to an improvement in the safety of the vehicle C1. Also, the driver D1 praised by the agent device 200 can continue the driving operation calmly and with peace of mind.

[0015] Also, for example, the agent device 200 executes an operation to notify the driver D1 during driving of the outside vehicle sound acquired by the sound acquisition unit 140 (see FIG. 4) based on the control of the control device 100. As the outside vehicle sound to notify the driver D1 during driving, for example, a siren (horn), the sirens or warning sounds of emergency vehicles, instructions from police officers, etc. are assumed. When these sounds are acquired by the sound acquisition unit 140, the agent device 200 executes an operation to notify the driver D1 during driving of these sounds. For example, the agent device 200 can perform an operation to output the same sound as those sounds, or an operation indicating urgency together with this output (e.g., an operation of speaking loudly with a surprised expression). In this way, since the agent device 200 is installed at a position where it is easily visible to the driver D1 and the face of the agent device 200 is directed towards the driver D1, it becomes possible to provide appropriate driving assistance when the driver D1 performs a driving operation.

[0016] Here, if driver D1 feels that the agent device 200 is unreliable or that they cannot develop an attachment to it, it is conceivable that driver D1 will not accept the driving assistance provided by the agent device 200. In this case, it may be difficult to improve the effectiveness of the driving assistance provided by the agent device 200. Therefore, in this embodiment, in order to increase driver D1's trust in and attachment to the agent device 200, the agent device 200 is made to perform related actions, such as mimicking actions, that relate to the operating patterns of objects inside or outside the vehicle C1. This allows driver D1 to appropriately accept the driving assistance provided by the agent device 200 and improves the effectiveness of that driving assistance.

[0017] Here, related actions refer to movements related to a stimulus inside or outside vehicle C1. In other words, related actions refer to appropriate movements that are natural and not unnatural in relation to that stimulus. For example, by having the agent device 200 mimic the movement of an object inside or outside vehicle C1, the occupants of vehicle C1 can be given the feeling that they are sharing the same movement. It can also show the occupants of vehicle C1 that the agent device 200 is present and understands the situation. Furthermore, it is conceivable that the occupants may develop an attachment to the agent device 200 by unconsciously mimicking its movements.

[0018] [Example of external configuration of agent device] Figure 3 is a simplified perspective view showing an example of the external configuration of the agent device 200.

[0019] Figure 3(A) shows a perspective view of the agent device 200. Figure 3(B) shows a front view of the agent device 200. Note that the agent device 200 shown in Figure 3(B) corresponds to the agent device 200 shown in Figure 2. Figure 3(C) shows a top view of the agent device 200. Figure 3(D) shows a side view of the agent device 200. Note that the agent device 200 shown in Figure 3(D) corresponds to the agent device 200 shown in Figure 1. In this embodiment, the state of the agent device 200 shown in Figures 3(A) to (D) will be described as the normal state. Figures 3(E) to (J) show examples of external configurations when the operating mode of the agent device 200 is changed.

[0020] The agent device 200 is a device in which a roughly spherical main body 201 is provided with a rod-shaped first movable part 202 and a second movable part 203. Various images, such as eye parts E1 and E2, are displayed on the surface of the main body 201. In this embodiment, an example is shown in which the agent device 200 is installed on the dashboard 2 of a vehicle C1, with the eye parts E1 and E2 facing inwards. For this reason, one surface of the agent device 200 that includes the part in which the eye parts E1 and E2 are displayed will be described as the front part or face part. The front part of the agent device 200 will be described as the front side of the agent device 200, and the side opposite the front part of the agent device 200 will be described as the rear side of the agent device 200. Furthermore, in the agent device 200, the side on which the first movable part 202 is provided will be described as the right side of the agent device 200, and the side on which the second movable part 203 is provided will be described as the left side of the agent device 200.

[0021] The main body 201 rotates in the front-to-back direction of the agent device 200, i.e., in the direction of arrow A1 (see Figure 3(A) and (D)), around the mounting portion on the dashboard 2 of the vehicle C1 or a portion adjacent thereto. Similarly, the main body 201 rotates in the left-to-right direction of the agent device 200, i.e., in the direction of arrow A4 (see Figure 3(B)), around the mounting portion on the dashboard 2 of the vehicle C1. For example, as shown in Figure 3(E), the main body 201 can be rotated to the right of the agent device 200, i.e., in the direction of arrow A10. Also, for example, as shown in Figure 3(F), the main body 201 can be rotated to the left of the agent device 200, i.e., in the direction of arrow A11. Similarly, the main body 201 rotates in the direction of rotation around the rigidity axis, i.e., in the direction of arrow A7 (see Figure 3(C)), around the mounting portion on the dashboard 2 of the vehicle C1. Thus, a drive device (corresponding to the drive unit 270 shown in Figure 4) for driving the main body 201 is provided at the bottom of the agent device 200. This makes it possible to rotate the main body 201 of the agent device 200 in the left-right direction, rotate the main body 201 in the front-back direction, swing the main body 201 in the left-right direction, and swing the main body 201 in the front-back direction.

[0022] The first movable part 202 rotates in the front-to-back direction of the agent device 200, i.e., in the direction of arrows A2 and A3 (see Figure 3(A) and (D)), around the mounting portion of the first movable part 202 on the main body 201 as the pivot point. The first movable part 202 also rotates in the left-to-right direction of the agent device 200, i.e., in the direction of arrow A5 (see Figure 3(B)), around the mounting portion of the first movable part 202 on the main body 201 as the pivot point. Furthermore, the first movable part 202 rotates in the direction of rotation around the rigidity axis, i.e., in the direction of arrow A8 (see Figure 3(C)), around the mounting portion of the first movable part 202 on the main body 201 as the pivot point.

[0023] Similarly, the second movable part 203 rotates in the front-to-back direction of the agent device 200, i.e., in the direction of arrow A3 (see Figure 3(A)(D)), around the mounting portion of the second movable part 203 on the main body 201 as the pivot point. The second movable part 203 also rotates in the left-to-right direction of the agent device 200, i.e., in the direction of arrow A6 (see Figure 3(B)), around the mounting portion of the second movable part 203 on the main body 201 as the pivot point. Furthermore, the second movable part 203 rotates in the direction of rotation around the rigidity axis, i.e., in the direction of arrow A9 (see Figure 3(C)), around the mounting portion of the second movable part 203 on the main body 201 as the pivot point.

[0024] Thus, the first movable part 202 and the second movable part 203 are made capable of rotating in any direction, up, down, left, or right, like the ears or hands of an animal, such as a rabbit's ears. This makes it possible for the agent device 200 to simulate actions such as an animal, such as a rabbit, waving its ears or hands.

[0025] As described above, a drive device (corresponding to the drive unit 270 shown in Figure 4) for driving the first movable part 202 is provided at the mounting portion of the first movable part 202. This makes it possible to rotate the first movable part 202 in the left-right direction, rotate it in the front-back direction, swing it in the left-right direction, and swing it in the front-back direction. The second movable part 203 can also be moved in a similar manner.

[0026] Furthermore, at least a portion of the surface of the main body 201 is equipped with a display medium capable of displaying various images. Various display media such as light-emitting sheets, liquid crystal displays, and organic EL (Electro-Luminescence) can be used as this display medium. For example, as shown in Figures 3(A) to (F) and (J), normal eye parts E1 and E2 can be displayed. Also, as shown in Figure 3(G), eye parts B3 and B4 with eyebrows added above the normal eyes can be displayed. Also, as shown in Figure 3(G), a predetermined color, such as a red circle, can be displayed in the parts CK1 and CK2 corresponding to the cheeks below the eyes on the face. This makes it possible to display the agent device 200 as if it is making an embarrassed face (e.g., blushing). Note that the operation of the first movable part 202 or the second movable part 203 to cover the part corresponding to the eyes may also be used to simulate the agent device 200 making an embarrassed face. Thus, in this embodiment, the agent device 200 can be changed by an operating mode that changes at least a part of each component of the agent device 200, and by a display mode that changes the display state of at least a part of the surface of the agent device 200. Furthermore, the occupants of the vehicle C1 who observe these changes will be able to recognize that the agent device 200 is making some kind of movement. For this reason, in this embodiment, these changes in the agent device 200 will be described as operating modes of the agent device 200.

[0027] Furthermore, as shown in Figure 3(H), for example, eye areas E5 and E6, including eyes and eyebrows that resemble smiling eyes, can be displayed. Also, as shown in Figure 3(H), for example, a mouth area M1 resembling a smiling mouth can be displayed in the part of the face corresponding to the mouth. This makes it possible to make the agent device 200 operate in a manner that makes it appear as if it is smiling.

[0028] Furthermore, as shown in Figure 3(I), for example, eye parts E7 and E8, including eyes, eyebrows, and tears that resemble crying eyes, can be displayed. Also, as shown in Figure 3(I), for example, a mouth part M2 resembling a crying mouth can be displayed in the part of the face corresponding to the mouth. This makes it possible to make the agent device 200 operate in a manner that makes it appear as if it is crying.

[0029] Furthermore, as shown in Figure 3(J), for example, the color of the part of the face corresponding to the head (head H1) can be changed to make it appear as if the device is wearing a hat. This makes it possible to make the agent device 200 behave as if it were wearing a hat.

[0030] The operating modes of these agent devices 200 are merely examples, and other operating modes are possible. For example, the agent device 200 can be slid within a predetermined range on the dashboard 2. For example, the agent device 200 can be slid in the forward, backward, left, and right directions. Furthermore, for example, it is possible to create an effect where the agent device 200 appears to be temporarily floating.

[0031] [Example of an information processing system configuration] Figure 4 shows an example of the system configuration of the information processing system 10 installed in vehicle C1.

[0032] The information processing system 10 comprises a control device 100, a vehicle information acquisition unit 110, sensors 120, an image acquisition unit 130, a sound acquisition unit 140, and an agent device 200. The control device 100 is connected to the network 20 by a wireless communication method. The network 20 is a public telephone network, the Internet, or other network. The agent device 200 may also be connected to the network 20 by a wireless communication method. The control device 100 and the agent device 200 are connected by a wired communication or wireless communication method. The control device 100 controls the agent device 200.

[0033] The vehicle information acquisition unit 110 acquires vehicle information obtained from vehicle C1 and outputs the acquired vehicle information to the control device 100. This vehicle information is, for example, information that can be acquired based on CAN communication and includes the position information of vehicle C1, vehicle speed, and movement instruction signals for each component.

[0034] The sensors 120 are various sensors installed on vehicle C1. The sensors 120 also output detection information acquired by each sensor to the control device 100. Examples of sensors include acceleration sensors, vehicle speed sensors, seating sensors, wind speed sensors, and position information acquisition sensors.

[0035] The image acquisition unit 130 is installed inside or outside the vehicle C1, and based on the control of the control device 100, it captures images (image data) of subjects inside and outside the vehicle C1 and outputs image information related to the generated images to the control device 100. The image acquisition unit 130 is composed of, for example, one or more camera devices or image sensors capable of capturing images of subjects. For example, one or more devices capable of acquiring images of subjects present in all directions around the vehicle 1 and subjects inside the vehicle 1, such as a 360-degree camera, may be used.

[0036] The sound acquisition unit 140 is installed inside or outside the vehicle C1 and, based on the control of the control device 100, acquires sounds from the surrounding area of ​​the vehicle C1 (sounds from outside the vehicle) and sounds from inside the vehicle, and outputs sound information related to the acquired sounds to the control device 100. For example, one or more microphones or sound acquisition sensors can be used as the sound acquisition unit 140.

[0037] The control device 100 includes a detection unit 101, a related operation determination unit 102, a control unit 103, a storage unit 104, and a communication unit 105.

[0038] The detection unit 101 detects the operation of an object inside or outside the vehicle C1 based on information output from the vehicle information acquisition unit 110, sensors 120, image acquisition unit 130, and sound acquisition unit 140, or information acquired via the network 20. The detection unit 101 then outputs the detection result to the related operation determination unit 102 and the control unit 103. The detection unit 101 is implemented by a processing unit such as a CPU (Central Processing Unit).

[0039] For example, the detection unit 101 performs a similar shape extraction process to extract similar shapes that are similar in shape to at least a part of the appearance of the agent device 200, and an operation mode detection process to detect the operation mode of the similar shape.

[0040] In the similar shape extraction process, similar shapes can be extracted based, for example, on images acquired by the image acquisition unit 130 and related operation information DB 150 stored in the storage unit 104. The related operation information DB 150 is information used to detect the operation patterns of objects located inside or outside the vehicle C1, and is a database in which information about similar shapes, the operation patterns of similar shapes, and related operations are associated. For example, if the bar B1 installed at gate G1 of the ETC (Electronic Toll Collection System) lane shown in Figure 6 is considered a similar shape, then information for extracting bar B1 as information about similar shapes, the operation of bar B1 opening and closing as the operation pattern of similar shapes, and instruction information for rotating the first movable part 202 or the second movable part 203 as related operations are associated and stored in the related operation information DB 150. In addition, a list of objects with similar shapes inside and outside the vehicle (parts list) may be stored as information about similar shapes.

[0041] Furthermore, in the similar shape extraction process, objects included in the image acquired by the image acquisition unit 130 are extracted, and from among the extracted objects, objects having a shape included in the related operation information DB 150 or a similar shape are extracted. For the object extraction process and the extraction process of objects with the same or similar shapes, known image recognition techniques can be used. Similar shapes can, for example, be the shape of a rod-shaped member similar to the first movable part 202 and the second movable part 203. Examples of rod-shaped members include the wipers W1 and W2 shown in Figure 2, the bar B1 shown in Figure 6, and the arm of the guide U1 shown in Figure 7.

[0042] Next, in the motion pattern detection process, for example, the transitions of similar shapes extracted in the similar shape extraction process are compared with the images acquired by the image acquisition unit 130, and the motion patterns of similar shapes can be detected based on this comparison result. Note that known motion detection techniques can be used for the motion pattern detection process.

[0043] Furthermore, the motion patterns of similar shapes contained in an image may be detected using artificial intelligence (AI). For example, similar shapes of objects and their motion patterns can be pre-trained, and this training data can be used for detection. Then, for an image acquired by the image acquisition unit 130, objects with similar shapes contained in the image and their motion patterns can be detected using the training data.

[0044] The above example illustrates how to detect the operation patterns of similar shapes based on images acquired by the image acquisition unit 130. However, operation patterns of similar shapes may also be detected based on other information. For example, operation patterns of similar shapes can be detected based on sounds acquired by the sound acquisition unit 140. For instance, for a rod-shaped member that emits a specific sound and performs a specific action, that specific sound can be acquired based on the sound acquired by the sound acquisition unit 140. Furthermore, if that specific sound has been acquired, the specific action of that rod-shaped member can be extracted.

[0045] The above examples illustrate the detection of motion patterns of similar shapes using similar shape extraction and motion pattern detection processes. However, motion patterns of objects inside or outside vehicle C1 may be detected by other methods. These detection methods will be explained as appropriate with reference to Figures 5 to 13. Furthermore, similar to the detection of motion patterns of similar shapes described above, the motion patterns of objects inside or outside vehicle C1 may be detected using artificial intelligence.

[0046] The related action determination unit 102 determines related actions related to the motion of the object detected by the detection unit 101 based on the related action information DB 150, and outputs the determined related actions to the control unit 103. In other words, the related action determination unit 102 functions as an action calculation unit that calculates related actions related to the motion of the object, such as mimicry actions. The related action determination unit 102 is implemented by a processing unit such as a CPU. The process of determining related actions by the related action determination unit 102 will be explained in detail with reference to Figures 5 to 13.

[0047] The control unit 103 controls each part based on various programs stored in the memory unit 104. The control unit 103 is implemented by a processing unit such as a CPU. The vehicle ECU (Electronic Control Unit) of vehicle C1 may be used as the control unit 103, or a processing unit different from the vehicle ECU may be provided as the control unit 103.

[0048] For example, the control unit 103 causes the agent device 200 to execute the related operation determined by the related operation determination unit 102. In other words, the control unit 103 functions as an operation reflection unit that reflects the related operation to the agent device 200. The control processing of the related operation by the control unit 103 will be explained in detail with reference to Figures 5 to 13.

[0049] The memory unit 104 is a storage medium for storing various types of information. For example, the memory unit 104 stores various types of information necessary for the control unit 103 to perform various processes (e.g., control programs, map information, related operation information DB 150). The memory unit 104 also stores various types of information acquired via the communication unit 105. The memory unit 104 can be, for example, ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof. The memory unit 104 is connected to various parts of the control device 100, and information is exchanged between them.

[0050] The communication unit 105, based on the control of the control unit 103, uses wireless communication to exchange various types of information with other devices.

[0051] The agent device 200 includes a sound acquisition unit 210, an image acquisition unit 220, a control unit 230, a storage unit 240, a sound output unit 250, a display unit 260, and a drive unit 270.

[0052] The sound acquisition unit 210 is located inside the agent device 200 and, based on the control of the control unit 230, acquires sounds from the vicinity of the agent device 200 and outputs sound information related to the acquired sounds to the control unit 230. For example, one or more microphones or sound acquisition sensors can be used as the sound acquisition unit 210.

[0053] The image acquisition unit 220 is located inside the agent device 200 and captures images (image data) of subjects outside the agent device 200. It also outputs image information related to the generated images to the control unit 230. The image acquisition unit 220 is composed of, for example, one or more camera devices or image sensors capable of capturing images of subjects. For example, the image acquisition unit 220 can be installed near the eye area of ​​the agent device 200 so that the line of sight of the driver D1 or other occupants is included in the imaging range.

[0054] The control unit 230 controls each part based on various programs stored in the memory unit 240. The control unit 230 is implemented by a processing unit such as a CPU.

[0055] The memory unit 240 is a storage medium for storing various types of information. For example, the memory unit 240 stores various types of information (e.g., control programs) necessary for the control unit 230 to perform various processes. The memory unit 240 also stores various types of information output from the control device 100. The memory unit 240 can be, for example, a ROM, RAM, HDD, SSD, or a combination thereof. The memory unit 240 is connected to various parts of the agent device 200, and information is exchanged between them.

[0056] The sound output unit 250 outputs sound into the vehicle C1 based on the control of the control unit 230. For example, one or more speakers can be used as the sound output unit 250.

[0057] The display unit 260 displays various images around the agent device 200 based on the control of the control unit 230. The display unit 260 can use various display media as described above, such as light-emitting sheets, liquid crystal displays, organic EL displays, etc.

[0058] The drive unit 270 drives each part of the agent device 200, namely the main body 201, the first movable part 202, and the second movable part 203, based on the control of the control unit 230. Various drive devices such as actuators and drive motors can be used as the drive unit 270.

[0059] [Examples of related operations of agent devices] This embodiment shows an example in which the agent device 200 is instructed to perform related operations depending on the operation of an object located outside or inside the vehicle C1. Figures 5 to 11 show an example of the correspondence between the operation of an object and the related operations to be performed by the agent device 200.

[0060] [Example of moving a movable part in accordance with the movement of the wiper] Figure 5 shows an example of how the first movable part 202 operates in accordance with the movements of wipers W1 and W2. Note that in Figure 5, for the sake of simplicity, only the movement of wiper W1 is shown in a simplified manner.

[0061] Figure 5(A) shows the wiper W1 in the stopped state. In this case, the driver D1 operates the wiper lever to turn on the wiper W1.

[0062] Figure 5(B) shows the state in which the movement of wiper W1 is detected after the wiper W1 has been turned on.

[0063] Figure 5(C) shows the state in which the agent device 200 performs related operations related to the movement of the wiper W1. In Figure 5, an example is shown in which the agent device 200 operates the first movable part 202 in response to the movement of the wiper W1. Specifically, the first movable part 202 is rotated in the forward and backward direction, i.e., in the direction of arrow A2. By operating the first movable part 202 in this way, the driver D1, seeing the related operation of the agent device 200 which mimics the movement of the wiper W1, can recognize that the agent device 200 is checking its surroundings. This can increase the reliability of the agent device 200.

[0064] The detection unit 101 can detect the start instruction signals for wipers W1 and W2 via CAN communication. Furthermore, the period, maximum angle, and angular velocity of the reciprocating motion of wipers W1 and W2 can be acquired as vehicle information based on preset values. The detection unit 101 can also acquire the phase, period, maximum angle, and angular velocity of the reciprocating motion of wipers W1 and W2 based on images acquired by the image acquisition unit 130. Therefore, based on the acquired phase and the preset period, maximum angle, and angular velocity of wipers W1 and W2, the operation to rotate the first movable part 202 can be determined. For example, the reversal of the reciprocating motion of wipers W1 and W2 can be matched with the reversal of the rotation of the first movable part 202. Alternatively, instead of rotating the first movable part 202, the second movable part 203 or the main body part 201 may be rotated, or at least two of these may be rotated.

[0065] [Example of operating a movable part in response to the movement of an ETC or parking lot gate bar] Figure 6 shows an example of how the first movable part 202 is operated in response to the opening and closing of the barrier B1 of an ETC or parking lot gate G1 located in front of the vehicle C1.

[0066] The left side of Figure 6(A) shows a simplified representation of the gate G1 with its bar B1 closed, located in front of vehicle C1. The right side of Figure 6(A) shows the agent device 200 in its normal state. When the gate G1's bar B1 is located in front of vehicle C1, the detection unit 101 extracts the bar B1 as a similar shape based on the image acquired by the image acquisition unit 130. In this case, the detection unit 101 may also extract the bar B1 using the ETC and parking lot location information included in the map information. For example, the detection unit 101 determines whether vehicle C1 is located at or near the ETC and parking lot location based on the ETC and parking lot location information included in the map information. If vehicle C1 is located at or near the ETC and parking lot location, and the image acquired by the image acquisition unit 130 includes a rod-shaped object, the detection unit 101 can extract that rod-shaped object as the bar B1. This improves the accuracy of the similar shape extraction process.

[0067] The left side of Figure 6(B) shows a simplified representation of the gate G1 located in front of vehicle C1 with its bar B1 open. When the bar B1 of gate G1 opens in the direction of arrow A21, the detection unit 101 detects the movement of the bar B1, which has been extracted as a similar shape based on the image acquired by the image acquisition unit 130. When the movement of the bar B1 is detected in this way, the related operation determination unit 102 determines the operation to be performed by the first movable part 202 based on the related operation information DB 150. For example, the angle by which the first movable part 202 is rotated is determined according to the angle by which the bar B1 moved. For example, the first movable part 202 can be rotated in the direction of arrow A22 by the same angle by which the bar B1 moved. In this case, the first movable part 202 can be rotated with the same period and angular velocity as the period and angular velocity of the movement of the bar B1.

[0068] In this way, when bar B1 is opened, the first movable part 202 is made to perform a related operation, so that at the moment bar B1 is opened upward, the agent device 200 appears to raise its hand in imitation of bar B1. This makes it possible to make the agent device 200 perform a related operation in relation to changes in the front of vehicle C1.

[0069] [Example of a movable part operating in response to human-guided movements] Figure 7 shows an example of how the second movable part 203 is operated in response to the guiding actions of people present around vehicle C1. For example, it is assumed that when vehicle C1 is in a parking lot or gas station, there are people present around vehicle C1 who are guiding it. It is also assumed that there are police officers present around vehicle C1 for some reason who are guiding it. The guiding actions of people include, for example, hand signals to guide vehicle C1. For example, in parking lot guidance and gas station guidance, guidance is often provided by people using their actions in addition to verbal guidance such as "all clear, all clear."

[0070] The left side of Figure 7(A) shows a simplified representation of the guide U1 surrounding the vehicle C1. The right side of Figure 7(A) shows the agent device 200 in its normal state. When a guide U1 is present around the vehicle C1, the detection unit 101 extracts a rod-shaped member B2 held in the arm or hand of the guide U1 as a similar shape based on the image acquired by the image acquisition unit 130. In this case, the system may also detect a human or human face based on the image acquired by the image acquisition unit 130, extract the detected human hand, and perform a similar shape extraction process on that hand. Alternatively, the system may detect that the guide U1 is performing a guiding action based on a sound acquired by the sound acquisition unit 140 (e.g., "all clear"), and after this detection, perform a similar shape extraction process using the image. These measures can improve the accuracy of the similar shape extraction process.

[0071] The left side of Figure 7(B) shows the movement of the guide U1's hand, which is located around the vehicle C1. When the guide U1's hand moves in this way, the detection unit 101 detects the movement of the guide U1's hand or the rod-shaped member B2 held in the hand, which has been extracted as a similar shape based on the image acquired by the image acquisition unit 130. When the movement of the guide U1's hand or the rod-shaped member B2 held in the hand is detected in this way, the related action determination unit 102 determines the action to be performed by the second movable unit 203 based on the related action information DB 150. For example, the second movable unit 203 can be rotated by the same angle as the angle in which the guide U1's hand moved. In this case, the second movable unit 203 can be rotated with the same period as the period in which the guide U1's hand moved. That is, it is preferable to have the agent device 200 perform the same movement as what the driver D1 sees.

[0072] In this way, when the guide U1's hand moves, the second movable part 203 is activated, so that the agent device 200 appears to raise its hand at the moment the guide U1's hand moves. This makes it possible to have the agent device 200 perform actions in relation to changes in the front of the vehicle C1.

[0073] Thus, when the agent device 200 is to perform actions related to an object visible to the driver D1 in front of the vehicle C1, it is preferable to move each part of the agent device 200 in the same direction, at the same timing, and with the same period as the movement of that object. In other words, by mimicking the movement of an object that has a distinctive timing or movement that is distinctive from the driver D1's perspective, it is possible to make the driver D1 recognize that the agent device 200 is moving in the same way as the vehicle C1 with respect to the object visible to the driver D1. This can increase the reliability of the agent device 200 and make it easier for the driver to develop an attachment to the agent device 200.

[0074] Although Figures 5 to 7 show examples of the first movable part 202 or the second movable part 203 performing related operations, other related operations other than those described above may be performed by any one or more of the main body 201, the first movable part 202, or the second movable part 203.

[0075] [Example of lighting up the eyes according to the status of the headlights] Figure 8 shows an example of how the eye sections E1 and E2 of the agent device 200 are illuminated according to the illumination status of the headlights HL1 and HL2.

[0076] The left side of Figure 8(A) shows a simplified view of the vehicle C1. The right side of Figure 8(A) shows the agent equipment 200 in its normal state.

[0077] The left side of Figure 8(B) shows the state in which the headlights HL1 and HL2 are illuminated. The detection unit 101 can detect the illumination status of the headlights HL1 and HL2, for example, via CAN communication. When the illumination of the headlights HL1 and HL2 is detected in this way, the related operation determination unit 102 determines, based on the related operation information DB 150, to illuminate the eye parts E1 and E2 of the agent device 200 brightly. Alternatively, instead of illuminating the eye parts E1 and E2 brightly, the unit may perform an action to make the eye parts E1 and E2 blink once, or it may perform a display effect to make the eye parts E1 and E2 open wide.

[0078] In this way, when the headlights HL1 and HL2 are turned on, the eye sections E1 and E2 of the agent device 200 are made to light up brightly, so that the agent device 200 appears to notice the illumination of the headlights HL1 and HL2 at the moment they start to light up and changes its eye behavior. This makes it possible to have the agent device 200 perform related actions in relation to the changes in the headlights HL1 and HL2.

[0079] [Example of flashing the eye unit in accordance with the flashing status of the turn signal] Figure 9 shows an example of how the eye units E1 and E2 of the agent device 200 are made to flash according to the flashing state of the turn signals DI1 and DI2.

[0080] The left side of Figure 9(A) shows a simplified view of the exterior of vehicle C1. The right side of Figure 9(A) shows the agent equipment 200 in its normal state.

[0081] The left side of Figure 9(B) shows the state in which the turn signal DI1 is flashing. The detection unit 101 can detect the start instruction signal of the turn signal DI1, for example, via CAN communication. The flashing period of the turn signal DI1 can also be obtained from a preset value. Furthermore, the detection unit 101 can obtain a sound related to the turn signal DI1, for example, a "clicking" sound, from the sound acquisition unit 140, and based on that sound, it can obtain the phase of the sound related to the turn signal DI1, i.e., the flashing phase of the turn signal DI1. Based on the preset flashing period of the turn signal DI1 and its phase, the related operation determination unit 102 determines the operation to flash the eye unit E1 of the agent device 200 in accordance with the flashing period of the turn signal DI1. That is, the peak of the sound of the flashing of the turn signal DI1 is matched with the peak of the flashing of the eye unit E1. Note that instead of flashing the eye unit E1, the eye unit E2 may be flashed, or both eye units E1 and E2 may be flashed.

[0082] In this way, when the turn signal DI1 starts flashing, the eye units E1 and E2 of the agent device 200 are made to flash, so that the agent device 200 appears to have noticed the flashing of the turn signal DI1 at the moment it starts flashing and has changed its eye state. This makes it possible to have the agent device 200 perform related actions in relation to the changes in the turn signals DI1 and DI2.

[0083] Figures 8 and 9 show examples of how the agent device 200 performs related operations in response to the illumination status of the headlights HL1 and HL2 and the flashing status of the turn signals DI1 and DI2. However, the agent device 200 may also perform related operations in response to the illumination status of other lights besides these.

[0084] Furthermore, Figures 8 and 9 show an example in which the eye units E1 and E2 are made to flash or emit light as related operations performed in accordance with the illumination state of the headlights HL1 and HL2 and the flashing state of the turn signals DI1 and DI2. However, other related operations other than flashing or emitting light of the eye units E1 and E2 may be performed. For example, related operations using the main body 201, the first movable part 202, or the second movable part 203 may be performed. In addition, the flashing or emitting light of the eye units E1 and E2 may be performed in conjunction with these related operations. For example, in the example shown in Figure 9, related operations can be performed to match the period of the sound of the flashing of the turn signal DI1 with the period of rotation of the main body 201, the first movable part 202, or the second movable part 203. These measures can increase the reliability of the agent device 200 and make it easier for users to develop an attachment to the agent device 200.

[0085] [An example of placing a hat on the top of the main unit in accordance with the opening and closing of the sunroof] Figure 10 shows an example of a case where a hat is placed on the head H1 of the main body 201 in accordance with the opening and closing of the sunroof SR1 installed on the vehicle C1.

[0086] The left side of Figure 10(A) shows a top view of the vehicle C1 with a sunroof SR1 installed on the roof. The right side of Figure 10(A) shows the agent equipment 200 in its normal state.

[0087] The left side of Figure 10(B) shows the case when the sunroof SR1 on the ceiling is opened by the occupant's operation. When the sunroof SR1 on the ceiling is opened in this way, the detection unit 101 detects a sunroof open signal from CAN communication. When a sunroof open signal is detected in this way, the related operation determination unit 102 decides to change the color of the upper part of the main body 201 of the agent device 200, i.e., the head H1. The operation mode for changing the color of this head H1 is the same as the example shown in Figure 3(J).

[0088] In this way, when the sunroof SR1 is opened, the color of the head H1 of the main body 201 of the agent device 200 is changed, making the agent device 200 appear as if it is wearing a hat. This makes it possible to have the agent device 200 perform actions in relation to changes in the upper part of the vehicle C1.

[0089] In this example, the agent device 200 is shown as performing an action that resembles putting on a hat, but other related actions may be performed. For example, the action may be a hand movement that resembles putting on a hat, or an action that rotates the first movable part 202 or the second movable part 203 from bottom to top. Alternatively, the action may be one that makes the eyes feel dazzled, or an action that makes the eyes feel dazzled, such as a hand movement that covers the eyes. For example, the action of rotating the first movable part 202 or the second movable part 203 from bottom to the vicinity of the eye parts E1 and E2 can be a hand movement that covers the eyes.

[0090] [Example of operating the main unit according to the vehicle's condition on a curve] Figure 11 shows an example of how the main unit 201 is operated according to the state of a vehicle C1 traveling on a road R1 that includes straight and curved sections.

[0091] The left side of Figure 11(A) shows a vehicle C1 traveling on a straight road R1. In this case, the agent device 200 is in its normal state, as shown on the right side of Figure 11(A).

[0092] The left side of Figure 11(B) shows a vehicle C1 traveling on a curved road R1. When vehicle C1 travels on a curved road R1, the lateral tilt of vehicle C1 on the curve, i.e., the tilt outward or inward, can be detected by sensors, such as an acceleration sensor. When the detected value of the acceleration sensor exceeds a predetermined value, the movable part in the direction exceeding that predetermined value is operated. In this case, the direction of the curve as seen from the driver D1 may be the same as the direction of the movable part to be operated, or the movable part may be operated in the opposite direction. Alternatively, an associated operation may be performed to rotate the main body 201 in the direction in which the detected value of the acceleration sensor exceeded the predetermined value. Figure 11 shows an example in which an associated operation is performed to rotate the main body 201 in the direction of arrow A26, where the detected value of the acceleration sensor exceeded the predetermined value.

[0093] Although Figure 11 shows an example where vehicle C1 travels on a curved road R1, related actions may be performed based on other movements of vehicle C1. For example, related actions may be performed based on the acceleration or deceleration of vehicle C1. For instance, if the value of the acceleration sensor exceeds a predetermined value, related actions such as tilting or shaking the agent device 200 in the direction exceeding that predetermined value, for example, in the forward, backward, left, or right directions of the agent device 200, can be performed.

[0094] Note that the relationship between the operation of an object located outside or inside vehicle C1 and the related operation of the agent device 200, as shown in Figures 5 to 11, is just one example, and other relationships, i.e., other relationships between the operation of an object and its related operation, may be implemented.

[0095] For example, the agent device 200 may perform related operations based on the operating mode or movement of an object present in the vehicle C1 itself. Alternatively, the agent device 200 may perform related operations based on the operating mode or movement of an object, such as an accessory, that has been retrofitted to the vehicle C1.

[0096] Objects present in the vehicle C1 itself include, for example, rod-shaped members such as the AT shift lever, MT manual shift lever, wiper lever, and turn signal lever, annular members such as the steering wheel, doors, air conditioning controls, seat belts, etc.

[0097] For example, when a rod-shaped member such as an AT shift lever, an MT manual shift lever, a wiper lever, or a turn signal lever is operated by driver D1, the detection unit 101 can detect the operation based on the image acquired by the image acquisition unit 130. Furthermore, the detection unit 101 can detect each operation and each control amount via CAN communication.

[0098] For example, in the case of an automatic transmission (AT) shift lever, it is preferable to have the agent device 200 perform related operations that mimic the timing and angle of the shift lever's movement. For example, the agent device 200 can detect the relative angle (amount of change) and angular velocity of the AT shift lever's movement and rotate its movable part at the same angle and angular velocity as the detected values. For example, if the AT shift lever moves from P to D by a predetermined angle, the agent device 200 can rotate its movable part in accordance with the angle and angular velocity of that movement. In this way, by slightly mimicking the angle and speed of the AT shift lever, it is possible to make driver D1 recognize that the agent device 200 is moving in a similar manner to driver D1. Furthermore, in the case of a manual transmission (MT) shift lever, since its movement is distinctive, it is preferable to have the agent device 200 perform related operations that mimic that movement.

[0099] Furthermore, when these rod-shaped members are manipulated, the main body 201, the first movable part 202, or the second movable part 203 can be rotated in accordance with the movement of each rod-shaped member. For example, as shown in Figures 5 to 7, the main body 201, the first movable part 202, or the second movable part 203 can be rotated in accordance with the movement of each rod-shaped member.

[0100] Furthermore, for example, when an annular member such as a steering wheel is operated by the driver D1, the operating angle can be detected based on the image acquired by the image acquisition unit 130 or CAN communication. In this case, for example, if a steering operation greater than a predetermined angle, such as 90 degrees, is detected, the related operation can be performed by rotating one of the main body 201, first movable part 202, or second movable part 203 of the agent device 200 by the same angle as the steering operation. For example, by rotating the main body 201 of the agent device 200 by the same angle as the steering operation in the direction of arrow A7 (see Figure 3(C)), the driver D1 can be given the impression that the round-shaped agent device 200, similar to a steering wheel, rotates in accordance with the rotation of the steering operation.

[0101] Furthermore, it is assumed that the agent device 200 will perform related operations based on the operation mode of the door or its movement, as an object present in the vehicle C1 itself. In this case, if the vehicle C1 shakes due to the opening and closing of the door, the related operations can be performed by shaking one of the main body 201, the first movable part 202, or the second movable part 203 of the agent device 200 in accordance with that shaking. The opening and closing of the door can be detected by acquiring the door opening and closing signal based on CAN communication.

[0102] This gives driver D1 the impression that the agent device 200 is shaking due to the impact of the door opening and closing. In this example, we have shown an example of shaking one part of the agent device 200, but other related actions may be performed. For example, the related action may be performed so that the agent device 200 faces the direction of the door that is being opened and closed. This gives driver D1 the impression that it is reacting to the impact of the door opening and closing.

[0103] Furthermore, we consider a case where the agent device 200 performs related operations based on the operation mode or movement of the power window, as an object present in the vehicle C1 itself. In this case, when the power window is opening or closing, the related operation can be performed by rotating one of the main body 201, the first movable part 202, or the second movable part 203 of the agent device 200 in accordance with the opening or closing operation. For example, an operation reminiscent of the operation of rotating a manual lever used to manually open and close a window in the past, such as the rotation of either the first movable part 202 or the second movable part 203, may be performed as the related operation. This makes it possible to give driver D1 the impression that the ears of the agent device 200 are rotating.

[0104] Furthermore, we consider a scenario where the agent device 200 performs related operations based on the operating mode or movement of the air conditioner, as an object present in the vehicle C1 itself. In this case, when the air conditioner is operating, the related operations can be performed by operating one of the main body 201, the first movable part 202, or the second movable part 203 of the agent device 200 in response to the sound of the air conditioner's fan. For example, by shaking either the first movable part 202 or the second movable part 203, it is possible to give driver D1 the impression that the agent device 200's ears are flapping.

[0105] Furthermore, it is assumed that the agent device 200 will perform related actions based on the operation of the seat belt or its movement, as an object present in the vehicle C1 itself. For example, if an occupant of vehicle C1 fastens their seat belt, the agent device 200 can perform related actions such as moving the first movable part 202 from the upper right to the lower left, or moving the second movable part 203 from the upper left to the lower right, in response to the fastening action. These related actions can give the occupant the impression that the agent device 200 is pretending to fasten their seat belt. The action of an occupant of vehicle C1 fastening their seat belt can be detected based on images acquired by the image acquisition unit 130. Alternatively, instead of the agent device 200 performing related actions such as fastening a seat belt, the agent device 200 may perform an action such as rotating its main body 201 in the left-right direction to check its surroundings after fastening the seat belt.

[0106] Furthermore, the agent device 200 may perform related operations based on the operating modes or movements of other objects or components other than those described above, as objects present in the vehicle C1 itself.

[0107] Let's consider a scenario where an accessory, as an object retrofitted to vehicle C1, performs related operations on the agent device 200 based on its operating mode or movement. The accessory is, for example, a traffic safety charm. For instance, if the charm shakes, the agent device 200 can perform related operations by causing one of its main body 201, first movable part 202, or second movable part 203 to shake in response to the shaking. The shaking of the accessory can be acquired based on an image obtained by the image acquisition unit 130. Alternatively, the related operations may be performed based on the image if the shaking of the accessory exceeds a predetermined value.

[0108] Furthermore, objects present in the vehicle C1 itself may include, for example, the driver D1 and other occupants. In this case, the agent device 200 may perform related actions based on the driving operations of driver D1, and the actions or postures of driver D1 and other occupants. For example, the actions or postures of driver D1 and other occupants could include checking their surroundings, crossing their arms, sneezing, rubbing their eyes, or sleeping.

[0109] For example, in response to driver D1's action of checking its surroundings, the main body 201 of the agent device 200 can be rotated left and right, and the agent device 200 can perform an action as a related action that makes it appear as if it is checking left and right. This related action makes it possible to give driver D1 or other occupants the impression that the agent device 200 is mimicking driver D1's action of checking its surroundings by checking left and right.

[0110] Furthermore, as a related action, it is possible to move the first movable part 202 and the second movable part 203 from below to the front, bringing them closer together, for example, in response to the driver D1 or another occupant crossing their arms. This related action makes it possible to give the driver D1 or other occupants the impression that the agent device 200 is mimicking the act of crossing arms.

[0111] Furthermore, as a related action, it is possible to perform an action such as moving the first movable part 202 or the second movable part 203 from below to the vicinity of the eye in response to the driver D1 or another occupant rubbing their eyes. This related action makes it possible to give the driver D1 or other occupants the impression that the agent device 200 is mimicking rubbing their eyes.

[0112] For example, the agent device 200 may perform relevant operations based on the behavior or movement of objects surrounding vehicle C1, or the environment surrounding vehicle C1. For instance, if vehicle C1 is experiencing a strong crosswind, it is possible to perform relevant operations related to that crosswind. The strength of the crosswind experienced by vehicle C1 can be determined by installing a wind speed sensor outside vehicle C1 and obtaining the wind speed value from that sensor. Alternatively, weather information can be obtained from external devices connected via the communication unit 105 and network 20, and the wind speed and wind direction included in that weather information can be used. In this case, weather information for the location where vehicle C1 is traveling can be obtained based on the location information of vehicle C1 obtained by the location information acquisition sensor.

[0113] For example, if it is detected that the wind speed in the lateral direction relative to the direction of travel of vehicle C1 is greater than a predetermined value, the agent device 200 can be made to perform a related operation based on the lateral direction and the strength of the wind speed. For example, the main body 201 of the agent device 200 can be rotated so that it tilts in the lateral wind direction. This related operation can give the driver D1 or other occupants the impression that the agent device 200 is adjusting its position and bracing itself to prevent being blown away by the crosswind. This can give the driver D1 or other occupants the impression that they, the vehicle C1, and the agent device 200 are sharing the same space.

[0114] [Example of control device operation] Figures 12 and 13 are flowcharts illustrating an example of device control processing in the control device 100. This device control processing is executed by the control device 100 based on a program stored in the memory unit 104. This device control processing shows an example where related operations are performed when the detected object has a similar shape. This device control processing is always executed at each control cycle. This device control processing will be explained with reference to Figures 1 to 11 as appropriate.

[0115] In step S501, the control unit 103 determines whether the driving load (workload) of driver D1 while driving vehicle C1 is below a standard. For example, if vehicle C1 is driving in a specific location, or is approaching that specific location, it is determined that the driving load of driver D1 is higher than the standard. This specific location refers to, for example, a section of road or place where driver D1 of vehicle C1 needs to pay particular attention while driving. For example, specific locations include road merging points, lane change points, parking lots, intersections, etc. This specific location can be determined based on attribute information associated with map information. Also, if driver D1 is listening to something while driving vehicle C1, for example, listening to music, radio, etc., it is determined that the driving load of driver D1 is higher than the standard. These can be determined by obtaining the output status from the sound output unit (not shown) installed in vehicle C1. If the driving load of driver D1 is below the standard, the process proceeds to step S502. On the other hand, if the operating load of driver D1 is higher than the standard, the device control processing will be terminated.

[0116] In step S502, the control unit 103 determines whether the load (computational load) of the control processing of the agent device 200 is below a threshold. This threshold is a fixed value set in advance and can be set as appropriate based on experimental data, etc. Alternatively, it may be determined whether the content of the control processing instructed to the agent device 200 is a specific type of control, for example, whether it is a control to dance in time with music. If the load of the control processing of the agent device 200 is below the threshold, the process proceeds to step S503. On the other hand, if the load of the control processing of the agent device 200 is greater than the threshold, the operation of the device control processing is terminated.

[0117] Furthermore, it may be determined whether the computational load on agent device 200 is high or low. For example, if agent device 200 is instructed to perform a pre-configured task, it is determined that the computational load on agent device 200 is high, and if agent device 200 is not instructed to perform a pre-configured task, it is determined that the computational load on agent device 200 is low. In other words, when agent device 200 is receiving a predetermined execution command value (instruction), it is determined that the computational load on agent device 200 is high, and the operation of the device control process is terminated.

[0118] Furthermore, if the execution time of the related operations of the agent device 200 is short, for example, if the execution time falls below a threshold, the currently running task may be temporarily suspended to perform the related operations, and then the task may be restarted.

[0119] In step S503, the control unit 103 determines whether the detection unit 101 has detected the movement pattern of an object located outside or inside the vehicle C1. The detection process for detecting the movement pattern of an object by the detection unit 101 is the same as the detection process described above. If the movement pattern of an object located outside or inside the vehicle C1 is detected, the process proceeds to step S504. On the other hand, if the movement pattern of an object located outside or inside the vehicle C1 is not detected, the device control process is terminated.

[0120] In step S504, the related operation determination unit 102 determines whether the object whose operation mode was detected in step S503 has a similar shape. For example, as shown in Figures 5 to 7, if the operation modes of the wipers W1 and W2, the bar B1, and the arm of the guide U1 are detected, the object whose operation mode was detected is determined to have a similar shape. If the object whose operation mode was detected has a similar shape, the process proceeds to step S504. On the other hand, if the object whose operation mode was detected does not have a similar shape, the operation of the equipment control process is terminated.

[0121] In step S505, the related operation determination unit 102 extracts similar shapes with a high priority from among the similar shapes detected in step S503.

[0122] Here, priority can be set based on the operation time required to have the agent device 200 perform the related operation, whether the similar shape is an object inside or outside the vehicle, whether the similar shape is part of driver D1, etc. For example, since the operation of a similar shape outside the vehicle is often instantaneous, it is preferable to give a higher priority to similar shapes outside the vehicle. Here, an example is shown in which the determination is made on two axes: whether it is inside or outside the vehicle, and the length of time. For example, if the object and space are used as the basis, the priority of similar shapes of objects outside the vehicle can be set to the highest, the priority of similar shapes of parts of the occupants inside the vehicle to the next highest, and the priority of similar shapes of objects other than the occupants inside the vehicle to the lowest. Also, for example, if the time axis is used as the basis, the priority of similar shapes with the shortest operation time required to have the agent device 200 perform the related operation can be set to the highest, and the priority of similar shapes can be lowered as the operation time increases.

[0123] For example, it is preferable to have the agent device 200 react to actions that are likely to catch the occupant's eye. For instance, the operation of the ETC barrier is instantaneous, and therefore this action is likely to catch the occupant's eye. Thus, the operation of the ETC barrier can be given high priority. On the other hand, actions such as wipers and turn signals are performed repeatedly over a long period of time. Therefore, actions such as wipers and turn signals can be given low priority. Furthermore, for actions such as wipers and turn signals, the related actions should be stopped after a certain number of related actions have been performed. If only one similar shape is detected in step S504, that similar shape is extracted.

[0124] In step S506, the related operation determination unit 102 determines the related operation of the agent device 200 based on the similar operation modes with high priority extracted in step S505. For example, based on the similar operation modes, related operations as shown in Figures 5 to 11 are determined.

[0125] In step S520, the control unit 103 executes related operation processing to control the related operation of the agent device 200 based on the related operation determined in step S506. This related operation processing will be described in detail with reference to Figure 13.

[0126] As shown in Figure 13, in step S521, the control unit 103 determines whether the operation of the similar shape extracted in step S504 is perceptible to the occupant. The state of being perceptible means a state in which the operation of the similar shape is recognizable to the occupant, such as when the similar shape remains in the occupant's field of vision or when the sound emitted by the similar shape is audible to the occupant. Whether or not it is perceptible can be determined based on information acquired from the image acquisition unit 130 and the sound acquisition unit 140. If the operation of the extracted similar shape is perceptible to the occupant, the process proceeds to step S522. On the other hand, if the operation of the extracted similar shape is not perceptible to the occupant, the related operation processing is terminated.

[0127] In step S522, the control unit 103 determines whether the operation mode of the similar shape extracted in step S504 is continuing. If the extracted operation mode of the similar shape is continuing, the process proceeds to step S523. On the other hand, if the extracted operation mode of the similar shape is not continuing, the operation of the related operation process is terminated. That is, if the extracted operation mode of the similar shape has finished or stopped, it is considered unlikely that the occupants will notice the operation of the related operation of that operation mode even if the agent device 200 performs the operation. Therefore, if the extracted operation mode of the similar shape has finished or stopped, the agent device 200 is not allowed to perform the operation of the related operation of that operation mode and the process is terminated. However, in the case of an operation mode that is performed for only a moment, the determination process in step S522 may be omitted. For example, the operation of the ETC barrier is performed for only a moment, but because it has a high priority, the determination process in step S522 can be omitted.

[0128] In step S523, the control unit 103 executes a process to cause the agent device 200 to perform the related operation determined in step S506. That is, the control unit 103 outputs a control signal to the agent device 200 that instructs the related operation determined in step S506.

[0129] In step S524, the control unit 103 determines whether or not to instruct the agent device 200 to perform a specific process. Here, the specific process is a process that causes the control processing load of the agent device 200 to exceed a threshold, or a pre-configured task. The criteria for this determination may be the same as the determination in step S501. The pre-configured task is a process that executes a more conspicuous operation, an important event, etc. If the agent device 200 is instructed to perform a specific process, the process proceeds to step S525. On the other hand, if the specific process is not instructed, the process proceeds to step S526.

[0130] In step S525, the control unit 103 terminates the related operations of the agent device 200 and instructs the agent device 200 to execute a specific process. That is, the control unit 103 outputs a control signal to the agent device 200 instructing it to terminate the related operations of the agent device 200 and execute a specific process. In this way, if a specific process that exceeds the control processing load of the agent device 200 is instructed to the agent device 200 after the related operations of the agent device 200 have started, the related operations are terminated taking into consideration the computational load of the agent device 200.

[0131] In step S526, the control unit 103 determines whether the duration of the related operation started in step S513 has exceeded a first predetermined time from its start. If the first predetermined time has elapsed from the start of the related operation, the process proceeds to step S527. On the other hand, if the first predetermined time has not elapsed from the start of the related operation, the process proceeds to step S528.

[0132] Here, the first predetermined time is a period of time sufficient for driver D1 to recognize the related operation, and can be set appropriately based on experimental data, etc. For example, a value of about 2 to 3 seconds can be set as the first predetermined time. If the operation of the object targeted by the related operation is completed in an instant, the related operation may be executed for the duration of that operation and then terminated. This makes it possible to execute the related operation for an appropriate period of time, sufficient for driver D1 to recognize it, even when performing a related operation on an object that operates for a relatively long time.

[0133] In step S527, the control unit 103 causes the agent device 200 to perform an action that indicates boredom for a predetermined period of time. This action is achieved, for example, by interrupting a related action that is currently being performed. In this way, by interrupting the related action after a certain period of time has elapsed, the agent device 200 can express that it has grown bored with the related action. This makes it possible to cause the agent device 200 to perform an action that fosters affection. The predetermined period in step S527 is a period of time that allows the driver D1 to recognize the related action that indicates boredom, and can be set appropriately based on experimental data, etc. For example, a value of about 1 to 2 seconds can be set as this predetermined period.

[0134] In step S528, the control unit 103 determines whether or not the occupants of vehicle C1 saw the related operation of the agent device 200. Specifically, the control unit 103 acquires image information acquired by the image acquisition unit 220 of the agent device 200 and detects the direction of the gaze of the driver D1 or other occupants included in this image information. Next, the control unit 103 determines whether or not the detected direction of the gaze matches (or approximately matches) a specific direction connecting the eyes of the occupant being detected and the agent device 200. If it is determined that the detected direction of the gaze matches (or approximately matches) the specific direction, the control unit 103 determines that the occupants of vehicle C1 saw the related operation of the agent device 200. On the other hand, if it is determined that the detected direction of the gaze does not match (or approximately matches) the specific direction, the control unit 103 determines that the occupants of vehicle C1 did not see the related operation of the agent device 200. In this way, it is determined whether or not the occupants of vehicle C1 saw the related operation of the agent device 200. For face detection and gaze detection, known image recognition technologies can be used. If the occupants of vehicle C1 see the related operations of agent device 200, the process proceeds to step S529. On the other hand, if the occupants of vehicle C1 do not see the related operations of agent device 200, the process proceeds to step S530.

[0135] In step S529, the control unit 103 causes the agent device 200 to perform an embarrassing action for a predetermined time. This embarrassing action is performed by interrupting any related actions that are currently being performed. For example, an action like the one shown in Figure 3(G) can be performed as an embarrassing action. This makes it possible to cause the agent device 200 to perform an action that fosters affection. The predetermined time in step S529 is a time sufficient for the driver D1 to recognize the embarrassing action, and can be set appropriately based on experimental data, etc. For example, a value of about 1 to 2 seconds can be set as this predetermined time.

[0136] In the example shown in Figure 13, if the related actions of the agent device 200 are observed by the occupant of vehicle C1, the agent device 200 is instructed to perform an embarrassed action. However, if the related actions of the agent device 200 are observed by the occupant of vehicle C1, the speed or amount of the related actions of the agent device 200 may be increased. This allows the agent device 200 to express an exaggerated embarrassed action because its related actions have been observed by the occupant of vehicle C1. Furthermore, if the related actions of the agent device 200 are observed by the occupant of vehicle C1, the related actions have been recognized by the occupant of vehicle C1, and the related actions may be terminated and the system may transition to driver assistance control.

[0137] In step S530, the control unit 103 determines whether the duration of the related operation started in step S523 has exceeded a second predetermined time from its start. If the second predetermined time has elapsed from the start of the related operation, the process proceeds to step S531. On the other hand, if the second predetermined time has not elapsed from the start of the related operation, the process returns to step S524. The second predetermined time is a period of time sufficient for the driver D1 to recognize the related operation, and can be set appropriately based on experimental data, etc. For example, the second predetermined time can be set to a value of approximately 2 to 3 seconds, similar to the first predetermined time.

[0138] In step S531, the control unit 103 terminates the related operation of the agent device 200 and sets it to increase the amount of operation of the agent device 200 during the next related operation. This setting is notified to the related operation determination unit 102. When this setting is made, the related operation determination unit 102 decides to increase the amount of operation of the agent device 200 above the reference value when determining the related operation next time. In this example, an example of setting to increase the amount of operation during the next related operation is shown, but the agent device 200 may be made to perform some operation during the processing of step S531. For example, it can perform an operation as shown in Figure 3(I), i.e., an operation that looks like crying. This can express that the agent device 200 is sad and crying because the related operation of the agent device 200 was not observed by the occupants of vehicle C1. It is expected that the occupants who see the agent device 200 crying in this way will feel more attached to the agent device 200.

[0139] Note that the processing steps shown in Figures 12 and 13 are examples of how to implement this embodiment, and the order of some of the processing steps may be changed, some of the processing steps may be omitted, or other processing steps may be added, as long as the embodiment is feasible.

[0140] For example, in the case of an agent device that is present but does not appear to understand the situation, it is conceivable that users may find it difficult to trust that agent device and would be reluctant to entrust it with intelligent processing. Furthermore, users may perceive that the agent device lacks autonomy. Therefore, even if the agent device has the capability to assist vehicle occupants, it may not be able to do so effectively.

[0141] On the other hand, if an agent device is too conspicuous in its presence and behavior that demonstrates an understanding of the situation, it may attract too much user attention and become a hindrance. In this case, the user may find it difficult to trust the agent device.

[0142] Therefore, in this embodiment, the agent device 200 is made to perform related actions in relation to the behavior of objects present inside or outside the vehicle C1, so as to appear to be present in the environment and to understand the situation. This increases the user's trust in the agent device 200, and it is expected that the user will want to entrust intelligent processing to the agent device 200. It is also expected that the user will perceive the agent device 200 as autonomous and develop an attachment to it. As a result, the agent device 200 can fully demonstrate its capabilities to support the vehicle occupants.

[0143] Furthermore, the related operations of the agent device 200 are performed in an inconspicuous manner, so they do not attract the user's attention excessively and do not interfere with the user's experience. For this reason, it is expected that the user will perceive the agent device 200 as highly reliable.

[0144] Thus, in this embodiment, the agent device 200 can be perceived as both autonomous and responsive by sometimes mimicking something and sometimes not, giving the user the impression of an intelligent social presence and making them want to entrust complex tasks to the agent device 200. Furthermore, because the agent device 200 not only reacts according to instructions but also possesses autonomy, it can give the user the impression that they want to entrust intelligent processing to it.

[0145] Furthermore, in this embodiment, the agent device 200 mimics characteristic changes in the surroundings of vehicle C1 (for example, the interior of the vehicle, the state of the occupants, and the view from vehicle C1) without receiving instructions from the user. As a result, the user can get the impression that the agent device 200 is autonomous and responsive.

[0146] Furthermore, it is thought that, for example, synchronicity at the appropriate time makes an entity more likely to be perceived as intelligent. Also, the impression that an entity is listening tends to make an entity more likely to be perceived as intelligent. For example, when we see dogs or babies mimicking behavior, we tend to perceive them as autonomous and intelligent beings, and our trust in them increases. Similarly, when we see agent device 200 mimicking behavior, we tend to perceive it as autonomous and intelligent beings, and our trust in it increases.

[0147] Furthermore, performing synchronization actions at the appropriate timing can enhance the mirroring effect. Similarly, the mirroring effect can be enhanced when the agent device 200 performs synchronization actions at the appropriate timing. The mirroring effect is also referred to as the synchronization effect or posture reflection.

[0148] In this embodiment, an example is shown in which the control device 100 performs the device control processing. However, all or part of the device control processing may be performed by other devices. For example, all or part of the device control processing may be performed by the agent device 200. In this case, the information processing system is composed of each device that performs part of the device control processing. Furthermore, in this embodiment, an example is shown in which the control device 100 manages the related operation information DB 150. However, the related operation information DB 150 may be managed by one or more other devices other than the control device 100, and the control device 100 may acquire the information from the related operation information DB 150 managed by the other devices and use it for device control processing.

[0149] Furthermore, some (or all) of the information processing system capable of executing the functions of the control device 100 may be provided by an application that can be provided via a predetermined network such as the Internet. This application may be, for example, SaaS (Software as a Service).

[0150] Furthermore, each process shown in this embodiment is executed based on a program that causes a computer to execute each processing procedure. For this reason, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each of these processes, and a recording medium that stores that program. For example, by performing an update process to add a new function to the control device, the program can be stored in the storage device of the control device. This makes it possible to have the updated control device perform each of the processes shown in this embodiment.

[0151] [Configuration and Effects of This Embodiment] The control method according to this embodiment is a control method for an agent device that assists in driving a vehicle C1. This control method includes a detection process (steps S503 to S505) for detecting the operating mode of an object located inside or outside the vehicle C1, and a control process (steps S506, S520) for causing the agent device 200 to perform related operations related to the operating mode of the object in a manner different from driving assistance, once the operating mode of the object has been detected.

[0152] This configuration allows the occupant of vehicle C1 to perceive both autonomy and responsiveness by having the agent device 200 perform related actions related to the movement of objects, in a manner distinct from driving assistance. This increases the occupant's trust in the agent device 200 and allows them to develop an attachment to it. Therefore, when driving assistance is performed using the agent device 200, the occupant can appropriately accept the driving assistance, thereby increasing its effectiveness.

[0153] Furthermore, in the control method according to this embodiment, in the detection process (steps S503, S504), based on the captured image of the inside or outside of the vehicle C1, similar shapes that are similar in shape to at least a part of the appearance of the agent device 200 are extracted from among the objects present inside or outside the vehicle C1, the movement of the extracted similar shapes is detected as the movement mode of the object, and in the control process (steps S506, S520), the agent device 200 is made to perform a mimicry operation that imitates the movement of the similar shapes as a related operation.

[0154] With this configuration, the agent device 200 performs mimicking movements that mimic the movements of similar shapes present inside or outside vehicle C1, making it easier for the occupants of vehicle C1 to notice these mimicking movements. As a result, the occupants of vehicle C1 can further increase their trust in the agent device 200 and develop a greater attachment to it, thereby appropriately accepting the driver assistance provided by the agent device 200 and improving the effectiveness of that assistance.

[0155] Furthermore, in the control method according to this embodiment, if multiple similar shapes are extracted in the detection process (step S505), one similar shape is selected from among the extracted similar shapes based on a pre-set priority, and in the control process (steps S506, S520), the agent device 200 is instructed to perform a mimicry operation that imitates the movement of the selected similar shape.

[0156] In this configuration, if multiple similar shapes are extracted, one similar shape is selected based on a pre-set priority, and the agent device 200 is instructed to perform a mimicry action that imitates the movement of this similar shape. This makes it easier for the occupants of vehicle C1 to notice, for example, the mimicry action with a higher priority.

[0157] Furthermore, in the control method according to this embodiment, the agent device 200 includes a rod-shaped first movable part 202 and a second movable part 203 (an example of a movable part), and the similar shape is at least one of the following: rod-shaped members mounted on the vehicle C1 (for example, wipers W1, W2 (see Figure 2), shift lever, wiper lever, turn signal lever), rod-shaped members located outside the vehicle C1 (for example, bar B1 (see Figure 6), the arm of guide U1 (see Figure 7)), and the arm of an occupant riding in the vehicle C1, and in the control processing (steps S506, S520), the first movable part 202 and the second movable part 203 are made to perform imitation movements to mimic the movement of the similar shape.

[0158] This configuration allows for rapid extraction of similar shapes by pre-setting specific similar shapes.

[0159] Furthermore, in the control method according to this embodiment, in the control processing (steps S526, S527), the related operation is terminated after a first predetermined time (an example of a predetermined time) has elapsed since the start of the related operation. For example, it is possible to represent an operation that interrupts the related operation that is currently being executed and becomes bored.

[0160] With this configuration, the agent device 200 can express that it has grown tired of the associated operation by terminating the operation after a certain period of time has elapsed. This makes it possible to get the agent device 200 to perform actions that foster a sense of attachment.

[0161] Furthermore, in the control method according to this embodiment, in the control processing (steps S524, S525), after starting the related operation, if a control instruction for operation support to the agent device 200 (for example, an instruction for a specific process) is detected, the related operation is terminated.

[0162] With this configuration, if a control instruction for operation support is issued after the related operation of the agent device 200 has started, causing the control processing load of the agent device 200 to exceed a threshold, the related operation can be terminated taking into account the computational load of the agent device 200. This makes it possible to perform operation support by the agent device 200 more appropriately.

[0163] In this embodiment, the agent device 200 includes a face portion (for example, a portion including the eye portions E1 and E2) and is installed on the dashboard 2 of the vehicle C1. Furthermore, the control method according to this embodiment includes a determination process (step S528) that detects the gaze of the occupant of the vehicle C1 and determines whether or not the occupant of the vehicle C1 has looked at the agent device 200, and in the control process (step S529), if it is detected that the occupant of the vehicle C1 has looked at the agent device 200 after the related operation has started, the related operation is terminated.

[0164] With this configuration, if the relevant operation of the agent device 200 is detected by the occupant of vehicle C1, the relevant operation can be terminated and the agent device 200 can perform driving assistance. This increases the reliability of the agent device 200, fosters a sense of attachment to the agent device 200, and enables the agent device 200 to perform driving assistance more appropriately.

[0165] Furthermore, in the control method according to this embodiment, if, after the related operation has started, the control process (step S529) detects that the occupant of vehicle C1 has looked at the agent device 200, the related operation is terminated and the agent device 200 is instructed to perform an action to express embarrassment with its face. For example, an action like the one shown in Figure 3(G) can be performed as an action to express embarrassment.

[0166] This configuration makes it possible to have the agent device 200 perform actions that foster affection, further increasing the degree of affection towards the agent device 200.

[0167] Furthermore, in the control method according to this embodiment, in the control processing (steps S530, S531), if it is detected that the occupant of vehicle C1 has seen the agent device 200 after the related operation has started, the next related operation is set to an operation amount greater than a preset operation amount (an example of a control amount).

[0168] With this configuration, if the related operation of the agent device 200 is not observed by the occupants of vehicle C1, the amount of operation for the next related operation can be increased to make the related operation more likely to be observed by the occupants of vehicle C1.

[0169] Furthermore, in the control method according to this embodiment, in the control processing (steps S502, S520), if the computational load of the operation support performed by the agent device 200 is lower than a standard, the agent device 200 is instructed to perform the related operation.

[0170] This configuration allows for priority execution of operational support by the agent device 200, and reduces the computational processing load on the agent device 200.

[0171] Furthermore, the control method according to this embodiment further includes a load detection process (step S501) that detects the load of the driver of vehicle C1 based on the driving conditions of the driver of vehicle C1, and in the control process (step S520), if the load of the driver of vehicle C1 is lower than a preset standard, the agent device 200 is instructed to perform the relevant operation.

[0172] With this configuration, the relevant action is performed only when the driver of vehicle C1 is under low load, making the relevant action more easily observable by the occupants of vehicle C1.

[0173] Furthermore, the control device 100 is a control device for agent equipment that assists the driver of the vehicle C1. The control device 100 includes a detection unit 101 that detects the operating mode of an object located inside or outside the vehicle C1, and a control unit 103 that, when the operating mode of an object is detected, causes the agent equipment 200 to perform related operations related to the operating mode of the object in a manner different from driving assistance.

[0174] This configuration allows the occupant of vehicle C1 to perceive both autonomy and responsiveness by having the agent device 200 perform related actions related to the movement of objects, in a manner distinct from driving assistance. This increases the occupant's trust in the agent device 200 and allows them to develop an attachment to it. Therefore, when driving assistance is performed using the agent device 200, the occupant can appropriately accept the driving assistance, thereby increasing its effectiveness.

[0175] Although embodiments of the present invention have been described above, these embodiments only represent a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. [Explanation of Symbols]

[0176] 10 Information processing system, 20 Network, 100 Control device, 101 Detection unit, 102 Related operation determination unit, 103, 230 Control unit, 104, 240 Storage unit, 105 Communication unit, 110 Vehicle information acquisition unit, 120 Sensors, 130, 220 Image acquisition unit, 140, 210 Sound acquisition unit, 250 Sound output unit, 260 Display unit, 270 Drive unit

Claims

1. A control method for an agent device that assists the driver of a vehicle and has a movable rod-shaped part, A detection process for detecting the operating mode of an object located inside or outside the vehicle, If the operating mode of the object is detected, the control process includes causing the agent device to perform an action related to the operating mode of the object, In the detection process, based on the captured images taken of the interior or exterior of the vehicle, a similar shape is extracted from among the objects present inside or outside the vehicle that is similar in shape to at least a part of the appearance of the agent device, and is at least one of the following: a rod-shaped member mounted on the vehicle, a rod-shaped member present outside the vehicle, and the arm of an occupant riding in the vehicle. The movement of the extracted similar shape is then detected as the motion of the object. In the control process, the agent device is instructed to perform a mimicry operation as a related operation, causing the movable part to mimic the movement of the similar shape. Control method.

2. A control method for an agent device that assists the driver of a vehicle, A detection process for detecting the operating mode of an object located inside or outside the vehicle, If the operating mode of the object is detected, the control process includes causing the agent device to perform an action related to the operating mode of the object, In the control process described above, after the related operation has started, if a control instruction is detected to cause the agent device to execute a pre-set specific process that supports the operation, the related operation is terminated. Control method.

3. A method for controlling an agent device installed on the dashboard of a vehicle, which assists the driver of a vehicle, A detection process for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control process is performed to cause the agent device to execute related operations related to the operating mode of the object. The process includes detecting the gaze of the occupant of the vehicle and determining whether or not the occupant of the vehicle looked at the agent device, In the control process described above, if it is detected that the occupant of the vehicle has seen the agent device after the associated operation has started, the associated operation is terminated. Control method.

4. A method for controlling an agent device that assists the driver of a vehicle, has a face, and is installed on the dashboard of the vehicle, A detection process for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control process is performed to cause the agent device to execute related operations related to the operating mode of the object. The process includes detecting the gaze of the occupant of the vehicle and determining whether or not the occupant of the vehicle looked at the agent device, In the control process described above, if it is detected that the occupant of the vehicle has looked at the agent device after the associated operation has started, the associated operation is terminated and the agent device is instructed to perform an action expressing embarrassment with its face. Control method.

5. A method for controlling an agent device installed on the dashboard of a vehicle, which assists the driver of a vehicle, A detection process for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control process is performed to cause the agent device to execute related operations related to the operating mode of the object. The process includes detecting the gaze of the occupant of the vehicle and determining whether the occupant of the vehicle is looking at the agent device, In the control process described above, if it is not detected that the occupant of the vehicle has seen the agent device after the associated operation has started, the next associated operation will be controlled by a control amount greater than the preset control amount. Control method.

6. A control method for an agent device that assists the driver of a vehicle, A detection process for detecting the operating mode of an object located inside or outside the vehicle, If the operating mode of the object is detected, the control process includes causing the agent device to perform an action related to the operating mode of the object, In the control process, if the computational load of the processing performed by the agent device to support the operation is lower than a standard, the agent device is instructed to perform the related operation. Control method.

7. A control method for an agent device that assists the driver of a vehicle, A detection process for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control process is performed to cause the agent device to execute related operations related to the operating mode of the object. This includes a load detection process that detects the load on the driver of the vehicle based on the driving conditions of the driver of the vehicle, In the control process described above, if the load on the vehicle driver is lower than a preset standard, the agent device is instructed to perform the relevant operation. Control method.

8. A control method according to claim 1, In the detection process, if multiple similar shapes are extracted, one similar shape is selected from among the extracted multiple similar shapes based on a predetermined priority. In the control process, the agent device is instructed to perform the imitation operation that mimics the movement of the selected similar shape. Control method.

9. A control method according to any one of claims 1 to 8, In the control process described above, the related operation is terminated after a predetermined time has elapsed since the start of the related operation. Control method.

10. A control device for an agent device that assists the driver of a vehicle and has a rod-shaped movable part, A detection unit for detecting the operating mode of an object located inside or outside the vehicle, The system includes a control unit that, when the operating mode of the object is detected, causes the agent device to perform an action related to the operating mode of the object, The detection unit, based on the captured image of the inside or outside of the vehicle, extracts a similar shape from among the objects inside or outside the vehicle that is similar in shape to at least a part of the appearance of the agent device, and is at least one of the following: a rod-shaped member mounted on the vehicle, a rod-shaped member located outside the vehicle, and the arm of a passenger in the vehicle, and detects the movement of the extracted similar shape as the motion of the object. The control unit causes the agent device to perform a mimicry operation as a related operation, which causes the movable part to mimic the movement of the similar shape. Control device.

11. A control device for an agent device that assists the driver of a vehicle, A detection unit for detecting the operating mode of an object located inside or outside the vehicle, The system includes a control unit that, when the operating mode of the object is detected, causes the agent device to perform an action related to the operating mode of the object, After the associated operation has started, the control unit detects a control instruction to cause the agent device to execute a pre-set specific process that supports the operation, and terminates the associated operation. Control device.

12. A control device for an agent device installed on the dashboard of a vehicle, which assists the driver of the vehicle's occupants in driving, A detection unit for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control unit causes the agent device to perform an action related to the operating mode of the object, The system includes a determination unit that detects the gaze of the vehicle occupant and determines whether or not the vehicle occupant has looked at the agent device, If the control unit detects that a vehicle occupant has seen the agent device after starting the associated operation, it terminates the associated operation. Control device.

13. A control device for an agent device that assists the driver of a vehicle, has a face unit, and is installed on the dashboard of the vehicle, A detection unit for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control unit causes the agent device to perform an action related to the operating mode of the object, The system includes a determination unit that detects the gaze of the vehicle occupant and determines whether or not the vehicle occupant has looked at the agent device, If the control unit detects that a vehicle occupant has looked at the agent device after initiating the associated operation, it terminates the associated operation and causes the agent device to perform an action expressing embarrassment with its face. Control device.

14. A control device for an agent device installed on the dashboard of a vehicle, which assists the driver of the vehicle's occupants in driving, A detection unit for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control unit causes the agent device to perform an action related to the operating mode of the object, The system includes a determination unit that detects the gaze of the vehicle occupant and determines whether or not the vehicle occupant is looking at the agent device, If the control unit does not detect that the occupant of the vehicle has seen the agent device after the associated operation has started, it sets the next associated operation to a control amount greater than a preset control amount. Control device.

15. A control device for an agent device that assists the driver of a vehicle, A detection unit for detecting the operating mode of an object located inside or outside the vehicle, The system includes a control unit that, when the operating mode of the object is detected, causes the agent device to perform an action related to the operating mode of the object, The control unit causes the agent device to perform the relevant operation when the computational amount of the processing performed by the agent device to support the operation is lower than a standard. Control device.

16. A control device for an agent device that assists the driver of a vehicle, A detection unit for detecting the operating mode of an object located inside or outside the vehicle, When the operating mode of the object is detected, a control unit causes the agent device to perform an action related to the operating mode of the object, The vehicle includes a load detection unit that detects the load on the driver of the vehicle based on the driver's driving conditions, The control unit causes the agent device to perform the relevant operation when the load on the vehicle driver is lower than a preset standard. Control device.

Images