Control device and control method

Abstract

To provide a control apparatus and a control system for realizing an environment which can be used by a user smoothly with comfort in an interactive system.SOLUTION: In an in-vehicle system, a control apparatus includes: a display device 14 serving as a first display unit; and an operation unit 20 arranged in a position facing the first display unit, equipped with a cylindrical enclosure 201, having a second display unit on a front face 202 which is a surface of the enclosure opposite the first display unit, and configured to receive an input of a user. Accordingly, the user can operate the operation unit 20 while grasping information displayed on the operation unit 20 serving as the second display unit which is different from the display device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a control device and a control method.

Background Art

[0002] Currently, there are interactive systems with an agent function in various scenarios such as smart speakers and in-vehicle devices. In the agent function, for example, services provided by an assistant that employs Artificial Intelligence (AI) are offered.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Patent Document 8

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, conventional interactive systems only provide mechanical responses or feedback to user input such as speech, and the variety of feedback expressions is insufficient. As a result, it is difficult for users to understand the status of the interactive system. Consequently, conventional interactive systems do not provide a sufficient environment for users to use them comfortably and smoothly. One aspect of the disclosed embodiment is to create an environment in the interactive system that allows users to use it comfortably and smoothly. [Means for solving the problem]

[0005] Embodiments of the disclosure are illustrated by a control device. This control device comprises a first display unit and a cylindrical housing located opposite the first display unit, having a second display unit on the opposite side of the housing, and an operation unit for receiving user input. [Effects of the Invention]

[0006] The control unit of this in-vehicle system can provide an environment in which users can comfortably and smoothly utilize the interactive system. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is an example diagram illustrating the display device and control panel of an in-vehicle system. [Figure 2] Figure 2 illustrates the configuration of a mechanism that moves the position of the control unit in the left-right direction of the screen and a mechanism that changes the tilt of the control unit. [Figure 3] Figure 3 illustrates the operation of the mechanism for changing the tilt of the control unit. [Figure 4] Figure 4 is an exploded perspective view illustrating the structure of the control panel. [Figure 5] Figure 5 illustrates the effect of the bundle of optical fibers within the control unit. [Figure 6] Figure 6 illustrates the effects of modifying the operating section. [Figure 7]FIG. 7 is a diagram illustrating the hardware configuration of the in-vehicle system. [Figure 8] FIG. 8 is an example of symbols used in anthropomorphic communication provided by the in-vehicle system through an operation unit. [Figure 9] FIG. 9 shows the display of the operation unit when the in-vehicle system guides a user for fingerprint-based personal authentication. [Figure 10] FIG. 10 is a diagram illustrating the procedure of feedback from a user to the anthropomorphic in-vehicle system. [Figure 11] FIG. 11 is a diagram illustrating the procedure for a user to operate the anthropomorphic in-vehicle system. [Figure 12] FIG. 12 is a diagram illustrating the extension process of the recommendation function by the in-vehicle system. [Figure 13] FIG. 13 is a diagram illustrating the authentication process in the in-vehicle system. [Figure 14] FIG. 14 is a diagram illustrating the details of the agent process. [Figure 15] FIG. 15 is a diagram illustrating the details of the dialogue process. [Figure 16] FIG. 16 is a diagram illustrating the details of the result output. [Figure 17] FIG. 17 is a diagram illustrating the details of the image correspondence process.

DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, referring to the drawings, an in-vehicle system 100 according to an embodiment and a control method executed by the in-vehicle system 100 will be described. The in-vehicle system 100 has, for example, a Center Information Display (CID) disposed in front of the driver's seat of a vehicle, and as an example of a control device, executes processing as an interactive system for a user.

[0009] <Configuration> First, Figures 1 to 7 illustrate the configuration of each part provided by the in-vehicle system 100. Figure 1 is an example of the display device 14 and operation unit 20, which serve as the CID of the in-vehicle system 100. In the examples shown in Figures 1 to 3, the display device 14 is embedded in the vehicle's instrument panel 40.

[0010] The operating unit 20 comprises a housing 201 and a support mechanism 30 that supports the housing 201. The housing 201 is, for example, cylindrical and has two end faces perpendicular to the central axis. The housing 201 also has openings formed in the two end faces, and the edges of the openings and the cross-section of the housing 201 are ring-shaped. The operating unit 20 may be cylindrical or rectangular. However, the shape of the operating unit 20 is not limited to cylindrical, rectangular, etc., and may be various shapes such as cylindrical shapes with taper or gradient, rectangular shapes, polyhedra, or shapes including quadratic curved surfaces. The operating unit 20 is provided in a position facing the display device 14, which is the first display unit. Here, the back surface of the housing 201 facing the display device 14 is called the facing surface, and the opposite surface of the facing surface is called the front surface 202. Within the housing 201, a bundle of optical fibers 203 (see Figures 4 to 6) is arranged in a direction connecting the opening on the front surface 202 and the opening on the opposite surface (back surface), forming a light guide path that projects the light of the display device 14A screen onto the front surface 202. Therefore, the front surface 202 of the operation unit 20 acts as a display unit, and a portion of the screen 14A facing the opposite surface (back surface), which is the opposite side of the front surface 202, can be projected onto the front surface 202. In other words, in the in-vehicle system 100, the display device 14 functions as a first display unit. The front surface 202 of the operation unit 20 functions as a second display unit. Thus, the operation unit 20 can be said to have a second display unit on the front surface 202, which is the surface of the housing 201 opposite to the opposite surface. Note that the operation unit 20 itself may also have the function of a display device. Specifically, a display device such as a display may be provided inside the housing 201, and the display may be wired via a support mechanism 30 or the like.

[0011] Furthermore, the operating unit 20 is supported vertically from below by the support mechanism 30. The support mechanism 30 also supports the operating unit 20 so that its position can be moved horizontally in the left-right direction of the screen 14A and the tilt of the front surface 202 can be changed. As described above, the display device 14 is the vehicle's instruction manual. Since it is embedded in the instrument panel 40, the left-right direction of the screen 14A is the left-right direction with respect to the direction of travel of the vehicle.

[0012] Figure 2 illustrates the configuration of a support mechanism 30 that moves the position of the operating unit 20 in the left-right direction of the screen 14A and a mechanism that changes the inclination of the front surface 202 of the operating unit 20. As shown in Figure 2, the support mechanism 30 has a support base 31 and a slide actuator 33 that can move the support base 31 left and right on both sides of the screen 14A. The slide actuator 33 also has a belt 32B to which the support base 31 is fixed inside the housing of the display device 14 (or inside the instrument panel 40), and a pair of left and right rollers 32L and 32R that can wind up or unwind the belt 32B. In Figure 2, the support shaft 35 that supports the support base 31 and the rotary actuator 34 that rotates the support mechanism 30 around the support shaft 35 are also shown. In Figure 2, the belt 32B and the pair of rollers 32L and 32R are located inside the housing of the display device 14 (or inside the instrument panel 40), so they are drawn with dotted lines.

[0013] A pair of rollers 32L and 32R are rotatable by a motor. A motor may be provided for each of the pair of rollers 32L and 32R, or a single motor may drive the pair of rollers 32L and 32R individually via a clutch mechanism. In either configuration, the motor allows the belt 32B to move between the rollers 32L and 32R by rotating one of the pair of rollers 32L and 32R while keeping the other free-rotating in a neutral state. Since the support base 31 is fixed to the belt 32B inside the housing of the display device 14 (or inside the instrument panel 40), the support base 31 moves left and right on the screen 14A of the display device 14 as the belt 32B moves in the left-right direction of the vehicle.

[0014] The configuration of the slide actuator 33, which moves the position of the operating unit 20 in the left-right direction of the screen 14A, is not limited to a pair of rollers 32L, 32R, a belt 32B, and a motor. For example, a rail may be provided instead of the belt 32B, and the axle may rotate along the rail to move the position of the operating unit 20 in the left-right direction of the screen 14A. Alternatively, the position of the operating unit 20 may be moved by a screw mechanism such as a ball screw and a motor. The slide actuator 33 can be considered an example of a movement mechanism that can move the operating unit 20 in the left-right direction of a vehicle on which the in-vehicle system 100 is mounted.

[0015] The in-vehicle system 100 detects the speech of the occupant user and moves the position of the control unit 20 to approach the user who is speaking. This movement of the control unit 20 simulates the action of moving closer to the user who is speaking. As a result, the user who is speaking feels as if they are talking to a person and gets the impression that it is friendly and approachable.

[0016] Figure 3 illustrates the operation of a mechanism for changing the inclination of the front surface 202 of the operating unit 20. As shown in Figure 3, the support base 31 is pivotally supported by a pivot shaft 35 so as to be rotatable. The support mechanism 30 also has a rotary actuator 34 that rotates the support base 31 in a vertical plane around the pivot shaft 35. The rotary actuator 34 includes, for example, a spring that biases the support base 31 toward the display device 14 around the pivot shaft 35, and a drive unit 34A that drives the support base 31 toward the display device 14, against the elastic force of the spring. In Figure 3, the drive unit 34A and the driven portion of the support base 31 driven by the drive unit 34A are located inside the housing of the display device 14 (or inside the instrument panel 40), and are therefore depicted with dotted lines. The drive unit 34A is, for example, a piezoelectric element whose dimensions change due to the piezoelectric effect. However, the rotary actuator 34 is not limited to a configuration including a spring and a piezoelectric element. For example, the rotary actuator 34 does not have to have a spring. That is, the rotary actuator 34 may cause the support base 31 to rotate and return around the pivot shaft 35 by changing its dimensions (expanding and contracting) due to the piezoelectric effect or the like. Instead of a magnetic element, an electromagnet may be used to rotate the support base 31 using magnetic repulsion. Alternatively, the rotary actuator 34 may use a belt and motor to wind the belt so that the support base 31 rotates around the pivot shaft 35. In other words, the rotary actuator 34 may rotate the support base 31 around the pivot shaft 35 away from the display device 14 by winding the belt, against the elastic force of the spring. The rotary actuator 34 can be considered an example of an angle-changing mechanism that can change the orientation of the operating unit 20.

[0017] The in-vehicle system 100 detects the speech of the user, who is an occupant of the vehicle (driver and passenger), and changes the tilt of the front surface 202 of the control unit 20 vertically in response to the user's speech. By changing the tilt of the control unit 20 in this way, the in-vehicle system 100 simulates a nodding motion in accordance with the content of the user's speech. As a result, the user speaking feels as if they are talking to a person and gets the impression that it is friendly and approachable.

[0018] Figure 4 is an exploded perspective view illustrating the structure of the control unit 20. As explained in Figures 1 to 3, the control unit 20 is mounted with its back surface facing the display device 14. As shown in Figure 4, the control unit 20 includes a housing 201, a fingerprint sensor 22, a touch sensor 23, a dial switch 25, a push switch 26, and an optical fiber 203. The control unit 20 displays the processing status of the in-vehicle system 100 on its front surface 202, which serves as a second display unit, and accepts user input via the fingerprint sensor 22, touch sensor 23, dial switch 25, push switch 26, etc. In addition to the configuration illustrated in Figure 4, as explained in Figure 7, the control unit 20 also includes a thermosensor 24A, an infrared sensor 24B, and an illuminance sensor 27.

[0019] The housing 201 is cylindrical, forming a ring-shaped side surface in cross-section of the operating section 20, and also forming ring-shaped edges at both ends of the cylindrical shape. As a result, the housing 201 forms an outer shell with ring-shaped openings at both ends. The housing 201 may also be decorated and is sometimes called a decorative section.

[0020] A flat, circular touch sensor 23 is provided on the front surface 202 of the operating unit 20. The touch sensor 23 has a configuration similar to that of a touch panel provided in a typical mobile device, such as a smartphone. For example, the touch sensor 23 has a first electrode row arranged parallel to a first direction (the X-axis direction) on a plane, and a second electrode row arranged parallel to a second direction (the Y-axis direction) perpendicular to the first direction. The touch sensor 23 uses the two electrode rows to detect the movement of the user's finger in X-axis and Y-axis coordinate values ​​based on the change in capacitance from the user's finger. However, the touch sensor 23 is not limited to a capacitive type sensor; for example, it may be a pressure-sensitive sensor. Although omitted in Figure 4, a transparent protective film may be provided on the top surface (front surface 202 side) of the touch sensor 23.

[0021] The fingerprint sensor 22 is, for example, a capacitive sensor. That is, the fingerprint sensor 22 has electrodes similar to those of the touch sensor 23, but in a narrower area and at narrower intervals than the touch sensor 23. Therefore, the fingerprint sensor 22 forms a finer-grained, higher-resolution image than the touch sensor 23. In other words, the fingerprint sensor 22 forms a fingerprint image by measuring the capacitance based on the unevenness of the fingerprint that contacts the front surface 202 of the operation unit 20 and forming a two-dimensional image. However, the fingerprint sensor 22 is not limited to a capacitive type. For example, the fingerprint sensor 22 may be an array of Charge Coupled Device (CCD) or Metal-Oxide-Semiconductor (MOS) image sensors. If the fingerprint sensor 22 is an array of image sensors, it generates a fingerprint image in the same way as the camera 21.

[0022] The dial switch 25 outputs a physical quantity or electrical signal corresponding to the amount of rotation when the user rotates the housing 201 clockwise or counterclockwise. (Automotive system 100) The device obtains a physical quantity value or an electrical signal strength value from the dial switch 25, corresponding to the amount of rotation. The dial switch 25 has a cylindrical shape that slides inward within the cylindrical inner surface of the housing 201.

[0023] When the user rotates the housing 201 clockwise or counterclockwise, the contact point between the inner surface of the housing 201 and the dial switch 25 changes, and the electrical resistance, voltage, etc. between the terminals drawn from the dial switch 25 changes. The in-vehicle system 100 detects the amount and direction of rotation of the housing 201 from the changes in the electrical resistance, voltage, etc. between the terminals of the dial switch 25. For example, when the housing 201 rotates clockwise, the electrical resistance, voltage, etc. increase (or decrease) from the initial values. Also, for example, when the housing 201 rotates counterclockwise, the electrical resistance, voltage, etc. decrease (or increase) from the initial values. Furthermore, the in-vehicle system 100 may determine that the switch is off if the change in the electrical resistance, voltage, etc. between the terminals output from the dial switch 25 is within a reference value, and that the switch is on if the change exceeds the reference value. As described above, the operating unit 20 can accept the user's rotation operation. That is, the operating unit 20 may be physically rotatable. However, the dial switch 25 may be omitted. If the dial switch 25 is omitted, the housing 201 of the operating unit 20 does not need to rotate. If the housing 201 of the operating unit 20 does not rotate, the operating unit 20 may detect rotation by the user's hand using the touch sensor 23.

[0024] The dial switch 25 is not limited to detecting changes in electrical resistance, voltage, etc., between terminals. For example, a general rotary encoder may be used as the dial switch 25. For example, the dial switch 25 may have multiple windows arranged in a circumferential direction on its cylindrical side wall, and the presence or absence of light transmission between the inside and outside (inner surface of the housing 201) of the cylindrical side wall of the dial switch 25 can be determined. For example, an LED or other light-emitting part can be provided on the inner surface of the housing 201, and a photodiode or other light-receiving part can be provided on the inside of the cylindrical side wall of the dial switch 25 to receive light from the light-emitting part that has passed through the windows. By arranging the windows in stages such as large, medium, and small, the amount of light or the reception time received by the light-receiving part changes. The direction of rotation can also be determined by whether the direction of this change is large, medium, or small.

[0025] When a push switch 26 is pressed, its contacts close, an electrical signal flows between the contacts, and the user's operation is detected. In this embodiment, four push switches 26 are provided between the dial switch 25 and the ring 204 located below it. The four push switches 26 are positioned in the central locations of the arcs that divide the ring 204 into four sections, for example, in the sector-shaped arcs that divide the circular ring 204 at a 90-degree angle at its center. Therefore, when a user evenly presses the housing 201, all four push switches 26 can be simultaneously switched from off to on. Furthermore, when a user presses the housing 201 at a location corresponding to the middle of one of the two divided sections of the ring 204, two push switches 26 can be switched from off to on. Additionally, when a user presses the housing 201 at a location corresponding to one of the four 90-degree sector-shaped sections that divide the ring 204, only one push switch 26 can be switched from off to on. The number of push switches 26 is not limited to four; it can be one to three, or five or more. If the number of push switches 26 is less than four, the in-vehicle system 100 can construct a simple user interface. If the number of push switches 26 is five or more, the in-vehicle system 100 can detect user operations with greater precision.

[0026] The optical fibers 203 are inserted in a bundle within the cylindrical space inside the housing 201, between the opening on the back of the control unit 20 and the opening on the front 202. The bundle of optical fibers 203 projects the portion of the screen 14A facing the back of the control unit 20 onto the front 202 of the control unit 20.

[0027] Figure 5 illustrates the effect of the bundle of optical fibers 203 within the control unit 20. As described in Figure 4, the optical fibers 203 project the portion of screen 14A facing the back of the control unit 20 onto the front surface 202 of the control unit 20. As a result, the user perceives the display of screen 14A as appearing on the front surface 202 of the control unit 20, improving the sense of interaction with the anthropomorphic in-vehicle system 100, which is the agent. In other words, the communication capabilities of the in-vehicle system 100 as an agent are improved.

[0028] Figure 6 illustrates the effect of a modified version of the operating unit 20 shown in Figure 5. Specifically, in Figure 6, the bundle of optical fibers 203 gradually expands from the back surface to the front surface 202 of the operating unit 20, forming a tapered shape. That is, in the bundle of optical fibers 203, the gaps between the optical fibers 203 are narrow and dense in the part closer to the back surface of the operating unit 20, while the gaps between the optical fibers 203 widen in the part closer to the front surface 202 of the operating unit 20. Alternatively, the outer diameter of each optical fiber 203 may be thin in the part closer to the back surface of the operating unit 20, and the outer diameter of the optical fibers 203 may be thicker in the part closer to the front surface 202 of the operating unit 20. With such a bundle of optical fibers 203, the number of pixels per unit area of ​​the image on the front surface 202 decreases compared to the screen 14A of the display device 14, resulting in a decrease in fineness (resolution), but the entire image is enlarged. In other words, the portion of the screen 14A facing the back of the control unit 20 is enlarged and projected onto the front 202. Therefore, the communication capabilities or presentation of the in-vehicle system 100 as an agent are further improved.

[0029] Figure 7 illustrates the hardware configuration of the in-vehicle system 100. The in-vehicle system 100 includes a control unit 10 and external devices connected to the control unit 10 via an interface (I / F). The control unit 10 includes a CPU 11 and a main memory 12. The external devices include an external memory device 13, a display device 14, an input unit 15, a communication device 16, a microphone 17, a camera 21, a speaker 28, and an operation unit 20.

[0030] The CPU 11 executes computer programs loaded into the main memory 12 to provide the functions of the in-vehicle system 100. The CPU 11 is also called a processor. However, the CPU 11 is not limited to a single processor and may be in a multi-processor configuration. Furthermore, the CPU 11 may be a single processor connected via a single socket and may be in a multi-core configuration. In addition, at least some of the processing of the in-vehicle system 100 may be provided by dedicated processors such as a Digital Signal Processor (DSP), Graphics Processing Unit (GPU), numerical processing processor, vector processor, image processing processor, Application Specific Integrated Circuit (ASIC), etc. At least a portion of the system 100 may be a dedicated large-scale integration (LSI) such as a Field-Programmable Gate Array (FPGA), or other digital circuits. At least a portion of the Tem 100 may include analog circuitry.

[0031] The main memory 12, also simply called memory, stores computer programs executed by the CPU 11, data processed by the CPU 11, and so on. The main memory 12 can be Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), etc.

[0032] The external storage device 13 is used, for example, as a storage area that supplements the main memory 12, and stores computer programs executed by the CPU 11, data processed by the CPU 11, etc. The external storage device 13 is a hard disk drive, a solid state drive (SSD), etc. Furthermore, the in-vehicle system 100 may be provided with a drive device for a removable storage medium. The removable storage medium may be, for example, a Blu-ray disc, a Digital Versatile Disc (DVD), These include Compact Discs (CDs), flash memory cards, etc.

[0033] The display device 14 is, for example, a liquid crystal display, an electroluminescent panel, etc. The input unit 15 is, for example, a keyboard, a pointing device, etc. In this embodiment, a touch panel is exemplified as a pointing device. The communication device 16 exchanges data with other devices on the wireless network. The wireless network is, for example, a mobile phone network. The microphone 17 acquires the user's voice. The microphone 17 is not limited to one, and multiple microphones may be provided. The microphone 17 may also be provided within the operation unit 20.

[0034] The camera 21 may be located near the operating unit 20, or it may be located anywhere in the vehicle interior. Like the fingerprint sensor 22, the camera 21 has an array of CCD or MOS image sensors and transmits an image, which is the data of the pixel array, to the CPU 11. The CPU 11 acquires video from the camera 21 at a predetermined frame rate, for example, from 10 frames / second to 120 frames / second. In this embodiment, an image containing a series of multiple frames is referred to as video. Also, what is contained in a single frame is simply referred to as an image.

[0035] The speaker 28 has, for example, a piezoelectric element (piezo element). The speaker 28 receives an electrical signal, converts the electrical signal into mechanical vibrations of the piezoelectric element (piezo element), and generates sound in front of the display device 14. The speaker 28 may be placed near the operating unit 20. Alternatively, the speaker 28 may be placed anywhere in the vehicle interior.

[0036] As shown in Figures 2, 3, and 4, the operating unit 20 includes a fingerprint sensor 22, a touch sensor 23, a thermosensor 24A, an infrared sensor 24B, a dial switch 25, a push switch 26, an illuminance sensor 27, a slide actuator 33, and a rotary actuator 34. Each of these parts of the operating unit 20 is connected to the CPU 11 via an interface I / F. Of these, the fingerprint sensor 22, touch sensor 23, dial switch 25, and push switch 26 have already been explained in Figure 4. The slide actuator 33 and rotary actuator 34 have been explained in Figures 2 and 3.

[0037] The thermosensor 24A is also called a thermal camera. The thermosensor 24A measures the temperature distribution of an object from the intensity of infrared or far-infrared radiation in an image using infrared or far-infrared radiation. The thermosensor 24A outputs infrared image data, for example, through an array of pixel sensors. The pixel sensors are made of materials such as InGaAs, GaAs, AL, GaAs, InAs, GaSb, etc. The device is a compound semiconductor or a photodiode, phototransistor, or other element formed by stacking such a compound semiconductor or a compound semiconductor. However, instead of an array of compound semiconductor elements, an array of thermocouples (thermopile) may be used as the pixel sensor. The thermosensor 24A may have a filter that selects infrared light within a specific wavelength range from the reflected light from the object. The thermosensor 24A may be placed on the edge of the operating unit 20 (housing 201). Alternatively, the thermosensor 24A may be placed separately from the operating unit 20, but near the operating unit 20. Furthermore, the thermosensor 24A may be placed at a distance from the operating unit 20, at any location inside the vehicle.

[0038] The infrared sensor 24B measures the temperature of an object based on the infrared intensity emitted from the object. However, unlike the thermosensor 24A, the infrared sensor 24B measures the temperature based on the amount of infrared radiation from a specific location on the object, rather than the temperature distribution of the object. The infrared sensor 24B may, for example, have a thermopile. However, the infrared sensor 24B may also utilize the pyroelectric effect of ferroelectric ceramics. Alternatively, the infrared sensor 24B may use elements such as photodiodes and phototransistors made of compound semiconductors, similar to the thermosensor 24A. The infrared sensor 24B may be placed on the edge of the operating unit 20 (housing 201). Alternatively, the infrared sensor 24B may be separated from the operating unit 20 and placed near the operating unit 20. Alternatively, the infrared sensor 24B may be placed at any location within the vehicle interior, separated from the operating unit 20.

[0039] The illuminance sensor 27 uses elements such as a photodiode, phototransistor, or photoresistor to measure the illuminance near the front surface 202 of the operating unit 20. The illuminance sensor 27 may also be placed on the edge of the operating unit 20 (housing 201).

[0040] <Example Function> The functions of the in-vehicle system 100 are illustrated with reference to Figures 8 to 12. Figure 8 shows an example of symbols used by the in-vehicle system 100 for anthropomorphic communication provided by the control unit 20. Here, a symbol is a symbolic graphic representation (mark), also called a pattern, to represent the state of the in-vehicle system 100. The in-vehicle system 100 communicates its state to the user in an easy-to-understand manner using graphic representations or the movement of the control unit 20. For example, the control unit 20 projects the screen portion of the display device 14, which is facing the back of the control unit 20, onto the front surface 202 of the control unit 20 using a bundle of optical fibers 203, and operates as a display. In Figure 8, the pattern projected by the control unit 20 onto the front surface 202 is illustrated along with the state of the in-vehicle system 100. The control unit 20 is movable in the left-right direction of the screen 14A by a slide actuator 33, as illustrated in Figure 2. In this case, the CPU 11 of the in-vehicle system 100 can recognize the position of the operating unit 20 from the control amount instructed to the slide actuator 33. Alternatively, for example, the CPU 11 can recognize the position of the operating unit 20 from the capacitance distribution between the back surface of the operating unit 20 and the touch panel of the display device 14. Therefore, the CPU 11 of the in-vehicle system 100 can display a symbol like the one in Figure 8 on the screen 14A of the display device 14 at a position facing the back surface of the moved operating unit 20.

[0041] Figure 8A is a symbol indicating that the in-vehicle system 100 is listening to the user's speech. Figure 8B is a symbol indicating that the in-vehicle system 100 is responding to the user and speaking (also called talking). Figure 8C is a symbol indicating that the in-vehicle system 100 is performing some kind of processing in response to the user's speech. In this embodiment, performing some kind of processing is also referred to as "thinking." "Thinking" can also be, for example, a pre-processing step to make some kind of decision or judgment. Figure 8D is a symbol indicating that the in-vehicle system 100 has made some kind of decision as a result of some kind of processing. Figure 8E is a symbol indicating that the in-vehicle system 100 has issued a warning as a result of detecting some kind of event or as a result of some kind of processing. Figure 8F is a symbol indicating that the in-vehicle system 100 is monitoring the user's speech to collect further information.

[0042] Figure 9 shows the display on the control panel 20 when the in-vehicle system 100 guides the user for fingerprint authentication. When the in-vehicle system 100 performs fingerprint authentication, it displays a symbol resembling a fingerprint on the control panel 20. As explained in Figures 4 and 7, the control panel 20 is equipped with a fingerprint sensor 22. When the user touches their finger to the front surface 202 of the control panel 20 where the symbol is displayed, the in-vehicle system 100 acquires an image of the user's fingerprint via the fingerprint sensor 22 of the control panel 20 and identifies the individual user. Once the individual is identified, the in-vehicle system 100 provides recommendations to the identified user that reflect each user's personal settings or preferences. In this embodiment, recommendations refer to suggestions, suggestions, and responses to questions from the user that the in-vehicle system 100 provides to the user. In the recommendation system, the in-vehicle system 100 displays the symbol exemplified in Figure 8 on the control unit 20 and operates the control unit 20 as an agent accompanied by simulated actions (approaching motion, nodding motion) exemplified in Figures 2 and 3. Furthermore, the in-vehicle system 100 provides services to identified users that reflect each user's individual settings or preferences, and realizes an in-vehicle space. For example, it may recommend music that the user is presumed to like, set the cabin lighting to a level that the user is presumed to like, and automatically adjust the seat position to suit the user's physique. The in-vehicle system 100 may perform personal authentication when, for example, it starts up from a stopped state, or when the vehicle is stopped and it recognizes that the user has boarded. Furthermore, after startup, the in-vehicle system 100 may perform personal authentication as appropriate according to user operations.

[0043] Figure 10 illustrates the procedure for user feedback to the anthropomorphic in-vehicle system 100. As explained in Figures 4 and 7, the operation unit 20 is equipped with a touch sensor 23. When the in-vehicle system acts as an agent and makes a good recommendation, the user performs a stroking motion on the ring portion of the operation unit 20 (the front edge portion of the housing 201). The in-vehicle system 100 then detects the user's action via the touch sensor 23 and receives positive feedback of "good" from the user. Upon receiving positive feedback, the in-vehicle system 100 functions as an agent and outputs a happy action through the display of a symbol and a sound. For example, it displays a heart mark and outputs a cheerful melody. By repeating this process of making recommendations and receiving feedback on those recommendations, the in-vehicle system 100 can improve its subsequent recommendations to be more user-friendly and accurate.

[0044] Figure 11 illustrates the procedure for a user to operate the anthropomorphic in-vehicle system 100. As explained in Figures 4 and 7, the operating unit 20 is equipped with an illuminance sensor 27. When a user covers the operating unit 20 with their hand, for example, the in-vehicle system 100 detects that the operating unit 20 is covered via the illuminance sensor 27. The in-vehicle system 100 then reduces or mutes the sound output from the speaker 28. Thus, the in-vehicle system 100 reduces or mutes the sound output from the speaker 28 with a single touch from the user. For example, Figure 11 illustrates a change in sound volume from 50 to 0. The user's action of covering the operating unit 20 with their hand is reminiscent of covering a person's mouth, and the in-vehicle system 100 realizes the acceptance of anthropomorphic operation. The in-vehicle system 100 may also detect that the operating unit 20 is covered via the camera 21.

[0045] Figure 12 illustrates the extended processing of the recommendation function by the in-vehicle system 100. As explained in Figures 4 and 7, the operation unit 20 is equipped with a camera 21, a thermosensor 24A, and an infrared sensor 24B. The in-vehicle system 100 grasps the conditions of the driver and passengers through one or more of the camera 21, thermosensor 24A, and infrared sensor 24B. The in-vehicle system 100 then automatically adjusts the in-vehicle environment to an appropriate environment based on the grasped conditions and makes appropriate recommendations.

[0046] For example, if the in-vehicle system 100 detects from the image of the camera 21 that the passenger is asleep, it reduces the volume of the sound output from the speaker 28 to a level lower than normal. At this time, the in-vehicle system 100 may also inform the driver that the volume has been reduced because the passenger is asleep. Through such communication, the in-vehicle system 100, as an agent, can demonstrate its ability to communicate with the driver and passengers, fostering a friendly atmosphere.

[0047] Similarly, the in-vehicle system 100 detects if the passenger is asleep and, upon detecting fatigue in the driver, prompts the driver to take a break. Through these processes, the in-vehicle system 100, acting as an agent, avoids situations prone to accidents, enhances operational safety, and builds trust with the driver.

[0048] Furthermore, the in-vehicle system 100 acquires the temperature distribution in a two-dimensional plane as seen from the operating unit 20 inside the vehicle using a thermosensor 24A. In addition, the in-vehicle system 100 uses the image from the camera 21. The system extracts the user's image from the image. The in-vehicle system 100 then estimates the temperature near each part of the user's body by superimposing the acquired temperature distribution with the extracted user image. Based on the estimated temperature distribution, the in-vehicle system 100 determines appropriate air conditioner settings and controls the air conditioner. For example, if the system estimates that the user's lower body is cold and their upper body is hot, the system controls the air conditioner so that cool air is delivered only to the upper body. The in-vehicle system 100 may also perform such control based on the temperature distributions from the driver's seat and the passenger seat. <Example of processing> An example of processing by the in-vehicle system 100 will be explained with reference to Figures 13 to 17. The CPU 11 of the in-vehicle system 100 executes the processing shown in Figures 13 to 17 using a computer program that has been loaded into the main memory 12 in an executable format.

[0049] Figure 13 illustrates the authentication process in the in-vehicle system 100. The authentication process in Figure 13 is executed, for example, after the in-vehicle system 100 has been started and the user in the driver's seat is recognized.

[0050] Here, the in-vehicle system 100 determines, for example, whether or not to perform fingerprint authentication (S1). For example, if the in-vehicle system 100 displays a fingerprint authentication symbol on the operation unit 20, and the user presses their finger against the front surface 202 of the operation unit 20, and the fingerprint sensor detects the user's fingerprint, fingerprint authentication is performed (S2). If fingerprint authentication is not performed for a preset time or longer (NO in S1), the in-vehicle system 100 performs other authentication processing (S3). For example, the user selects "Other Authentication" from the graphics object, icon, menu, etc. displayed on the screen 14A using the input unit 15 (touch panel, etc.) on the display device 14 (NO in S1). Other authentication methods include, for example, facial recognition, iris recognition, password entry, etc.

[0051] Then, once the authentication process in S2 or S3 is complete, the in-vehicle system 100 performs user-specific environment settings and recommendation characteristics (preferences) settings (S4). Specifically, the in-vehicle system 100 suggests playing music that matches the user's preferences. The in-vehicle system 100 also sets the user's preferred interior lighting, a seat position suitable for the user, etc. Furthermore, in subsequent agent processing (S5), the in-vehicle system 100 refers to the individual user's characteristics and performs recommendation processing to match the user's characteristics.

[0052] Figure 14 illustrates the details of agent processing (S5 in Figure 13). The CPU 11 of the in-vehicle system 100 executes the process shown in Figure 14 as an example of agent processing. In this process, the in-vehicle system 100 detects the occurrence of an event (S51). When the occurrence of an event is detected (YES in S51), the in-vehicle system 100 executes the processes from S52 onwards for each event that occurred.

[0053] For example, the in-vehicle system 100 determines whether the event that occurred is the detection of a user's utterance (S52). If the event that occurred is the detection of a user's utterance (YES in S52), the in-vehicle system 100 performs dialogue processing (S53). The in-vehicle system 100 detects the user's utterance, for example, using the microphone 17. After the dialogue processing is completed, the in-vehicle system 100 proceeds to control S5E.

[0054] If the event that occurred is not the detection of a user's speech (NO in S52), the in-vehicle system 100 determines whether the event that occurred is the detection of an image change (S54). Here, the detection of an image change means, for example, that the movement or physical condition of an occupant (driver or passenger) is detected from the image of the camera 21, or that a change in the passenger's level of alertness is detected, and the details will be explained in accordance with Figure 17. If the event that occurred is the detection of an image change (YES in S54), the in-vehicle system 100 performs image processing (S55). After the image processing is completed, the in-vehicle system 100 proceeds to control S5E.

[0055] If the event that occurred is not the detection of an image change (NO in S54), the in-vehicle system 100 determines whether the event that occurred is the detection of a temperature change (S56). A temperature change is, for example, a change in the body temperature of an occupant. Alternatively, the temperature change may be a change in the temperature inside the vehicle. If the event that occurred is the detection of a temperature change (YES in S56), the in-vehicle system 100 performs temperature control processing (S57). In the temperature control processing, as explained in Figure 12, the in-vehicle system 100 controls at least one of the temperature inside the vehicle, the airflow direction, and the airflow volume by the air conditioner according to the temperature distribution of the driver's seat user's body, the passenger seat user's body, or the temperature inside the vehicle. Note that the processing in S57 is not limited to cases where the event that occurred is the detection of a temperature change as described above. The in-vehicle system 100 may, for example, periodically perform the temperature control processing in S57. After the completion of the temperature control processing, the in-vehicle system 100 proceeds to control S5E.

[0056] If the event that occurred is not a detection of a temperature change (NO in S56), the in-vehicle system 100 determines whether the event that occurred is a detection of a warning state (S58). If the event that occurred is a detection of a warning state (YES in S58), the in-vehicle system 100 executes a warning response process (S59). In the warning response process, the in-vehicle system 100 displays a warning symbol on the operation unit 20 and notifies the user of the occurrence of a warning state with voice or warning sound from the speaker 28. Warning states include, for example, malfunctions in the drive system such as the engine, battery, or motor, or situations in which a collision is expected via external cameras, etc. If a collision is expected, the in-vehicle system 100 executes collision avoidance processing such as activating the brakes. After the warning response process is completed, the in-vehicle system 100 proceeds to control S5E.

[0057] If the event that occurred is not a detection that requires attention (NO in S58), the in-vehicle system 100 determines whether the event that occurred is a timer event (S58). A timer event occurs when a specified time set in advance on the timer has elapsed. In this embodiment, for processes that need to be executed periodically, such as the process of detecting changes in the body temperature of the occupants, a specified time is set in the timer in advance as a timer event. When the specified time has elapsed and a timer event occurs, the in-vehicle system 100 executes the periodic process set in association with the occurred timer event. It may be possible to set the execution of different periodic processes in association with multiple specified times. If the event that occurred is a timer event (YES in S5A), the in-vehicle system 100 executes the periodic process (S5B). In the periodic process, the in-vehicle system 100 executes a process that is specified to be executed periodically. For example, the in-vehicle system 100 may measure the temperature distribution in the room, reset the air conditioner, or periodically speak to the driver. After the completion of the periodic process, the in-vehicle system 100 proceeds to control S5E.

[0058] If the event that occurred is not a timer event (NO in S5A), the in-vehicle system 100 determines whether the event that occurred is a detection of a processing request (S5C). If the event that occurred is a detection of a processing request (YES in S5C), the in-vehicle system 100 executes the requested processing (S5D). Processing requests are detected, for example, by the operation of a push switch 26 on the operation unit 20, a graphic object, icon, or menu via the input unit 15. Examples of processing requests include changing the music to be played, changing the video output by the display device 14, setting or changing the navigation destination, changing the temperature and lighting settings inside the vehicle, changing the seat position or posture, etc. After processing is completed, the in-vehicle system 100 proceeds to control S5E.

[0059] If the event that occurred is not a detection of a processing request (NO in S5C), the in-vehicle system 100 determines whether or not to terminate the agent processing (S5E). If the agent processing is not terminated (NO in S5E), the in-vehicle system 100 returns control to S51.

[0060] It should be noted that agent processing is not limited to that performed based on event detection in S51. For example, the in-vehicle system 100 may periodically execute each of the processes in Figure 14. For example, the in-vehicle system 100 may execute the processes from S52 onward according to the frame period of the images acquired by the camera 21. That is, the in-vehicle system 100 may consider the acquisition of an image from the camera 21 as the occurrence of an event. In that case, the in-vehicle system 100 should determine the presence or absence of each event from S52 to S5E for each frame period and execute the processing corresponding to the event that is determined to have occurred. Therefore, in one frame period, the occurrence of multiple events exemplified in Figure 14 may be detected, and multiple processes may be executed.

[0061] Figure 15 illustrates the details of the dialogue processing (S53 in Figure 14). Here, for example, the in-vehicle system 100 first determines whether the user who spoke to it is in the driver's seat or the passenger seat, based on the video from the camera 21. Then, it recognizes the position of the user and uses the slide actuator 33 to bring the operation unit 20 closer to the user (S531). If the in-vehicle system 100 has multiple microphones 17 in the vehicle interior, it may also determine whether the user who spoke to it is in the driver's seat or the passenger seat based on the intensity of the user's voice detected by the multiple microphones 17. In the process of S531, the in-vehicle system 100 identifies the position of the moved operation unit 20 and recognizes the position of the screen 14A corresponding to the back surface of the operation unit 20. This allows the in-vehicle system 100 to project a symbol from the screen 14A of the display device 14 onto the front surface 202 of the operation unit 20.

[0062] The in-vehicle system 100 then determines whether the user is continuing to speak and whether it is currently listening to their speech (S532). If the in-vehicle system 100 is currently listening (YES in S532), it displays the listening status as a symbol on the operation unit 20, performs a nodding motion using the rotary actuator 34, and performs speech recognition (S533). Here, the nodding motion is an action that changes the tilt of the front surface 202 of the operation unit 20 in the vertical up and down direction. Changing in the vertical up and down direction means, for example, that the normal of the front surface 202 of the operation unit 20 moves up and down in a vertical plane. The process in S533 can be said to be an example of changing the orientation of the operation unit 20 in the vertical direction within a predetermined angular range using the rotary actuator 34, which is an angle changing mechanism, in accordance with the user's speech. Here, the angle range is, for example, the angle at which the front surface 202 of the operating unit 20 faces forward in the normal direction (horizontal direction), and the angle at which it appears to be nodding (for example, 10 to 90 degrees downward from the horizontal). Here, 90 degrees downward is the angle at which the normal of the front surface 202 of the operating unit 20 is vertically downward. After voice recognition, the in-vehicle system 100 determines whether information processing is necessary (S534). If the in-vehicle system 100 decides to perform information processing (YES in S534), it displays a thinking state as a symbol on the operating unit 20 and performs the information processing (S535). After the information processing is completed, the in-vehicle system 100 proceeds to control S53C.

[0063] If the in-vehicle system 100 does not perform information processing based on the determination in S534 (NO in S534), it determines whether or not speech is required (S536). If speech is required (YES in S536), the in-vehicle system 100 displays the status of "talking" as a symbol on the operation unit 20 and performs speech processing. After the speech processing is completed, the in-vehicle system 100 proceeds to control S53C.

[0064] If the in-vehicle system 100 determines in S536 that speech processing is unnecessary (NO in S536), it determines whether monitoring is necessary (S538). If monitoring is necessary (YES in S538), the in-vehicle system 100 operates the monitoring status using a symbol. The information is displayed in section 20, and information collection and monitoring are performed. After the completion of information collection and monitoring, the in-vehicle system 100 proceeds to control S53C.

[0065] If the in-vehicle system 100 determines in S538 that monitoring is unnecessary (NO in S538), it determines whether a decision can be made (S53A). If a decision can be made (YES in S53A), the in-vehicle system 100 displays the decision status as a symbol on the operation unit 20 and outputs the result, i.e., performs a recommendation (S53B). After the recommendation, the in-vehicle system 100 proceeds to control S53C.

[0066] The in-vehicle system 100 then determines whether or not to terminate the dialogue process (S53C). If the in-vehicle system 100 does not terminate the dialogue process (NO in S53C), it returns control to S532. On the other hand, if the in-vehicle system 100 terminates the dialogue process (YES in S53C), it returns the position of the operation unit 20 to its original predetermined position using the slide actuator 33 (S53D) and terminates the process. For example, if voice cannot be obtained from the user for a predetermined period (timeout period) after the completion of S53B, the in-vehicle system 100 terminates the dialogue process. Note that each process in Figure 15 is an example of how the control unit 10, consisting of the CPU 11 and main memory 12, outputs the processing status of the in-vehicle system 100 via the operation unit.

[0067] Figure 16 illustrates the details of the output of the result (S53B in Figure 15). Here, the in-vehicle system 100 provides recommendations according to the user's environment settings and recommendation characteristics (preferences) (S53B1). The in-vehicle system 100 then determines whether or not positive feedback has been received from the user via the touch sensor 23 within a predetermined period after the recommendation is provided (S53B2). In this embodiment, positive feedback is, for example, an operation such as stroking the ring-shaped portion on the front of the housing 201 of the operation unit 20. Such positive feedback is an example of a predetermined operation.

[0068] When the in-vehicle system 100 recognizes a predetermined action, which is positive feedback (YES in S53B2), it displays a heart symbol on the display device 14 so that it is projected onto the operation unit 20. The in-vehicle system 100 also responds to the user with a sound that expresses joy (S53B3). The system then saves the user's request, the user's or vehicle's status, sensor information regarding the surrounding environment, and the recommendation content obtained through the processes shown in Figures 14 and 15 to the database of the external storage device 13. The database information may also be stored in a database on an external server connected via the communication device 16. The information stored in these databases will be referenced as correct examples for the next recommendation. Furthermore, if the in-vehicle system 100 is equipped with a deep learning system or is linked to a deep learning system, the correct examples stored in the database will be used as training data for the deep learning system.

[0069] The in-vehicle system 100 terminates the output of the result if it does not recognize positive feedback (NO in S53B2) and after processing S53B3.

[0070] Figure 17 illustrates the details of the image processing (S55 in Figure 14). Here, the in-vehicle system 100 determines whether the driver's physical condition, such as fatigue level, has changed based on the image from the camera 21 or the information from the thermosensor 24A (S551). If the driver's physical condition has changed (YES in S551), the in-vehicle system 100 outputs information, video, or sound corresponding to the driver's physical condition (S552). The information including video or sound can also be called multimedia information. For example, if it is determined that the driver is fatigued, the in-vehicle system 100 recommends a suitable stop nearby in the direction the vehicle is traveling and encourages the driver to take a break. After that, the in-vehicle system 100 proceeds to control S553.

[0071] If the passenger's level of alertness changes (YES in S553), the in-vehicle system 100 outputs information, video, or sound corresponding to the passenger's state. For example, if it is determined that the passenger has fallen asleep, the in-vehicle system 100 reduces the sound output from speaker 28. The in-vehicle system 100 also stops the video output from display device 14 (S554). After that, the in-vehicle system 100 proceeds to control S555.

[0072] If the illuminance detected by the illuminance sensor 27 falls below a predetermined limit (YES in S555), the in-vehicle system 100 reduces or mutes the sound output from the speaker 28 (S556). The predetermined value to be compared with the detected illuminance can be determined in advance from, for example, the illuminance measurement when the user covers the operation unit 20 with their hand, and stored in the main memory 12 or external memory 13 of the in-vehicle system 100. Through the process in S556, the in-vehicle system 100 can treat the operation unit 20 as a human mouth and behave anthropomorphically as if a human mouth were being covered. In addition, when the in-vehicle system 100 reduces the sound output, it may reduce the level of the sound output in accordance with the decrease in the measured illuminance. Note that the judgment in S555 is not limited to cases where the detected illuminance falls below a predetermined limit. For example, the in-vehicle system 100 may execute the process in S556 when it detects that the detected illuminance value, which is the amount of light incident on the operating unit 20, has decreased by more than a predetermined limit.

[0073] Furthermore, the decrease in illuminance recognized by the operation unit 20 may be determined without using the illuminance sensor 27. For example, the in-vehicle system 100 may determine that the user has covered the operation unit 20 with their hand based on the relative brightness change for each frame in the video acquired by the camera 21. Alternatively, the in-vehicle system 100 may integrate the brightness of each image for each frame included in the video acquired by the camera 21. That is, the in-vehicle system 100 may integrate the brightness of each pixel for each image (or average the brightness values ​​of the pixels) and use this as the brightness detection value detected by the operation unit 20. As for the predetermined value to be compared with the brightness detection value, for example, the in-vehicle system 100 may measure it using the same procedure with the user covering the operation unit 20 with their hand and determine the predetermined value in advance. The in-vehicle system 100 may then store the predetermined brightness value in the main memory 12 or external memory 13 of the in-vehicle system 100. After processing in S556, the in-vehicle system 100 proceeds with control in S557.

[0074] If the video acquired by camera 21 contains an image code (YES in S557), the in-vehicle system 100 decodes the destination information from the image code and sets it in the car navigation system (S558). Here, the car navigation system is a system that is installed in the in-vehicle system 100 or a system that works in conjunction with the in-vehicle system 100. The image code may be either a one-dimensional code or a two-dimensional code, such as a QR code (registered trademark) or a barcode. After processing in S558, the in-vehicle system 100 proceeds to control in S559.

[0075] If it is determined that the video acquired by camera 21 contains a forgotten item (YES in S559), the in-vehicle system 100 notifies the user of the forgotten item via speaker 28 (S55A). Whether or not a forgotten item is included in the video can be determined by comparing the images of each frame included in the video taken when the driver and passengers get out of the vehicle with reference images taken in advance of the empty vehicle interior. The reference images are taken in advance by camera 21 and stored in the database of the external storage device 13. After processing in S55A, the in-vehicle system 100 proceeds to control S55B.

[0076] If the video acquired by camera 21 indicates a situation requiring attention (YES in S55B), the in-vehicle system 100 executes attention response processing (S55C). A situation requiring attention is, for example, if the user's body temperature acquired from the thermosensor 24A is higher than the normal value. This includes cases where the user's seatbelt position differs from the normal position. Note that the normal body temperature is stored in the database of the external storage device 13 beforehand. Similarly, images showing the seatbelt in the normal position are also stored in the database of the external storage device 13 beforehand. Therefore, the in-vehicle system 100 can determine whether a situation warrants attention by comparing the user's body temperature or an image of the user wearing the seatbelt with the information or images stored in the database. The in-vehicle system 100 then terminates the image matching process.

[0077] <Effects of the Embodiment> As described above, the in-vehicle system 100 of this embodiment has an operation unit 20. The in-vehicle system 100 has a display function as a second display unit on the front surface 202 of the housing 201, which is the surface opposite the display device 14 as the first display unit, and also has an operation unit 20 that accepts user input. Therefore, the user can operate the operation unit 20 while understanding the information displayed on the operation unit 20, which is a second display unit different from the display device 14. For this reason, the in-vehicle system 100 can provide a user interface with enhanced operability. The in-vehicle system 100 has a CPU 11 and a main memory 12 that execute agent processing as the control unit 10 and agent processing unit as illustrated in Figure 14. In agent processing, the in-vehicle system 100 executes the display of various symbols illustrated in Figure 8, the response to feedback illustrated in Figure 10, the response in dialogue processing illustrated in Figure 15, or the image-corresponding processing illustrated in Figure 17. Through these displays, responses, dialogue processing, and image processing, the in-vehicle system 100 outputs its own processing status to the user, thereby improving the user's interactive experience. In other words, the in-vehicle system 100, acting as an agent, can smoothly interact with the user. Furthermore, the user can feel more like they are interacting with the agent.

[0078] Furthermore, the in-vehicle system 100 has a slide actuator 33, as shown in Figure 2, which is a movement mechanism that allows the operation unit 20 to move in the left-right direction of the vehicle on which the in-vehicle system 100 is mounted. The in-vehicle system 100 detects the start of a user's speech via a microphone 17, which is a sound sensor, and determines the user who is speaking via a camera 21, which is an image sensor, etc., on the operation unit 20. The in-vehicle system 100 then moves the operation unit 20 using the slide actuator 33 to approach the user who is speaking. In this way, the in-vehicle system 100 simulates the action of moving closer to the user who is speaking. As a result, the in-vehicle system 100 can give the user who is speaking the feeling that they are talking to a person, and give a friendly impression.

[0079] Furthermore, the in-vehicle system 100 has a rotary actuator 34 as an angle-changing mechanism that can change the orientation of the control unit 20, as shown in Figure 3. The in-vehicle system 100 detects the start of user speech via the microphone 17 and uses the rotary actuator 34 to change the orientation of the control unit 20 vertically within a predetermined angle range in accordance with the user's speech. By changing the tilt of the control unit 20 in this way, the in-vehicle system 100 simulates a nodding motion in accordance with the content of the user's speech. As a result, the in-vehicle system 100 can give the user the feeling that they are talking to a person, creating a friendly impression.

[0080] Furthermore, as illustrated in Figure 11, if a user covers the control unit 20 with their hand, the in-vehicle system 100 detects that the control unit 20 is covered, for example, via the illuminance sensor 27. When the in-vehicle system 100 detects that the amount of light incident on the control unit 20 falls below a predetermined limit or decreases by more than a predetermined limit, it reduces or mutes the sound output from the speaker 28. The user's action of covering the control unit 20 with their hand is reminiscent of covering one's mouth, and the in-vehicle system 100 realizes the acceptance of anthropomorphic operation. Alternatively, the in-vehicle system 100 may determine that the user has covered the control unit 20 with their hand based on the relative change in brightness frame by frame in the video acquired by the camera 21. Nevertheless, the in-vehicle system 100 can behave anthropomorphically, as if the control unit 20 were a human mouth and a human mouth were being covered. The user can directly operate the in-vehicle system 100, or at least have the sensation of directly operating it.

[0081] Furthermore, the in-vehicle system 100 has a fingerprint sensor 22 and authenticates the user using fingerprint authentication as illustrated in Figure 9. This allows the in-vehicle system 100 to easily guide the user and identify the individual user. The in-vehicle system 100 can also provide the identified user with services that reflect each user's personal settings or preferences, and can create a comfortable in-vehicle environment. In addition, the in-vehicle system 100 can provide recommendations tailored to the authenticated user.

[0082] Furthermore, the in-vehicle system 100 detects positive feedback (e.g., a stroking motion) as a predetermined user action from the touch sensor 23 on the operation unit 20, as illustrated in Figure 10, in response to the provided recommendation. The in-vehicle system 100 then outputs a response using at least one of a symbol, such as a smart mark, and a sound expressing joy. Thus, the in-vehicle system 100 can achieve close communication with the user while they are speaking. Additionally, the in-vehicle system 100 can steadily accumulate correct examples to refer to for future recommendations based on its interaction with the user. The user can directly operate the in-vehicle system 100, or at least have the sensation of directly operating it.

[0083] Furthermore, if the in-vehicle system 100 detects that the passenger is asleep based on detection data from the camera 21 of the control unit 20, it reduces the sound output of the in-vehicle system 100 to a predetermined reference value (for example, a pre-set value). Also, if the in-vehicle system 100 detects that the passenger is asleep based on detection data from the image sensor of the control unit 20, it stops the video output of the in-vehicle system 100. In other words, the in-vehicle system 100 can adjust and output information, video, or sound according to the state of the passenger. The predetermined reference value may be stored in the external storage device 13 in advance as a default value by the in-vehicle system 100. Alternatively, the predetermined reference value may be stored in the external storage device 13 in advance according to the preferences of each user authenticated by the in-vehicle system 100 in the S2 process.

[0084] Furthermore, the in-vehicle system 100, based on detection data from the camera 21 on the control unit 20, detects the driver's fatigue and recommends suitable places to stop and encourages the driver to take a break. The in-vehicle system 100 can provide information, images, or sounds that are appropriate to the driver's condition.

[0085] Furthermore, the in-vehicle system 100 detects the body temperature distribution of the driver's seat user, the passenger seat user, or the interior temperature distribution of the vehicle based on the detection data from the thermosensor 24A, which is a temperature sensor provided in the control unit 20. The in-vehicle system 100 controls the air conditioner according to these temperature distributions. Therefore, as an agent, the in-vehicle system 100 can appropriately adjust the air conditioner based on the occupant's condition or the environment of the vehicle interior.

[0086] Furthermore, if the in-vehicle system 100 detects an image code in the data detected by the camera 21 on the operation unit 20, it decodes destination information from the image code and sets the decoded destination information in the navigation system. Therefore, the in-vehicle system 100 enables easy destination setting, for example, when an image code is included in a book, flyer, or website searched by the user.

[0087] Furthermore, the in-vehicle system 100 detects items left behind when at least one of the driver's seat user and the passenger seat user gets out of the vehicle, based on detection data from the camera 21 provided in the control unit 20. The system then notifies the user of the presence of a forgotten item inside or outside the vehicle. Specifically, the in-vehicle system 100 notifies the user of the presence of a forgotten item via speaker 28. This reduces the likelihood of the user forgetting something.

[0088] Furthermore, the control unit 20 is provided with an optical fiber 203 between its back surface and its front surface 202, which projects the image displayed on the screen facing the back surface onto the front surface 202 of the control unit 20. As a result, the user perceives the display on screen 14A as appearing on the front surface 202 of the control unit 20, improving the sense of interaction with the anthropomorphic in-vehicle system 100, which is the agent.

[0089] <Variation> In the above embodiment, a computer system that performs processing as an agent was illustrated using the in-vehicle system 100 as an example. However, the configuration of the above embodiment is not limited to in-vehicle systems. That is, the configuration of the above embodiment can be applied to various systems that function as interactive systems, such as general personal computers, multimedia terminals in convenience stores, and game consoles.

[0090] <Computer-readable recording medium> A program that enables a computer or other machine or device (hereinafter referred to as "computer, etc.") to perform any of the above functions can be recorded on a recording medium that the computer, etc. can read. By having the computer, etc. read and execute the program on this recording medium, it can be made to provide that function.

[0091] Here, a recording medium that can be read by a computer refers to a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical means and can be read by a computer. Examples of such recording media that can be removed from a computer include flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, Blu-ray discs, DATs, 8mm tapes, and memory cards such as flash memory. In addition, recording media that are fixed to a computer include hard disks and ROMs (read-only memory). Furthermore, SSDs (Solid State Drives) can also be used as a recording medium that can be removed from a computer. It can also be used as a recording medium fixed to a surface, etc. [Explanation of Symbols]

[0092] 10 Control Unit 11 CPU 12 Main storage 13 External storage device 14 Display device 15 Control section 16. Communication equipment 17 Microphone 20 Control section 30 Support stand 21 Cameras 22 Fingerprint Sensors 23 Touch Sensor 24A Thermal Sensor 24B Infrared Sensor 25 Dial switch 26 Push switches 27 Illuminance sensor 28 speaker elements 33 Slide Actuator 34 Rotary actuators 40 Instrument Panel 201 cabinet 203 Optical Fiber

Claims

1. The first display unit and It is provided in a position opposite to the first display unit and has a cylindrical housing, The housing has a second display unit on the opposite side of the first display unit, and an operation unit for receiving user input, It comprises an agent processing unit that responds based on user input, The second display unit is a control device that displays the processing status of the agent processing unit.

2. The operating unit is provided with a movement mechanism that allows the operating unit to be moved in the left-right direction of the first display unit. The control device according to claim 1, wherein the control device detects the start of a user's speech via a sound sensor, determines the user who is speaking via an image sensor, and moves the operating unit by the movement mechanism to approach the user who is speaking.

3. The operating unit is provided with an angle changing mechanism that can change the orientation of the operating unit. The control device according to claim 1, wherein the control device detects the start of a user's speech via a sound sensor and changes the orientation of the operating unit vertically within a predetermined angular range using the angle changing mechanism in accordance with the user's speech.

4. The control device according to any one of claims 1 to 3, wherein when it detects that the amount of light incident on the operating section falls below a predetermined limit or decreases by more than a predetermined limit, the control device reduces or silences the sound output by the control device.

5. The control device according to any one of claims 1 to 4, wherein the control device authenticates a user and sets the environment inside the vehicle interior where the control device is installed according to the authenticated user.

6. The control device according to any one of claims 1 to 5, wherein the control device authenticates a user and performs recommendation processing according to the authenticated user.

7. The control device controls the recommendation process using the touch sensor provided in the operation unit. The control device according to claim 6, which, upon detecting a predetermined action by the user, outputs a response using at least one of video and sound.

8. The control device according to any one of claims 1 to 7, wherein, based on detection data from an image sensor, when the control device detects that a user in the passenger seat is asleep, it reduces the output of sound emitted by the control device to a predetermined reference value.

9. The control device according to any one of claims 1 to 8, wherein the control device detects that a user in the passenger seat is asleep based on detection data from the image sensor, and stops outputting video from the first display unit.

10. The control device, according to any one of claims 1 to 9, detects fatigue in the driver's seat based on data detected from an image sensor, recommends a suitable place to stop, and encourages the user to take a break.

11. The control device according to any one of claims 1 to 10, which detects the body temperature distribution of the user in the driver's seat, the body temperature distribution of the user in the passenger seat, or the temperature distribution inside the vehicle based on the temperature data detected from the temperature sensor, and controls at least one of the temperature, airflow direction, and airflow volume of the air conditioner according to the detected temperature distribution.

12. The control device according to any one of claims 1 to 11, wherein the control device decodes destination information from the image code when the detection data from the image sensor includes an image code, and sets the decoded destination information in the navigation system.

13. The control device according to any one of claims 1 to 12, wherein, based on detection data from an image sensor, the control device detects that an item has been left behind when at least one of the user in the driver's seat and the user in the passenger seat has gotten out of the vehicle, and notifies the presence of the item inside or outside the vehicle of the vehicle on which the control device is installed.

14. The control device according to any one of claims 1 to 13, wherein the back surface of the operation unit, which is the opposite side to the second display unit, faces the screen of the first display unit, and an optical fiber is provided between the back surface of the operation unit and the second display unit for projecting the image displayed on the screen facing the back surface onto the front surface of the operation unit.

15. The first display unit and It is provided in a position opposite to the first display unit and has a cylindrical housing, A control device having a second display unit on the opposite side of the housing from the first display unit, and an operation unit for receiving user input, Agent processing that responds based on user input, The process of displaying the processing status of the agent process on the second display unit, A control method for executing this.

Images