In-vehicle information processing device, information processing method, and program

Abstract

In an information processing system that allows users to view content and utilize processing and services in an interactive format, the aim is to enable easy re-execution or reuse of previously performed processes and services. [Solution] The in-vehicle information processing device includes a control unit that receives input from a vehicle occupant through a conversational user interface and executes functions corresponding to the received input. The control unit displays the conversation history as a chat-style log on a display unit, and also displays graphical user interface (GUI) components for operating the executed functions in the log.

Description

Technical Field

[0001] The present invention relates to an in-vehicle information processing apparatus, an information processing method, and a program.

Background Art

[0002] Patent Document 1 describes that in an information processing system that provides services in an interactive format with a user, the conversation history with a conversation agent is displayed in a chat format. By the way, in recent years, an information processing system that utilizes a conversation agent has been introduced in vehicles, and various services are provided in an interactive format. For example, when a user who is a passenger in a vehicle asks a conversation agent "Is there any recommended content?", the conversation agent presents candidates for music such as "The recommended content is ○○, △△, or □□. Which one do you want to play?", and when the user says "Play ○○", the content specified by ○○ is played.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, information processing systems that can be used in such a conversational format have usability issues. For example, a user may want to rewatch content that was previously specified and played by the conversational agent in such a conversational format, or they may want to watch a different piece of content (△△ or □□) from the options presented by the conversational agent. In such cases, the user would have to start the same conversational format interaction with the conversational agent again from the beginning and repeat the process. Such repetition of conversations is cumbersome, and conventional information processing systems have been inconvenient for users. The challenge of the disclosed embodiment is to make it easy to re-execute or reuse previously executed processes or services in an information processing system that allows for the execution of processes or the use of services, such as content viewing, in a conversational format. [Means for solving the problem]

[0005] An embodiment of the disclosure is exemplified by an in-vehicle information processing device. This in-vehicle information processing device includes a control unit that receives input from a vehicle occupant through a conversational user interface and executes functions corresponding to the received input. The control unit is characterized by displaying the history of the conversation as a chat-format log on a display unit, and displaying graphical user interface (GUI) components for operating the executed functions in the log. [Effects of the Invention]

[0006] This in-vehicle information processing system performs functions based on input from a conversational user interface with the vehicle's occupants, and displays the history of these conversations as a chat-style log on the display unit. Furthermore, the system displays the log including a GUI for operating the executed functions. Users can operate the log on the screen and execute previously provided functions using GUI components. In other words, in an information processing system where processing such as content viewing and services can be used in an interactive format, users can easily re-execute or reuse previously executed processes and services. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows an example of a screen output by an information processing device to a display unit. [Figure 2A] Figure 2A shows an example (1) of the screen area of ​​the dialogue history section formed on the display unit of an information processing device. [Figure 2B] Figure 2B is an example of a mini-player displayed in the screen area of ​​the dialogue history section. [Figure 3] Figure 3 shows an example (part 2) of the screen area of ​​the dialogue history section formed on the display unit of the information processing device. [Figure 4] Figure 4 shows an example (part 3) of the screen area of ​​the dialogue history section formed on the display unit of the information processing device. [Figure 5] Figure 5 illustrates the content playlist editing function. [Figure 6] Figure 6 is a diagram illustrating the hardware configuration of an information processing device. [Figure 7] Figure 7 is a flowchart illustrating AI dialogue processing by an agent. [Figure 8] Figure 8 is a flowchart illustrating the details of the processing required in response to a request. [Figure 9] Figure 9 is a flowchart showing the first example of the history rewriting process. [Figure 10]Figure 10 is a flowchart showing a second example of the history rewriting process. [Figure 11] Figure 11 is a flowchart illustrating the details of the playback and queue management processes. [Modes for carrying out the invention]

[0008] Hereinafter, an information processing device 1, an information processing method, and a program according to one embodiment will be described with reference to the drawings.

[0009] (Screen layout) Figure 1 shows an example screen output by the information processing device 1 to the display unit 14 (see Figure 6). The information processing device 1 is, for example, called an in-vehicle information device or an in-vehicle information processing device. The information processing device 1 may also be called an audiovisual navigation device. The information processing device 1 provides an interactive user interface to users in the vehicle, such as the driver, passengers, and occupants. That is, the information processing device 1 receives requests from the user in an interactive format, presents suggestions in response to the requests, and provides information processing and services according to the user's responses, selections, instructions, etc. In this way, the information processing device 1 receives input through an interactive user interface with the occupants of the vehicle and executes functions according to the received input.

[0010] Dialogue may be conducted through speech, i.e., voice, or through the input and output of text information. Dialogue may also be conducted through gestures. Furthermore, a dialogue-based user interface is not limited to the input and output of voice or text information. That is, a dialogue-based user interface may include input functions such as the operation of scroll bars on the screen, the operation of on-screen or physical push buttons, the operation of graphics objects exemplified by icons on the screen, and the operation of menus. Here, graphics objects exemplified by icons are also called graphical user interface (GUI) components. Additionally, a dialogue-based user interface may include output functions for images, videos, etc., to the screen, and output functions for music, sound effects, and other sounds from speakers.

[0011] The information processing or services provided by the information processing device 1 include, for example, the function of an audiovisual navigation device. More specifically, the information processing or services provide visual functions such as receiving television broadcasts, playing videos, and running games; audio functions such as receiving radio broadcasts and playing music; telephone functions; and navigation functions. Here, the navigation functions include providing map information and traffic congestion information. In addition, the information processing or services also include vehicle status management, such as monitoring tire pressure and engine operation. This may include assistance with maintenance such as managing the air conditioner, and operation of the air conditioner.

[0012] Furthermore, the information processing device 1 records a history that includes the content of the interaction with the user and information about the functions provided as a result of that interaction. The information processing device 1 then displays the history, which includes the string information representing the content of the interaction with the user and GUI components indicating the provided functions, on the screen. The information processing device 1 then accepts operations on the GUI components displayed on the screen and controls the execution of the functions indicated by those GUI components. The execution of the functions indicated by the GUI components can also be the re-provision of functions that have been provided once.

[0013] GUI components displayed along with the history may include images and other components related to the provided function. For example, if the provided function is content playback, the GUI components may include an image representing the content and a title bar representing the title. GUI components displayed along with the history may also accept operations for executing the function. For example, if the provided function is content playback, they may include a seek bar indicating the current playback position during the elapsed playback time of the content, a play start button, a pause button, a fast forward button, etc.

[0014] As another example, when the provided function is a telephone function, the GUI components may include the name / phone number / face photo of the call recipient, and may also include buttons for receiving operations such as making a call / registering in the phone book. Also, when the provided function is a navigation function, it may include the name and photo of the destination, and may also include buttons for receiving operations such as obtaining the weather of the destination / starting guidance to the destination.

[0015] The information processing apparatus 1 executes a computer program (hereinafter simply referred to as a program) to provide functions to the user in an interactive manner as described above. This program may be, for example, something called an Artificial Intelligence (AI) agent, or may be something called a chatbot. This program may provide the user with a function of information processing that matches the user's intention through interaction with the user and cooperation with an AI system including generative AI.

[0016] In the example of FIG. 1, the screen output by the information processing apparatus 1 to the display unit 14 by executing the program has a dialogue history unit 141 and other parts 142. The other parts 142 include, for example, display of maintenance information such as the current tire air pressure, an operation screen for the radio, display of a call history and an incoming call history of the phone, and the like.

[0017] On the other hand, the dialogue history unit 141 displays a history including string information indicating the content of the dialogue with the user and GUI components indicating the executed functions. The history includes a message MSA from an agent AG that is configured by the information processing apparatus 1 by executing the program and provides a conversation anthropomorphically, and a message MSU input by the user.

[0018] In the example in Figure 1, the dialogue history unit 141 displays the message MSA, "Shall we play your usual playlist? Lately, you've been listening to these at this time of day." Two playlists are then presented as options: 1. Chopin Masterpieces, 2. Piano Masterpieces. In the example in Figure 1, the dialogue history unit 141 displays the options for selecting the two playlists using the titles (1. Chopin Masterpieces, 2. Piano Masterpieces) and the corresponding GUI components G1 and G2. The user responds with the message MSU, "I don't want to listen today."

[0019] The dialogue that forms the history displayed by the dialogue history unit 141 in Figure 1 may be performed by voice utterance or by inputting and outputting string information. If this occurs, the information processing device 1 should convert the utterance into string information, store it, and output it to the dialogue history unit 141. If the dialogue is conducted through the input and output of string information, the input and output should be performed by the dialogue history unit 141. Furthermore, when the user inputs string information to the information processing device 1, the display unit 14 on which the dialogue history unit 141 is formed and the operation unit 15 on which input operations can be performed should be used (see Figure 6).

[0020] Furthermore, if the user instructs the system to play any playlist, the information processing device 1 plays the instructed playlist and embeds the GUI component G1, etc., corresponding to the played playlist into the history of the dialogue history unit 141. Through this process, the dialogue history unit 141 displays string information indicating the content of the dialogue between the information processing device 1 and the user, and the GUI component G1, etc., corresponding to the played playlist, which is a provided function, in chronological order on the history.

[0021] In the area of ​​the dialogue history unit 141, the downward direction toward the screen (the T-axis direction in Figure 1) is the direction in which time progresses on the history (the direction of newer history). On the other hand, in the area of ​​the dialogue history unit 141, the upward direction toward the screen (the opposite direction of the T-axis in Figure 1) is the direction in which time goes backward on the history (the direction of older history). In the example shown in the figure, a scroll bar BA and a slider SL that can slide on the scroll bar BA are provided near the right edge of the area of ​​the dialogue history unit 141. By sliding the slider SL on the scroll bar BA, the user can move along the time axis in the history and display the history corresponding to the time they moved to in the dialogue history unit 141. However, in this embodiment, there is no limitation on the direction in which time progresses on the screen; the opposite of the case in Figure 1, i.e., the upward direction toward the screen, may be the direction in which time progresses.

[0022] Furthermore, in this embodiment, the user can quote a selected message MSA, MSU, etc., by selecting the agent AG's message MSA or the user's message MSU in the area of ​​the dialogue history unit 141. The user can then resume the dialogue with the agent from the location of the quoted message. In addition, in this embodiment, the user can select or manipulate a GUI component G1, etc., corresponding to a previously played playlist in the area of ​​the dialogue history unit 141. Through such operations or selections, the user can play the playlist corresponding to the GUI component G1, etc. (for example, a collection of Chopin's masterpieces) again.

[0023] In this embodiment, the functions provided by the information processing device 1 in accordance with the interaction of an agent AG, etc., are not limited to the playback of playlists. The information processing device 1 may provide visual functions, audio functions, game functions, telephone functions, navigation functions, etc. The information processing device 1 may also perform maintenance support, air conditioner operation, etc. However, in the following embodiment, the processing of the information processing device 1 will be explained using the provision of music via a playlist as an example. Music can be considered an example of content. However, content also includes video content, text information content, game content, map information content, etc.

[0024] In this embodiment, there are no limitations on how message MSUs, GUI components G1, etc., are selected or operated. For example, selection or operation may be performed by pressing or tapping a finger on a touch panel for a predetermined time. The tap may be a single tap or a double tap. The tap may also be a long tap. Furthermore, selection or operation may be performed by selecting an option from an option menu.

[0025] (Screen example) The following describes the display of history in the screen area of ​​the dialogue history unit 141 and examples of history operation with reference to Figures 2A to 5. Figure 2A is a diagram showing an example (1) of the screen area of ​​the dialogue history unit 141 formed on the display unit 14 by the information processing device 1. In Figures 2A to 5, rectangular blocks denoted by the symbol Dk (where k is an integer of 1 or more) describe the processing of this embodiment. This is an explanatory text and not a component of the information output to the screen area of ​​the display unit 14.

[0026] In the example in Figure 2A, the user asks, "Do you have any song recommendations?" (MSU1), and Agent AG presents options (1) through (3) (MSA1). The user then responds, "Okay, I'll take (1)" (MSU2), and in response to the user's response, Agent AG plays the song corresponding to option (1) (for example, a collection of Chopin's masterpieces). In this way, the GUI component G1 corresponding to the content selected and played by the user is embedded in the history and displayed in the screen area of ​​the dialogue history unit 141 (D1). The GUI component G1, etc., is also called the playback screen. The playback screen is a graphics object that exemplifies a component called a mini-player, for example.

[0027] Figure 2B illustrates a mini-player displayed in the screen area of ​​the dialogue history unit 141. The mini-player includes an image M1 (such as an album cover) indicating the currently playing song, a title bar M2, a seek bar M3, and components for controlling the playback status, such as a play start button and a pause button M4. The mini-player may also include components to indicate whether the song is playing or stopped. As an example, the disclosed mini-player includes an object M5 that mimics a waveform. When the song is playing, the object M5, which mimics the song's waveform, animates in a wave-like manner to indicate that it is playing. When playback stops, the animation also stops to indicate that playback is stopped. Note that the animating object does not have to mimic a waveform; for example, it may mimic a graphic equalizer or a level meter. Alternatively, instead of indicating playback / stoppage through animation, the object may indicate playback / stoppage through changes in color or shape.

[0028] Furthermore, in the example history in Figure 2A, Agent AG asks, "What would you like to play next?" (MSA2), and the user responds, "Then play (2)" (MSU3). In response to the user's response, a GUI component G2 is embedded and displayed in the screen area of ​​the dialogue history unit 141, recording that the music (for example, a collection of piano masterpieces) corresponding to option (2) was played. In this embodiment, as shown in Figure 2A, a history of dialogue written in dialogue format, such as Agent AG's messages MSA1 to MSA2 and the user's messages MSU1 to MSU3, is called a chat log or chat-format log. A chat log can also be described as a history of dialogues and activities in tools for chat (i.e., dialogue) on a network or in a Social Networking Service (SNS).

[0029] In this embodiment, if a GUI component G2 corresponding to a later content is tapped while the previous content (for example, the content corresponding to GUI component G1) is playing on the history timeline, the playback of the later content is put on hold. In this embodiment, being put on hold means being registered in the playback queue.

[0030] The content that is on hold and awaiting playback may be displayed in the screen area of ​​the dialogue history unit 141 by the corresponding GUI component G2, for example, along with a pending indication. When playback of the previous content finishes on the history timeline, the content registered in the queue that is on hold is played in the order in which it was registered (D2). In this embodiment, tapping a GUI component G1 or the like on the screen is an example of selecting or operating on the GUI component G1 or the like.

[0031] Furthermore, in this embodiment, while previous content (for example, content corresponding to GUI component G1) is playing on the history timeline, the user can tap GUI component G2 corresponding to pending content to instruct the information processing device 1. The information processing device 1 then stops playing the previous content that was playing and skips it, and starts playing the tapped pending content, i.e., the content waiting to be executed.

[0032] Figure 3 shows an example (part 2) of the screen area of ​​the dialogue history unit 141 formed on the display unit 14 by the information processing device 1. In the example in Figure 3, the user's responses to the agent AG's suggestions (MSA1) and questions (MSA2, MSA3) and their selection of options (MSU2 to MUS4) are recorded. Furthermore, the history records that content corresponding to GUI components G1 to G3 has been played, and GUI components G1 to G3 are embedded and displayed in the screen area of ​​the dialogue history unit 141. In other words, graphical user interface (GUI) components for controlling the execution of provided functions are embedded in the screen of the information processing device 1.

[0033] In the screen area of ​​the dialogue history unit 141, the user can slide the slider SL on the scroll bar BA to go back through the history and, for example, tap a GUI component G1 that has been recorded as having been played. Then, the content corresponding to the tapped GUI component G1 starts playing. That is, when the information processing device 1 receives an operation on a first GUI component included in the log, it executes a first function corresponding to the first GUI component. Furthermore, when playback of content with an older playback date on the history timeline than the content that has been restarted is restarted, content with a newer playback date than the content that has been restarted is automatically put on hold. That is, GUI components G2 and G3 corresponding to content with a newer playback date on the history than the content that has been restarted (GUI component G1) are registered in the playback queue in historical order. Then, when the restarted playback of older content on the history is completed, the content corresponding to GUI components G2 and G3 is played in chronological order on the history timeline (D5).

[0034] Figure 4 shows an example (third example) of the screen area of ​​the dialogue history unit 141 formed on the display unit 14 by the information processing device 1. In the example of Figure 4, similar to Figure 2A, the history of the user's responses and selections to the suggestions (MSA1) and questions (MSA2, MSA3) from agent AG after the user's request (MSU1) is recorded and displayed (MSU2 to MSU3). The history also records that the content corresponding to GUI components G1 and G2 was played. However, in Figure 4, the dialogue is interrupted by the user's response "end" (MSU4). Then, the next dialogue starts with the user's request "play AAAA" (MSU4), and the history records that the content corresponding to GUI component G4 was played.

[0035] In the screen area of ​​the dialogue history unit 141, the history records that in the past, the user selected option (1) from among the agent's suggestions (1) to (3), and the content corresponding to GUI component G1 was played. In the example in Figure 4, the history shows that in this instance, the user performed an operation to select option (2) instead of (1) from among the suggestions (1) to (3) (see the "Change History" arrow and D6). As a result of this user operation, the content corresponding to GUI component G2 is played instead of the content corresponding to GUI component G1 recorded in the history. Then, in the screen area of ​​the dialogue history unit 141, the GUI component G2 that was played in this instance is embedded and displayed in place of GUI component G1 recorded in the history (D7).

[0036] Furthermore, in Figure 4, in the previously recorded history, the user's selection changes from (1) to (2), and as a result of the playback of content corresponding to GUI component G2, the content of the dialogue after the changed content no longer matches the user's history changes. Specifically, in the history, in response to options (1) through (3), the user responds "Okay, I'll go with (1)" (MSU2), and the content of GUI component G1 is played. Then, agent AG asks "What would you like to do next?" (MSA2), and the user responds "Okay, I'll play (2)" (MSU3), and the content of GUI component G2 is played.

[0037] However, because the user's history has been changed, the content of GUI component G2 has already been played, and the series of interactions following the playback of the content of GUI component G1 before the history change are inconsistent with the content of GUI component G2 after the history change. Therefore, in this embodiment, when the history is rewritten (change from option (1) to option (2)), the information processing device 1 grays out the series of interactions (MSU2 to MSU4, MSA2) related to the history before it was rewritten (D8). This is because the content of these interactions is inconsistent with the content played after the change or the changed option, and is therefore assumed to be meaningless.

[0038] The grayed-out sections of the history are excluded from user interaction. In other words, the information processing device 1 excludes from processing history entries that were related to the first function but not to the modified second function, as a result of executing the second function instead of the first function in the history.

[0039] Then, the information processing device 1 removes any content (content of GUI component G2) that overlaps with the content that was played as a result of the re-selection of the options from the queue, and registers the next content after the removed content in the queue (D9). If the next content after the removed content is already registered in the queue, the information processing device 1 simply keeps the next content in the queue. Then, the information processing device 1 removes any content (content of GUI component G2) that overlaps with the content that was played and sequentially plays the content recorded in the history. As a result of this process, the repeated automatic playback of duplicate content due to changes in the history is suppressed.

[0040] Figure 5 illustrates the content playlist editing function. In the screen area of ​​the dialogue history unit 141, when the user performs a list creation operation, GUI components G1 etc. corresponding to the series of content recorded in the history are displayed in window 145 in list format. The list creation operation may be a simple flick operation. Alternatively, the list creation operation may be a two-step operation consisting of a first operation to specify a time axis range in the history and a second operation to instruct the creation of the list.

[0041] The user can input commands to create, save, and play a new playlist using the list of GUI components G1 and other elements displayed in window 145. The list of GUI components G1 and other elements displayed in window 145 also includes a checkbox. The user can select the content to include in the new playlist by checking or leaving the checkbox blank.

[0042] (Hardware configuration) Figure 6 illustrates the hardware configuration of the information processing device 1. The hardware configuration of the information processing device 1 is the same as that of a normal computer. The information processing device 1 has a CPU 11, a main memory unit 12, and external devices connected via an interface (I / F), and performs information processing through a program.

[0043] The CPU 11 executes computer programs that are loaded into the main memory 12 in an executable format, and provides the functions of the information processing device 1. The main memory 12 is also simply called memory and stores computer programs executed by the CPU 11, data processed by the CPU 11, etc. The CPU 11 is also called a processor. However, the CPU 11 is not limited to a single processor and may be in a multi-processor configuration. Furthermore, the CPU 11 may be a single processor connected by a single socket and may be in a multi-core configuration.

[0044] Furthermore, the CPU 11 may include multiple different types of processors. For example, at least some of the processing of the information processing device 1 may be handled by a Digital Signal Processor (DSP), Graphics Processor, etc. Dedicated processors such as GPUs, numerical processors, vector processors, and image processing processors, as well as Application Specific Integrated Circuits (ASICs), etc. It may also be provided as such. Furthermore, at least a part of the information processing device 1 may be field-programmable. It may be a dedicated large-scale integration (LSI) such as a Gate Array (FPGA), or other digital circuitry. Furthermore, at least a portion of the information processing device 1 may include analog circuitry.

[0045] The main memory unit 12 includes Dynamic Random Access Memory (DRAM) and Static Random Access Memory. These include Memory (SRAM), Read Only Memory (ROM), etc. Note that CPU11 and The main memory unit 12 can be considered a control unit.

[0046] Examples of external devices include an external storage unit 13, a display unit 14, an operation unit 15, a communication unit 16, a microphone 17, a camera 18, and a speaker 19. The external devices do not have to be integrated with the control unit (CPU 11, main memory unit 12, etc.). Each of the external devices may be connected to the housing of the information processing device 1 via an interface.

[0047] The external storage unit 13 is used, for example, as a storage area that assists the main memory unit 12, and stores computer programs executed by the CPU 11, data processed by the CPU 11, etc. The external storage unit 13 can be a hard disk drive, a solid state drive (SSD), etc. Furthermore, the information processing device 1 may be provided with a drive device for a removable storage medium. Examples of removable storage media include Blu-ray discs, Digital Versatile Discs (DVDs), Compact Discs (CDs), and flash memory cards.

[0048] The display unit 14 is, for example, a liquid crystal display, an electroluminescent panel, an organic light-emitting diode (OLED), etc. The operation unit 15 is, for example, an array of push buttons, a keyboard, a pointing device, etc. In this embodiment, a touch panel is exemplified as a pointing device. The keyboard may be displayed on the display unit 14 and key input operations may be detected by the touch panel. The communication unit 16 exchanges data with other devices on the network. The communication unit 16 accesses wireless networks such as Long Term Evolution (LTE), 4th Generation Mobile Communication System (4G), 5th Generation Mobile Communication System (5G), and Wireless Local Area Network (LAN). A line communication device may also be used.

[0049] The microphone 17 detects the speaker's voice, converts it into an electrical signal, and transmits the resulting voice data to the CPU 11. The camera 18 captures an image of the speaker and transmits the image data to the CPU 11. The camera 18 has, for example, a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD). The speaker 19 plays back the speech of agent AG, sounds contained in the content, etc.

[0050] (Processing procedure) Referring to Figures 7 through 11, examples of processes executed by a program (Agent AG) in the information processing device 1 are shown. Figure 7 is a flowchart illustrating AI dialogue processing by Agent AG. The process in Figure 7 is started, for example, by powering on the information processing device 1 or starting up the information processing device 1. The process in Figure 7 can be terminated by cutting off the power to the information processing device 1 or shutting down the information processing device 1.

[0051] In AI dialogue processing, the information processing device 1 detects the occurrence of an event and executes processing according to the event. Events include user dialogue input (speech or input of text information), user operation on the operation unit 15, and completion of content playback. However, events are not limited to these; for example, timer interrupts, incoming phone calls, glob al positioning system (GPS), Global Navigation Satellite System (GNSS) This may include inputting positioning information from other sources.

[0052] In the process shown in Figure 7, the information processing device 1 typically waits for an event to occur (S1). When an event occurs (YES in S1), the information processing device 1 determines the type of event that occurred. If the event is a request from the user (hereinafter referred to as a user request) (YES in S5), the information processing device 1 analyzes the user request. User requests include calls, requests, questions, etc., from the user to the agent AG. User requests are exemplified by string information such as "Do you have any song recommendations?", "Then I'll play (1)," and "Then play (2)." as illustrated in Figure 2A.

[0053] User requests are input to the information processing device 1, for example, as prompts to the generating AI. The information processing device 1 obtains user requests from the prompts input by the user. Then, the information processing device 1 executes processing according to the request (S6). The processing according to the request may include output from the speaker 19 in voice. Then, the information processing device 1 returns the processing to S1. The processing according to the request will be explained separately with reference to Figure 8.

[0054] If the event is a history scroll operation (YES in S7), the information processing device 1 executes the history scroll operation (S8). Then, the information processing device 1 returns the process to S1.

[0055] If the event is a history change operation (YES in S9), the information processing device 1 executes a history rewrite operation (S10). A history change operation is, for example, an operation that changes the selection of an option recorded in the history, as illustrated in Figure 4. Then, the information processing device 1 returns to processing S1. The history rewrite operation will be explained separately with reference to Figures 9 and 10.

[0056] If the event is a history citation operation (YES in S11), the information processing device 1 performs processing according to the citation location (S12). For example, the agent AG of the information processing device 1 restarts the same dialogue from the cited part of the history. Then, the information processing device 1 returns to processing S1.

[0057] If the event is a playback instruction (YES in S13), the information processing device 1 performs playback and queue management (S14). Then, the information processing device 1 returns the process to S1. Playback and queue management will be explained separately with reference to Figure 11.

[0058] If the event is the end of content playback (YES in S15), the information processing device 1 performs queue management processing (S16). In the queue management processing, the information processing device 1 removes the content that has finished playing from the queue, identifies the next content waiting to be played in the queue, and starts playback of the next content. Then, the information processing device 1 returns to processing S1.

[0059] If the event is a flick operation in the screen area of ​​the dialogue history unit 141 (YES in S17), the information processing device 1 performs list processing (S18). In the list processing, the information processing device 1 extracts GUI components Gk (where k is an integer) corresponding to the played content within a specified range on the history time axis or a predetermined default range, and displays them in window 145 in a list format as illustrated in Figure 5.

[0060] The information processing device 1 then creates, saves, or plays a content playlist based on all of the GUI components Gk, which are the list items extracted by the user, or the items selected with a checkmark. The information processing device 1 then returns the process to S1. In Figure 7, list processing is performed by a flick operation, but list processing may be performed by operations other than flick operations. For example, options in the area of ​​the dialogue history unit 141 The process in S18 may be executed by selecting an item in the menu.

[0061] Figure 8 is a flowchart illustrating the details of the request-based processing (S6 in Figure 7). Request-based processing is the processing that responds to requests from the user during the interaction with the user. The execution of request-based processing records a new history in the form of a chat log. In this process, the information processing device 1 determines whether the user's request is for content selection or specification (S61). Content selection is, for example, the selection from recommended content (1) to (3) proposed by agent AG, as illustrated in Figure 2A. Content specification means that the user specifies a particular piece of content in the prompt and requests its playback.

[0062] If the user's request is to select or specify content, the information processing device 1 displays the GUI component corresponding to the selected or specified content in the corresponding area of ​​the interaction history unit 141 and records it in the history (S62). The history recording process in S62 is an example of displaying the log including the GUI component for operating the executed function. Then, the information processing device 1 plays the selected or specified content (S63).

[0063] If the user's request is not for selecting or specifying content, the information processing device 1 determines whether the user's request is a query to agent AG (S64). A query to agent AG is, for example, input of phrases such as "Do you have any song recommendations?" (MSU2) as exemplified in Figure 2A. If the user's request is a query to agent AG, the information processing device 1 outputs a response to the query (S65).

[0064] The response to the inquiry is, for example, the presentation of options (1) to (3) as illustrated in Figure 2A (MSA1). When the response to the inquiry is displayed in the screen area of ​​the dialogue history unit 141, it is recorded as history. The response to the inquiry may also be output as voice from the speaker 19. The user request determined in S5 in Figure 7, the response to the inquiry by Agent AG in the processing of S65 in Figure 8, etc., are also recorded as history in a chat-format log and displayed in the dialogue history unit 141. Therefore, the process of recording these chat-format logs and displaying them in the dialogue history unit 141 is an example of displaying the history of the dialogue as a chat-format log on the display unit.

[0065] If the user's request is not a query to agent AG, the information processing device 1 performs other processing (S66). Other processing is other processing in response to prompts entered by the user. Other processing includes, for example, the execution of telephone functions, navigation functions including the provision of map information and the use of traffic congestion information. After completing the execution of processing S63, S65, or S66, the information processing device 1 returns to the processing shown in Figure 7.

[0066] Figure 9 is a flowchart of the first example of the history rewriting process (S10 in Figure 7). The process in Figure 9 is referred to as history rewriting process A. In this process, the information processing device 1 determines whether the option selected in a past dialogue has been changed in the history displayed in the screen area of ​​the dialogue history unit 141. That is, the information processing device 1 determines whether the played content has been changed to content with another option (S101). The user can specify another option in place of the option recorded in the history by tapping the options displayed in the area of ​​the dialogue history unit 141 (for example, options (1) to (3) as exemplified in Figures 2A and 4). The information processing device 1 can determine whether the option has been changed from the user's tap operation. If the content selection has not been changed, the information processing device 1 terminates the process and returns to Figure 7.

[0067] On the other hand, if the content selection is changed, the information processing device 1 will check the GU recorded in the history. The component G1, etc., is changed to content corresponding to the changed selection, displayed in the area of ​​the dialogue history unit 141, and recorded in the history (S102). Then, the information processing device 1 starts playback of the content corresponding to the changed selection (S103).

[0068] In other words, the interactive user interface in the information processing device 1 includes presenting options and accepting input for selection. When the information processing device 1 receives an operation in the history, i.e., the chat log, to select the playback of a second selectable content instead of the first content that was played, it plays the second content again instead of the first content. Here, content playback can be considered an example of function execution. Therefore, when the information processing device 1 receives input for selection in the log, it can be said that it executes the second function corresponding to the selected option. Furthermore, it can be said that the information processing device 1 replaces the first GUI component that operates the first function corresponding to a previously selected option in the log with the second GUI component that operates the second function and displays it.

[0069] Furthermore, when the content played back in the history changes, the information processing device 1 grays out the history related to the content before the change and not related to the content after the change, and excludes it from processing (S104). In other words, the information processing device 1 deletes the history of conversations related to previously selected options from the log, or displays them in a different manner from other histories.

[0070] Then, the information processing device 1 performs queue cleanup (S105). Specifically, the information processing device 1 removes content that was grayed out and excluded from processing by the interaction in S104 from the queue's waiting list. Also, if the same content that was started to play in S103 is registered in the queue's waiting list, the information processing device 1 removes the same content from the queue. In other words, the information processing device 1 prevents the same content that was started to play in S103 from being played repeatedly. Then, the information processing device 1 finishes processing and returns to Figure 7.

[0071] As shown in Figure 9, when the second content is played instead of the first content due to a change in content selection (the process when YES is selected in S101 above), the information processing device 1 prevents duplicate playback at a later point in time. That is, by organizing the queue, the information processing device 1 suppresses or excludes the registration of content identical to the second content in the queue among the third content played at a later point in the history timeline than the playback time of the first content. When the playback of the content started in S103 is completed, the content waiting to be played is played sequentially according to S15 and S16 in Figure 7. Here, content playback can be considered an example of function execution. Therefore, it can be said that the information processing device 1 sequentially executes the second function that has been started, and functions corresponding to GUI components that are chronologically later than the second GUI component in the log, and ensures that each function is executed only once.

[0072] Figure 10 is a flowchart showing a second example of the history rewriting process (details of S10 in Figure 7). The process in Figure 10 is referred to as history rewriting process B. The information processing device 1 may execute history rewriting process A in Figure 9 or history rewriting process B in Figure 10, according to pre-set control parameters.

[0073] In this process, the determination in S101 is the same as in Figure 9. If the content selection is changed in the determination in S101, the information processing device 1 determines whether the same GUI component is recorded in the history after the playback of the changed content (S101A). That is, the information processing device 1 determines whether the same GUI component is recorded in the history after the playback of the changed content and The system determines whether playback of the same content has been recorded. If the same GUI component has not been recorded in the history since the playback of the modified content, the information processing device 1 executes the processes from S102 to S105. The processes from S102 to S105 are the same as in Figure 9.

[0074] On the other hand, if the determination in S101A indicates that a GUI component corresponding to the same content as the modified content is recorded, the information processing device 1 plays the same content at the most recent point in the history. That is, if the same content has been played multiple times, the information processing device 1 starts playing the modified content based on the GUI component that records that it was played at the most recent point in the history (S101C). Therefore, with respect to the content that has been started to be played, there is no history of the same content being played after that most recent point in time. For this reason, the information processing device 1 can prevent the same content from being played repeatedly by the processes in S15 and S16 of Figure 7 after the point in time when the content that has been started to be played is recorded.

[0075] Subsequently, the information processing device 1 proceeds to processing S105. The processing in S105 is the same as in Figure 9. However, if playback of content modified by a GUI component recorded at the latest point in the history of the same content is started in S101C, content whose playback was recorded before the latest point in the history when playback started will be excluded from the queue.

[0076] In other words, the information processing device 1 adjusts the playback of the second content to be played in place of the first content recorded in the history as having been played, as the selection of an option recorded in the history has been changed by user operation and the history has been rewritten. In other words, if the same content as the second content is in the third content which was played at a later time than the playback time of the first content, the information processing device 1 moves to the latest time. In other words, the information processing device 1 starts playing the content which has been changed by the GUI component recorded at the latest time in the history of the same content.

[0077] Here, content playback can be considered an example of function execution. Therefore, it can be said that the information processing device 1 adjusts the execution of the second function if the third function, which is provided at a later time than the time the first function was provided as recorded in the history, contains a function identical to the second function. In other words, it can be said that the information processing device 1 controls the execution of the second function according to the GUI component corresponding to the second function that is embedded at the most recent time in the history.

[0078] Figure 11 is a flowchart illustrating the playback and queue management process (details of S14 in Figure 7). This process occurs when an operation is made to request playback of a GUI component on the screen area of ​​the dialogue history unit 141. In this process, the information processing device 1 determines whether there is a GUI component currently being played in the history prior to the GUI component for which playback has been requested (S141). If the determination in S141 is NO, the information processing device 1 starts playback of the content of the GUI component for which playback has been requested (S142).

[0079] In other words, when the information processing device 1 receives an operation to replay a GUI component G1, etc., of content that has been recorded as having been played back in the chat log as history (S13 in Figure 7), it plays back the content corresponding to the GUI component G1, etc. Here, the playback of content can be said to be the execution of a function. Therefore, when the information processing device 1 receives an operation on the first GUI component included in the log, it can be said that it executes the first function corresponding to the first GUI component.

[0080] On the other hand, if the determination in S141 is YES, the information processing device 1 determines whether or not the content of the GUI component designated for playback is registered in the queue (S143). If the content of the specified GUI component is already queued, the information processing device 1 stops playing the content currently being played and starts playing the content corresponding to the specified GUI component (S144).

[0081] In other words, if the information processing device 1 finds that a second content corresponding to a GUI component designated for playback has already been queued and is waiting to be played, it terminates the playback of the first content that is currently being played. Then, the information processing device 1 starts playing the second content that is waiting to be played. Here, content playback is defined as the execution of a function. Therefore, if a GUI component for controlling the execution of a second function that is waiting to be executed is operated while a first function recorded earlier on the timeline in the history is being executed, the information processing device 1 terminates the execution of the first function that is currently being executed and starts the execution of the second function. After processing in S142 and S144, the information processing device proceeds to processing in S146.

[0082] On the other hand, if the S143 determination indicates that the content of the specified GUI component is not registered in the queue, the information processing device 1 registers the content to be played back in the queue (S145). After processing S145, the information processing device returns to the process shown in Figure 7.

[0083] In the S145 process, while the first content that was played earlier in the history is being played, the information processing device 1 receives a playback instruction operation for the GUI component G1, etc., which controls the playback of the second content that was played after the first content. The information processing device 1 then puts the second content into a waiting state for playback. As a result, in the determination in S15 in Figure 7, when the playback of the first content is completed, the playback of the second content starts in the S16 process.

[0084] In other words, the information processing device 1 registers the second content in a queue for playback and plays the second content after the first content has finished playing. Here, content playback is defined as function execution. Therefore, if the information processing device 1 receives an operation on a second GUI component that is chronologically later than the first GUI component in the log while the first function is being executed, it can be said that it puts the second function corresponding to the second GUI component into a state of waiting to be executed. And it can be said that the information processing device 1 starts executing the second function after it has finished executing the first function.

[0085] The information processing device 1 then determines whether the playback of other GUI components is recorded in the history after the GUI component for which playback is specified (S146). If the determination in S146 is YES, the information processing device 1 registers the content of other GUI components after the GUI component for which playback is specified in the queue and puts them in a playback waiting state (S147). Therefore, it can be said that when the first function that was executed earlier in the history is in the process of being executed, the information processing device 1 puts the second function that was executed after the first function in the history into a waiting state. After that, the information processing device 1 returns to the process in Figure 7. Then, the information processing device 1 sequentially executes the functions corresponding to GUI components that are located chronologically later than the first GUI component through the processes in S15 and S16 in Figure 7. Therefore, together with the processes in S13 and S14 in Figure 7, it can be said that when the information processing device 1 receives an operation on the first GUI component, it sequentially executes the first function corresponding to the first GUI component, and the functions corresponding to GUI components that are located chronologically later than the first GUI component in the log.

[0086] (Effects of the embodiment) As described above, the information processing device 1 receives input from the vehicle occupant through a conversational user interface and executes functions corresponding to the received input. The information processing device 1 then displays the conversation history as a chat-style log on the conversation history unit 141, which serves as a display unit. The information processing device 1 also displays GUI components for operating the executed functions in the log. Display it including this.

[0087] The information processing device 1 then accepts operations on the displayed history and executes processes to manipulate the history. Therefore, the information processing device 1 provides functions such as content viewing in an interactive format, and furthermore, allows users to easily reuse functions previously provided. Consequently, users can easily repeat processes or services that have been previously executed or used.

[0088] In this embodiment, one example of a function provided in an interactive format is content playback. Therefore, when the information processing device 1 provides content to the user, the user can easily reuse content that they have used in the past.

[0089] When the information processing device 1 is playing the first content that was played earlier in the history, if it receives a playback instruction operation for the GUI component G1, etc., which controls the playback of the second content that was played after the first content, it puts the second content into a playback waiting state. In other words, the information processing device 1 registers the playback waiting state of the second content in a queue and plays the second content after the playback of the first content is finished. That is, when the information processing device 1 receives an operation for the second GUI component that is located chronologically later than the first GUI component in the log while the first function is being executed, it puts the second function corresponding to the second GUI component into a state of execution waiting, and executes the second function after the execution of the first function is finished. As a result, the user can easily execute functions that were previously provided sequentially in the order they were provided on the history timeline.

[0090] However, as shown in S143 and S144 of Figure 11, the process is different if the second content to which the playback instruction operation has been made is already registered in the queue and waiting to be played. In this case, the information processing device 1 terminates the playback of the first content that is currently being played and starts the playback of the second content that is waiting to be played. That is, if a GUI component for controlling the execution of the second function that is waiting to be executed is operated while the first function that is recorded earlier on the timeline in the history is being executed, the information processing device 1 terminates the execution of the first function that is currently being executed and starts the execution of the second function. Therefore, the information processing device 1 can provide the user with a process that cancels and skips the execution of a function that was provided earlier on the timeline in the history and forcibly moves the execution to a later function on the timeline in the history.

[0091] Furthermore, when the information processing device 1 receives an operation to instruct the GUI component G1, etc., of content that has been recorded as having been played in the history to play again (S13 in Figure 7), it plays the content corresponding to the GUI component G1, etc., again. In other words, when the information processing device 1 receives an operation on the first GUI component included in the log, it executes the first function corresponding to the first GUI component. Therefore, the information processing device 1 can easily allow the user to execute again the functions that have been recorded as having been provided in the history.

[0092] Furthermore, as described above, when the information processing device 1 receives an operation on the first GUI component, in conjunction with the processes S15 and S16 in Figure 7, and S13 and S14 in Figure 7, it sequentially executes the first function corresponding to the first GUI component, and the functions corresponding to GUI components that are chronologically later than the first GUI component in the log. Therefore, in a system where functions are executed interactively, the user can easily re-execute the function based on the history recorded interactively.

[0093] When the information processing device 1 receives a selection input in the log, it executes the second function corresponding to the selected option and replaces the first GUI component that operates the first function corresponding to a previously selected option in the log with the second GUI component that operates the second function. Therefore, the user can re-select a function and see a different option. The function can be easily executed.

[0094] Furthermore, when the second content is played instead of the first content due to a change in content selection (S9 history change operation in Figure 7), the information processing device 1 prevents duplicate playback at a later point in time. That is, by organizing the queue (S105 in Figures 9 and 10), the information processing device 1 suppresses or excludes the registration of content identical to the second content in the queue among the third content that has been recorded as having been played at a later point in the history timeline than the playback time of the first content. Here, content playback can be considered as the execution of a function. Therefore, the information processing device 1 prevents the second function from being executed in place of the first function. Furthermore, the system sequentially executes functions corresponding to GUI components that are chronologically later than the second GUI component in the log, and ensures that each function is executed only once. Therefore, when the information processing device 1 executes a function that has been recorded as having been provided in the history, it can suppress the repeated execution of the same function due to changes in the user's selection.

[0095] Furthermore, as shown in the processing of S104 in Figures 9 and 10, when the played content in the history is changed, the information processing device 1 grays out the history that is related to the content before the change and not related to the content after the change, and excludes it from processing. In other words, when the information processing device 1 executes the second function instead of the first function in the history, it excludes the history that is related to the first function and not related to the second function from processing. Therefore, the information processing device 1 deletes the history of conversations related to previously selected options from the log, or displays them in a different manner from other histories. As a result, the information processing device 1 can simplify the process of executing functions based on the history.

[0096] Furthermore, as shown in S101C of Figure 10 above, if the selection of an option recorded in the history is changed by user operation and the history is rewritten, the information processing device 1 adjusts the effect of the history being rewritten. That is, the information processing device 1 adjusts the playback of the second content to be played in place of the first content which is recorded as having been played in the history. For example, if the same content as the second content is in the third content which was played at a later time than the playback time of the first content, it moves to the latest playback time in the history. That is, the information processing device 1 starts playing the content which has been changed by the GUI component recorded at the latest time in the history of the same content.

[0097] Content playback can be considered an example of function execution. Therefore, it can be said that the information processing device 1 adjusts the execution of the second function when the third function, provided at a later time than the time the first function was provided as recorded in the history, includes a function identical to the second function. In other words, the information processing device 1 controls the execution of the second function according to the GUI component corresponding to the second function that is embedded at the most recent point in the history. In this way, when the information processing device 1 executes a previously provided function using the history, it limits the execution of the function to once and simplifies the process. That is, even if the history is rewritten due to a change in the selection of an option recorded in the history and the function to be executed is changed, the information processing device 1 can prevent the changed function from being executed repeatedly.

[0098] In this embodiment, the GUI components displayed in the dialogue history unit 141 along with the history include control components for operating content playback. These control components include, for example, a seek bar, a play start button, a pause button, and so on. Therefore, when a user plays content by operating the history displayed in the dialogue history unit 141, they can easily change the playback state of the content. Changing the playback state here means, for example, changing the playback position during the elapsed playback time, pausing playback, fast-forwarding, etc.

[0099] (A computer-readable recording medium) A program that enables a computer or other machine or device (hereinafter referred to as "computer, etc.") to perform any of the above functions can be recorded on a recording medium that the computer, etc. can read. By having the computer, etc. read and execute the program on this recording medium, it can be made to provide that function.

[0100] Here, a recording medium that can be read by a computer refers to a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical means and can be read by a computer. Examples of such recording media that are removable from a computer include flexible disks, magneto-optical disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray discs, and memory cards such as flash memory. Recording media fixed to a computer include hard disks and ROMs (Read Only Memory). Furthermore, there are SSDs (Solid State Drives). This can be used as both a removable recording medium from a computer or other device, and a recording medium fixed to a computer or other device. [Explanation of Symbols]

[0101] 1. Information Processing Device 11 CPU 12 Main memory 13 External storage unit 14 Display section 15 Control section 16 Communications Department 17 Microphone 18 Cameras 19 speakers 141 Dialogue History Section

Claims

1. It includes a control unit that accepts input through an interactive user interface with the vehicle's occupants and executes functions corresponding to the received input. The control unit is characterized by displaying the history of the dialogue as a chat-format log on the display unit, and by displaying graphical user interface (GUI) components for operating the executed functions in the log. In-vehicle information processing system.

2. The in-vehicle information processing device according to claim 1, which includes the playback of content.

3. The GUI component includes an operating component for controlling the playback of the content, as described in claim 2. In-vehicle information processing system.

4. The control unit, upon receiving an operation on the first GUI component included in the log, executes the first function corresponding to the first GUI component, as described in claim 1. In-vehicle information processing system.

5. The control unit, upon receiving an operation on the first GUI component, sequentially executes a first function corresponding to the first GUI component and a function corresponding to a GUI component located later in the log chronologically than the first GUI component, as described in claim 4. In-vehicle information processing system.

6. The control unit, when it receives an operation on a second GUI component located later in the log than the first GUI component during the execution of the first function, puts the second function corresponding to the second GUI component into a waiting state, and executes the second function after the execution of the first function is completed, as described in claim 4. In-vehicle information processing system.

7. The aforementioned interactive user interface includes presenting options and accepting input for selection, The control unit, upon receiving the input of the selection in the log, executes a second function corresponding to the selected option, and replaces a first GUI component that operates a first function corresponding to a previously selected option, which is included in the log, with a second GUI component that operates the second function, and displays it, according to claim 1. In-vehicle information processing system.

8. The control unit sequentially executes the second function and functions corresponding to GUI components located later in the log than the second GUI component, and ensures that each function is executed only once, according to claim 7. In-vehicle information processing system.

9. The control unit deletes the history of the dialogue related to the previously selected option from the log, or displays it in a manner different from other history, as described in claim 7. In-vehicle information processing system.

10. Computers It accepts input from the vehicle's occupants through an interactive user interface and executes functions corresponding to the received input. The history of the aforementioned dialogue is displayed on the display unit as a chat-format log, and the log is also displayed including graphical user interface (GUI) components for operating the executed functions. Information processing methods.

11. On the computer, It accepts input from the vehicle's occupants through an interactive user interface and executes functions corresponding to the received input. The history of the aforementioned dialogue is displayed on the display unit as a chat-format log, and the log is also displayed including graphical user interface (GUI) components for operating the executed functions. program.

Images