Information Processing Apparatus and Program

Abstract

To enable visual information provision by an AI assistant to be received while inhibiting a user-friendly application obtained by a display device of a large area of an information processing device having an openable and closable display mechanism from being damaged.SOLUTION: When a user performs a text input in a UI image 15a with the UI image 15a of an application displayed on a main display 14 by making a display mechanism a closed state, an AI assistant retrieves information related to the input, and generates provision information to the user. At that time, an information processing device 10 displays a provision information image 17a showing the provision information on a sub-display 16. When the user closes the display mechanism or reverses the information processing device 10, the user can look at the provision information image 17a displayed on the sub-display 16.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] Flexible displays using organic EL (electroluminescence), electronic paper, etc. have been developed. In recent years, the structures of flexible displays mounted on mobile terminals have been proposed (Patent Documents 1-3), and mobile terminals having such structures have been sold.

[0003] A smartphone equipped with a flexible display and foldable in two folds can achieve a width that can be held in one hand, for example, in a closed state, and can display a large screen comparable to that of a tablet terminal in an open state. Among this type of smartphone, there are those that can display a smaller screen area even in a closed state than in an open state. For example, a certain model of smartphone that folds with the large screen of the main display facing inward has a smaller sub-display on the surface located outside in the closed state. Also, a certain model of smartphone that folds with the large screen of the main display facing outward displays on half of the two-folded main display facing the user in the closed state. Also, these models of smartphones link the screen display between the closed state and the open state. For example, when they open the large screen, they display the same app screen on the large screen as the app screen that was previously displayed on the small screen in the closed state. Also, among this type of smartphone, there are those with a multi-window function such as dividing the open main display into a plurality of regions and displaying user interface images of different apps in each region.

[0004] In addition, the device described in Patent Document 4 displays a user interface (hereinafter referred to as UI) image on the entire touch panel when the foldable touch panel is opened flat. Then, in a state where the touch panel is bent at a certain angle, a soft keyboard is displayed on one half of the touch panel, and a UI image is displayed on the remaining half, providing a usability similar to that of a notebook PC.

[0005] In recent years, software agents called AI (Artificial Intelligence) assistants have become popular. AI assistants, also called virtual assistants, understand user instructions, questions, and other inputs and provide users with services corresponding to those inputs.

[0006] Currently, the mainstream method for AI assistants on mobile terminals such as smartphones is to display the UI image in the display area for application programs (hereinafter abbreviated as "apps") (hereinafter referred to as the "app display area") excluding areas indicating time, communication status, etc., or to display it in full-screen mode across the entire display area of the display.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Means for Solving the Problems

[0011] The invention according to claim 1 is Arranged on the side that the user visually recognizes in the unfolded state a first display device and the first display device Difficult to visually recognize simultaneouslyA second display device provided on a surface, and a processor, wherein the processor causes the first display device to display information input by a user, and in a state where the input information is being displayed on the first display device, causes the second display device to display information generated by an AI assistant based on the input information When there is information generated by the AI assistant based on the information input by the user and displayed on the first display device, cause the first display device to display that there is such generated information An information processing apparatus characterized by performing control.

[0012] The invention according to claim 2 is The information input by the user is information input by voice ..., which is the information processing apparatus according to claim 1, characterized in that

[0013] The invention according to claim 3 is The information input by the user is information directly input to the AI assistant ..., which is the information processing apparatus according to claim 1 described above.

[0014] The invention according to claim 4 is The AI assistant is activated by the input of the voice of a predetermined calling word by the user ..., which is the information processing apparatus according to claim 1 described above.

[0016] Claim 5 The invention according to... is Arranged on the side that the user visually recognizes in the unfolded state A computer including a first display device and a second display device provided on a surface of the first display device, which causes the first display device to display information input by a user, and in a state where the input information is being displayed on the first display device, causes the second display device to display information generated by an AI assistant based on the input information Difficult to visually recognize simultaneously to execute control. When there is information generated by the AI assistant based on the information input by the user and displayed on the first display device, cause the first display device to display that there is such generated information It is a program for causing

Effect of the Invention

[0019] According to the present invention, in a state where the information input by the user is displayed on the first display device, the information generated by the AI assistant based on the input information can be displayed on the second display device provided on the surface where the first display device is not provided

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0027] With reference to FIG. 1, an example of the display control of a two-fold foldable information processing apparatus 10 including a large-screen flexible display will be described. The information processing apparatus 10 is, for example, a portable information terminal (such as a smartphone or a tablet terminal), but is not limited thereto.

[0028] The information processing apparatus 10 illustrated in FIG. 1 includes a first body 12A and a second body 12B that are interconnected so as to be openable and closable. On one surface of the first body 12A and the second body 12B, a flexible main display 14 is provided across both of them. The main display 14 has a display area with an area close to the combined area of the surfaces of both the first body 12A and the second body 12B. Also, in this example, a sub-display 16 is provided on the surface of the first body 12A where the main display 14 is not provided. The sub-display 16 has an area that fits within the surface of the first body 12A and has a display area smaller than that of the main display 14. Also, the sub-display 16 does not have to be flexible. The main display 14 and the sub-display 16 are configured as, for example, touch panel displays. In this example, the main display 14 corresponds to the first display device and the sub-display 16 corresponds to the second display device. Also, the structure including the main display 14 and the sub-display 16 and the first body 12A and the second body 12B that support them corresponds to an openable and closable display mechanism.

[0029] On the left side of FIG. 1, the information processing apparatus 10 in the (a) closed state is shown, and on the right side, the information processing apparatus 10 in the (b) open state is shown.

[0030] The closed state means a state in which the display mechanism is closed to such an extent that the display area of the main display 14 cannot be sufficiently seen from the outside. In the closed state, from the outside of the information processing apparatus 10, instead of the display area of the main display 14 being sufficiently visible, the display area of the sub-display 16 is clearly visible. In the (a) closed state illustrated in the figure, the surfaces of the first body 12A and the second body 12B on the main display 14 side are in contact with each other or are close enough to be in contact. In this state, the angle formed by the surfaces of the main display 14 on the first body 12A side and the second body 12B side (i.e., the opening angle of the display mechanism) is approximately 0 degrees. Note that defining the case where the opening angle is approximately 0 degrees as the closed state is merely an example. Instead, a threshold value larger than 0 degrees may be set, and the case where the opening angle is within the range from 0 degrees to that threshold value may be defined as the closed state.

[0031] The open state means that the display mechanism is open to such an extent that the display area of the main display 14 can be clearly seen. In the open state, from the outside of the information processing apparatus 10, instead of the display area of the sub-display 16 being clearly visible, the display area of the main display 14 is clearly visible. In the figure, as an example, the state where the display mechanism is opened by 180 degrees is shown as the (b) open state, but this is just an example. For example, a second threshold less than 180 degrees may be defined, and the case where the opening angle is in the range from the second threshold to 180 degrees may be defined as the open state.

[0032] When the display mechanism of the information processing apparatus 10 is in the closed state, the information processing apparatus 10 performs screen display in the display area of the sub-display 16. In the closed state, the screen display of the main display 14 may be stopped (for example, the main display 14 may be turned off). Conversely, when the display mechanism is in the open state, the information processing apparatus 10 performs screen display in the display area of the main display 14. In the open state, the screen display in the display area of the sub-display 16 may be stopped.

[0033] In the figure example, on the closed sub-display 16, a user interface image (hereinafter referred to as a UI image) 17 showing information of a certain contact generated by the address book application is displayed in the display area assigned to the address book application (the area surrounded by the dashed line of 17). In this UI image 17, information such as the name, address, and phone number of the contact, an icon 20 for calling the phone application, an icon 22 for calling the email application, etc. are displayed. When the user opens the display mechanism from this state to the open state, the information processing device 10 starts to display on the main display 14. At this time, on the main display 14, in the display area assigned to the address book application (the area surrounded by the dashed line of 15), a UI image 15 showing information about the same contact of the same address book application as the UI image 17 displayed in the closed state is displayed. Here, since the display area of the main display 14 is larger than that of the sub-display 16, the application program is configured to display more display contents in the UI image 15 than in the UI image 17. For example, in the UI image 15, in addition to the display contents displayed in the UI image 17, an image of the face photo of the contact user not displayed in the UI image 17, an icon 24 for calling an SNS (Social Media Network) application, etc. are displayed. The UI image 15 of the application displayed on the main display 14 is an example of the first UI image, and the display contents (for example, consisting of character strings, face photo images, and icon groups) in the UI image 15 are examples of the display contents of the first UI image. Note that in the illustrated example, the UI image 15 of the address book application extends over the entire display area of the main display 14, but this is just an example. Depending on the settings of the information processing apparatus 10 and the OS (operating system) that controls it, there may be an area reserved for displaying the time, communication status, notifications, or an icon group of applications, etc., at a predetermined location within the display area of the main display 14, such as the upper or lower end portions thereof. In such a case, the UI image of the application opened by the user is displayed in the remaining area of the display area of the main display 14, excluding the area reserved in advance in this way. The area where the UI image of the application is displayed in this manner corresponds to the "display area assigned to the first application program". Note that the fact that the "display area" is "assigned to the first application program" is not limited to the case where a display area is assigned to each of a plurality of applications when there are a plurality of applications. It also includes the case where the same display area is commonly assigned to those plurality of applications. In this case, when any one of the applications is launched, the UI image of that application is displayed in the remaining common display area of the display area of the main display 14, excluding the area reserved in advance as described above.

[0034] Note that in this specification, an application refers to an application program executed by the information processing apparatus 10.

[0035] FIG. 2 shows an example of the main part of the hardware configuration of the information processing apparatus 10. This hardware configuration includes elements that a normal computer has, such as a processor 102, a memory 104 such as a random access memory, an auxiliary storage device 106 such as a flash memory, a display control circuit 108, an input control circuit 110, etc. Further, a communication circuit 112, a sensor 114 for detecting the open / closed state of the display mechanism, etc. are also included. In the illustrated example, a sensor control circuit 115 that controls the processor 102, the memory 104, the auxiliary storage device 106, the display control circuit 108, the input control circuit 110, the communication circuit 112, and the sensor 114 is connected to a bus 120. The processor 102 exchanges data with the memory 104, the auxiliary storage device 106, the display control circuit 108, the input control circuit 110, the communication circuit 112, and the sensor control circuit 115 via the bus 120. Note that in the example of FIG. 2, all elements from the processor 102 to the sensor control circuit 115 are flatly connected to the same bus 120, but this is merely an example. Instead, a hierarchical structure may be adopted in which some of these elements (for example, an element group including the processor 102) are integrated into one chip such as a SoC (System-on-a-Chip), and the remaining element group is connected to an external bus to which that chip is connected.

[0036] Here, the processor 102 refers to a processor in a broad sense and includes a general-purpose processor (for example, a CPU: Central Processing Unit, etc.) and a dedicated processor (for example, a GPU: Graphics Processing Unit, an ASIC: Application Specific Integrated Circuit, an FPGA: Field Programmable Gate Array, a programmable logic device, etc.).

[0037] Moreover, the operation of the processor 102 may be performed not only by one processor but also by a plurality of physically separated processors working together. Also, each operation of the processor 102 is not limited to the order described in the following embodiments and may be changed as appropriate.

[0038] The auxiliary storage device 106 has installed therein an OS of the information processing device 10, various applications, a program for interfacing with an AI assistant described later or an AI assistant on an external server, and a program for controlling the display of the AI assistant's provided information image in the present embodiment described later.

[0039] The display control circuit 108 controls the screen displays of the main display 14 and the sub-display 16. The input control circuit 110 detects and processes touch inputs to the touch panels provided in the main display 14 and the sub-display 16, and transmits the processing results to the processor. The communication circuit 112 executes signal processing for wireless communication of the information processing device 10 with an external network device.

[0040] The sensor 114 for detecting the open / closed state of the display mechanism may be, for example, a Hall element. In this case, a magnet is provided on one of the first body 12A or the second body 12B, and a Hall element is provided on the other. When the magnet and the Hall element approach each other as the display mechanism is closed, the Hall element detects magnetism and detects that the display mechanism is in the closed state. The sensor for detecting the open / closed state of the display mechanism is not limited to the Hall element. As this sensor 114, various sensors conventionally used for detecting the opening and closing of an openable device, such as an infrared sensor, an illuminance sensor, an acceleration sensor, a gyro sensor, a proximity sensor, and the like, may be used.

[0041] Moreover, the information processing device 10 may incorporate output devices such as a speaker and a vibration mechanism.

[0042] The information processing apparatus 10 is installed with a program for displaying the provided information provided by the AI assistant.

[0043] The AI assistant is a software agent that understands the user's input and provides the user with services corresponding to the input. In this embodiment, among the services provided by the AI assistant, focus is placed on the service of visually presenting the information (hereinafter referred to as "provided information") generated and provided by the AI assistant in response to the user's input.

[0044] The AI assistant is activated according to an explicit activation instruction from the user and receives input from the user after activation. Also, when the UI image of the app for which using the AI assistant is set is displayed on the main display 14 or the sub-display 16, the AI assistant receives the information input by the user for the app from the app.

[0045] The methods of user input to the app or the AI assistant can be various cases such as input from a keyboard, touch input to the screen, gesture input by the user's body such as a hand or finger, voice input, image input from a camera, etc. Also, the AI assistant may use the location of the information processing apparatus 10 carried by the user obtained using a GPS or an indoor positioning system, etc. as the input from the user.

[0046] The AI assistant searches for information related to the content of the user's input within the information processing apparatus 10 or from a server on the network (for example, the Internet) to which the AI assistant is connected, etc., and generates the provided information to be provided to the user based on the searched information. Among the provided information generated by the AI assistant, the visual provided information that this embodiment focuses on includes text, images, videos, videos with voice, etc.

[0047] The AI assistant may be installed in the information processing device 10 itself, or may be installed in a server on a network accessible from the information processing device 10. In the latter case, the information processing device 10 is installed with a program that performs a process of transmitting user input to the AI assistant on the server and a process of displaying provided information responded by the AI assistant in response to the transmission.

[0048] Consider a case where visual provided information generated by the AI assistant is provided to the user while the UI image of the app being executed by the information processing device 10 (hereinafter referred to as the "running app") is displayed on the main display 14. For example, when a setting to use the AI assistant is made for the running app, an example is a case where the AI assistant generates provided information related to the information (e.g., text information) input by the user to the UI image. Assume that the user is inputting to the UI image, for example, to create a document (i.e., a document) or the like.

[0049] In such a case, if an image for displaying the provided information from the AI assistant (hereinafter referred to as the "provided information image") is displayed on the main display 14 so as to completely or partially cover the UI image of the running app, part or all of the UI image will become invisible, which will interfere with the user's input work. Also, even if a button for deleting the image is provided on the provided information image and the UI image of the running app is displayed when the user presses the button, it takes time for the user to press the button.

[0050] Therefore, in such a case, the information processing device 10 according to the present embodiment displays the provided information image of the AI assistant on the secondary display 16 while the UI image of the running app is still displayed on the main display 14. The UI image of the running app displayed on the main display 14 corresponds to the first user interface image displayed on the first display device, and the information providing image displayed on the secondary display 16 corresponds to the second user interface image displayed on the second display device, respectively.

[0051] The display control of such provided information images will be further described with reference to the specific example shown in FIG. 3.

[0052] In the example of FIG. 3, a UI image 15a of a document creation application is displayed on the main display 14 of the display mechanism in the open state. This document creation application is the currently running application. The UI image 15a is a UI image for document creation and displays text, images, etc. input by the user. It is assumed that the document creation application has been previously set to use an AI assistant.

[0053] The AI assistant monitors the data (text in the example shown in the figure) input by the user within the UI image 15a displayed on the main display 14, searches for information related to the data from information sources such as the Internet, and generates provided information based on the information obtained as a result of the search. In the example shown in the figure, the user is creating a document with the title "Report: On the Growth of Each Country" and has written up to two and a half lines of the main text. The AI assistant monitors the input content (text data in this example) of the user with respect to the UI image 15a, searches for information related to the input content, and edits the searched information to generate the provided information to be provided to the user. For example, the AI assistant has learned to search for and generate information evaluated as useful for the input content, and uses the learning result to generate the provided information. In the example shown in the figure, the AI assistant uses a known keyword extraction technique to extract keywords "United States", "2019", and "nominal GDP" from the text in the UI image 15a, and searches for information strongly related to this set of keywords. Through this search, information on the nominal GDP of the United States in 2019 and a graph of the annual changes in the GDP of the United States strongly related to this are found. The AI assistant edits the information obtained by the search to generate the provided information or the information of the provided information image 17a for displaying the provided information.

[0054] When the AI assistant generates the provided information, the information processing apparatus 10 performs display control for displaying a provided information image 17a indicating the provided information within the sub-display 16. The function of this display control may be incorporated, for example, in the OS of the information processing apparatus 10, or in an AI assistant mounted on the information processing apparatus 10, or in a program or the like mounted on the information processing apparatus for the interface with an external AI assistant. Also, an independent program responsible for the display control function may be mounted on the information processing apparatus 10.

[0055] Here, if the provided information image 17a is immediately displayed on the secondary display 16 when the provided information is obtained from the AI assistant, the display will be made on the secondary display 16 that the user is not looking at, and the risk of information leakage will be unnecessarily high. Therefore, the information processing apparatus 10 displays the provided information image 17a on the secondary display 16 only when it is determined that the probability that the user is looking at the secondary display 16 is high. That is, the information processing apparatus 10 displays the provided information image 17a on the secondary display 16 when it is determined that the secondary display 16 is facing the user side, that is, roughly in the direction facing the user's face. For example, when the user folds the display mechanism of the information processing apparatus 10, or when the user turns over the opened information processing apparatus 10 so that the secondary display 16 faces the user's side, etc. are examples of states in which it is determined that the probability that the user is looking at the secondary display 16 is high. When such a state is reached, the display control function lights up the secondary display 16 so that the provided information image 17a on the secondary display 16 can be visually recognized. Whether the user has folded the display mechanism is determined from the signal output by the above-described sensor 114. Also, whether the user has turned over the information processing apparatus 10 may be determined based on one or more of the signals from the acceleration sensor, camera, or face authentication sensor built in the information processing apparatus 10. For example, when it is found from the direction of the gravitational acceleration detected by the acceleration sensor that the secondary display 16 is facing upward, it is determined that the user has turned over the information processing apparatus 10 and is in a state of looking at the secondary display 16. Also, when the user's face is detected from the image captured by the camera provided on the same side as the secondary display 16, or when the face authentication sensor provided on that surface recognizes the user's face, it is also determined that the user is in a state of looking at the secondary display 16. The information of the user's face to be compared with the image of the camera or the signal of the face authentication sensor may be registered in the information processing apparatus 10 in advance.

[0056] Also, just because the display mechanism is folded, the sub-display 16 does not necessarily face the user side. Therefore, in one example, when it is detected that it has been folded, it is further determined whether the sub-display 16 faces the user side based on signals from an acceleration sensor or the like. When it is determined that it is facing, the provided information may be displayed on the sub-display 16.

[0057] Here, the display control function of the information processing apparatus 10 may make the UI image 15a of the document creation application on the main display 14 invisible when it is detected that it has been turned over and the provided information is displayed on the sub-display 16. For this, for example, the main display 14 may be turned off. Thereby, the risk that the display information of the main display 14 that the user is not looking at is peeked by others is reduced.

[0058] Also, the information processing apparatus 10 (particularly the function of controlling the display of the provided information) may notify the user that there is provided information generated by the AI assistant. This notification may be made by vibrating the information processing apparatus 10 in a predetermined pattern using the vibration function built in the information processing apparatus 10. Also, the notification may be made by outputting a predetermined sound from a speaker of the information processing apparatus 10 or a voice output device such as earphones wired or wirelessly connected to the information processing apparatus 10. Further, the notification may be made by the emission or blinking of a lamp such as an LED (light emitting diode) provided in the information processing apparatus 10. As another example of the form of the notification, as illustrated in FIG. 5, a predetermined-shaped icon 20 indicating the presence of provided information from the AI assistant may be displayed in the UI image 15a on the main display 14. This icon 20 shall have a small area so as not to hide the display content of the UI image 15a as much as possible. If the icon 20 has a smaller area than the provided information image 17a, the area of the hidden portion of the UI image 15a is smaller than when the provided information image 17a is displayed on the main display 14. Also, the notification may be made by a combination of two or more of vibration, voice, emission of a lamp, display of the icon 20 on the UI image 15a, etc.

[0059] Upon receiving the notification, the user becomes aware of the existence of the provided information generated by the AI assistant, and closes, i.e., folds, or turns over the information processing apparatus 10. As a result, the user can view the provided information image 17a displayed on the sub-display 16.

[0060] After the user checks the content of the provided information from the AI assistant by looking at the provided information image 17a, the user opens the information processing apparatus 10 or turns it over again to view the UI image 15a on the main display 14 again and returns to the task of document creation.

[0061] The processing procedure executed by the information processing apparatus 10 for the display control of the provided information image described above is illustrated in FIG. 4.

[0062] Assume that the information processing apparatus 10 (particularly the processor 102) is in an open state and is displaying the UI image 15a of the current app on the main display 14. Also, the AI assistant operating in the background of the app analyzes the content of the user's input to the UI image 15a, and if useful information for providing to the user is found based on the analysis result, it generates provided information including that information.

[0063] The information processing apparatus 10 waits for the AI assistant to generate provided information (S10). When it detects that the AI assistant has generated provided information (the determination result in S10 is Yes), the information processing apparatus 10 prepares to display a provided information image 17a for displaying the provided information on the sub-display 16 (S12). For example, the provided information image 17a is drawn on the video memory for the sub-display 16 so that the provided information image 17a can be displayed at any time. At this point, the sub-display is in a turned-off (i.e., non-displayed) state or is displaying a predetermined standby image, and the provided information image 17a has not been displayed yet. Next, the information processing apparatus 10 executes a notification in a predetermined manner such as vibration, voice, or display of the icon 20 on the screen, indicating that there is provided information from the AI assistant (S14).

[0064] Thereafter, the information processing apparatus 10 waits for the display mechanism to be in the closed state or to be turned over while remaining in the open state (S16, S18).

[0065] When it is detected that the display mechanism has become closed (S16 is Yes) or the display mechanism has been turned over while remaining in the open state (S18 is Yes), the information processing apparatus 10 displays the provided information image 17a prepared in S12 on the sub-display 16 (S20). Here, for example, the provided information image 17a that has been previously drawn in the video memory for the sub-display 16 is displayed on the sub-display 16. At this time, the information processing apparatus 10 also turns off the main display 14 or the like so that the UI image 15a of the application that has been displayed until then cannot be visually recognized (S22). The display information of the UI image 15a of the application remains held in the video memory for the main display 14, and the state in which the UI image 15a can be displayed on the main display 14 at any time is maintained.

[0066] Next, the information processing apparatus 10 waits for the display mechanism to be in the open state or to be turned over (S24, S26). When it is detected that the display mechanism has become open (S24 is Yes) or the display mechanism has been turned over (S26 is Yes), the information processing apparatus 10 displays the UI image 15a of the application on the main display 14 by turning on the main display 14 or the like. At this time, the information processing apparatus 10 also makes the provided information image 17a invisible by turning off the sub-display 16 or the like (S28). Thereafter, the process of the information processing apparatus 10 returns to S10.

[0067] Next, with reference to FIG. 5, another example of the display control of the provided information of the AI assistant will be described.

[0068] In the example of Fig. 5, an icon 20 indicating that there is provided information from the AI assistant is displayed within the UI image 15a of the document creation app displayed on the main display 14 of the display mechanism in the open state. The icon 20 is displayed in the vicinity of the portion of the input content within the UI image 15a that the AI assistant analyzed to generate the provided information. By displaying the icon 20 in a blinking manner or in a conspicuous color, the user's attention may be strongly drawn.

[0069] When the user sees the display of this icon 20 and knows that there is provided information from the AI assistant, the user can make the sub-display 16 face the user side by closing the display mechanism or turning the entire information processing apparatus 10 over. Then, the information processing apparatus 10 displays a provided information image 17a indicating the provided information on the sub-display 16.

[0070] In the example of Fig. 5, the icon 20 is displayed, but the display for notifying the user of the existence of the provided information is not limited to the icon 20. For example, instead of this, a display showing a part of the provided information within a window smaller than the provided information image 17a of the sub-display 16 may be displayed on the UI image 15a as the display for the notification.

[0071] In the example of Fig. 5, a dialog 30 for inquiring whether to insert the information within the provided information image 17a into the UI image 15a of the app displayed on the main display 14 is displayed within the provided information image 17a. In the illustrated example, the dialog 30 inquires whether to use the graph 19 displayed within the provided information image 17a in the UI image 15a of the original app. When the user presses the "Yes" button 32 within this dialog 30, an instruction to add the graph to the empty area near the icon 20 of the UI image 15a is sent from the information processing apparatus 10 to the document creation app. The document creation app updates the display content of the UI image 15a in response to the instruction, and the updated display content is reflected in the video memory for the main display 14.

[0072] When the user finishes checking the provided information image 17a on the secondary display 16 and opens the display mechanism or flips the open display mechanism again, when the main display 14 faces the user, the main display 14 lights up again. At this time, in the UI image 15a of the document creation application displayed on the main display 14, a graph 19 is inserted according to the user's instruction for the dialog 30.

[0073] As described above, in the example of FIG. 5, a mechanism is provided to add the provided information or a part thereof displayed in the provided information image 17a on the secondary display 16 to the input content in the UI image 15a of the running application.

[0074] Next, with reference to FIG. 6, another example of display control of the provided information of the AI assistant will be described.

[0075] The example of FIG. 6 is for explaining the control when the amount of provided information generated by the AI assistant is equal to or greater than the threshold. For example, when the amount of provided information does not fit within the area of the secondary display 16, in order for the user to read all the provided information, operations such as scrolling the provided information image 17a or turning the pages displayed in the provided information image 17a are required. Such operations deteriorate the user experience. Also, even if scrolling and page turning are allowed to a certain extent, if a large amount of scrolling and page turning are required to view all the provided information, the operation burden on the user is large.

[0076] Therefore, in this example, a tolerable amount of provided information in terms of the user's operation burden is defined as a threshold. When the amount of provided information generated by the AI assistant exceeds the threshold, the information processing apparatus 10 displays a provided information image 17b indicating the provided information on the main display 14. If the provided information image 17b is displayed on the main display 14, a large amount of provided information can be displayed without scrolling or page turning, or even if an operation such as scrolling is required to display all the provided information, the amount of such operation can be reduced. Conversely, when the amount of provided information is below the threshold, the information processing apparatus 10 displays the provided information image 17a on the sub-display 16 in the same manner as in the example of FIG. 3.

[0077] However, if the provided information image 17b is suddenly displayed on the main display 14, the UI image 15a of the application may disappear, which may cause confusion to the user. Therefore, in the example of FIG. 6, when there is provided information from the AI assistant, first, the provided information image 17a is displayed on the sub-display 16, and a dialog 40 for inquiring whether to display the provided information on the main display 14 is displayed within the provided information image 17a.

[0078] When the user presses the "Yes" button within the dialog 40, the information processing apparatus 10 displays a provided information image 17b for displaying the provided information on the main display 14. In this display processing flow, the information processing apparatus 10, for example, draws a provided information image 17b adjusted to the size of the main display 14 in the video memory for the main display 14 and prepares to display it. When the user opens the closed display mechanism or flips the originally open display mechanism to face the main display 14 towards the user side, the information processing apparatus 10 displays the prepared provided information image 17b on the main display 14.

[0079] When the user presses the "No" button in the dialog 40, the dialog 40 disappears from the provided information image 17a on the sub-display 16, and the provided information is displayed in the form of scrolling or page turning within the provided information image 17a.

[0080] Also, when the provided information image 17b is displayed on the main display 14, the UI image 15b showing the display content of the running application that was previously displayed on the main display 14 may be displayed on the sub-display 16. In this case, if the UI image 15b of the running application on the sub-display 16 is in a landscape shape, the UI image 15b may be displayed horizontally on the sub-display 16 as shown in the figure. When a user who was browsing the provided information in the provided information image 17b on the main display 14 wants to check the display content of the UI image 15b of the running application, the user closes or flips the display mechanism to view the display on the sub-display 16.

[0081] Next, with reference to FIG. 7, yet another example of the display control of the provided information of the AI assistant will be described.

[0082] The information processing apparatus 10 in this example includes displays 14a and 14b that are substantially the same size as the above-described main display 14 on both sides of the open display mechanism. When the display mechanism is in the closed state, one half of the two-fold state of the outer display among the displays 14a and 14b may function as the sub-display.

[0083] In this example, when the UI image 15a of the running application is being displayed on the display 14a and provided information is provided from the AI assistant, the information processing apparatus 10 displays the provided information image 17c on the back display 14b. Instead of immediately displaying the provided information image 17c, when it is detected that the display 14b faces the user side, the control may be such that the provided information image 17c is displayed.

[0084] In this example, the provided information of the AI assistant is also displayed in a large size.

[0085] Although the embodiments of the information processing apparatus 10 have been described above, the embodiments described above are merely exemplary. Various modifications and improvements can be made within the scope of the technical idea of the present invention.

[0086] The above embodiment described the mechanism of display control of the provided information by taking, as an example, the provided information generated by the AI assistant in response to a user's input to the running application (e.g., the "document creation application" in the example of FIG. 3). However, the display control of the above embodiment is also applicable when the provided information generated by the AI assistant in response to the input content directly input by the user is displayed not for the running application but for the AI assistant. For example, while the user is working with the UI image 15a of the running application open on the main display 14 and activates the AI assistant with a predetermined call word and inputs questions or requests verbally, the AI assistant generates the provided information corresponding to the questions or requests. At this time, the provided information image for displaying the provided information is displayed not on the main display 14 but on the sub-display 16. The hiding of the display of the UI image 15a is prevented or reduced.

Description of reference numerals

[0087] 10 Information processing apparatus, 12A First body, 12B Second body, 14 Main display, 15a, 15b (UI images of the running application), 16 Sub-display, 17a, 17b, 17c Provided information images.

Claims

1. A first display device arranged on the side visible to the user in an open state, and a second display device provided on a surface that is difficult to view simultaneously with the first display device, a processor, comprising: The processor displays the information input by the user on the first display device, in a state where the input information is being displayed on the first display device, displays the information generated by the AI assistant based on the input information on the second display device, when there is information generated by the AI assistant based on the information input by the user and displayed on the first display device, causes the first display device to display that there is such generated information An information processing apparatus characterized by performing control.

2. The information input by the user is information input by voice, The information processing apparatus according to claim 1, characterized in that.

3. The information input by the user is information directly input to the AI assistant, The information processing apparatus according to claim 1, characterized in that.

4. The AI assistant is activated by voice input of a predetermined calling word by the user, The information processing apparatus according to claim 1, characterized in that.

5. A first display device arranged on the side visible to the user in an open state, a second display device provided on a surface that is difficult to view simultaneously with the first display device, In a computer comprising: causes the first display device to display the information input by the user, in a state where the input information is being displayed on the first display device, causes the second display device to display the information generated by the AI assistant based on the input information, when there is information generated by the AI assistant based on the information input by the user and displayed on the first display device, causes the first display device to display that there is such generated information A program for executing control.

Images