Information processing device and program

Abstract

To enable visual information provision by an AI assistant to be received while inhibiting a user-friendly application obtained by a display device of a large area of an information processing device having an openable and closable display mechanism from being damaged.SOLUTION: When a user performs a text input in a UI image 15a with the UI image 15a of an application displayed on a main display 14 by making a display mechanism a closed state, an AI assistant retrieves information related to the input, and generates provision information to the user. At that time, an information processing device 10 displays a provision information image 17a showing the provision information on a sub-display 16. When the user closes the display mechanism or reverses the information processing device 10, the user can look at the provision information image 17a displayed on the sub-display 16.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Flexible displays using organic electroluminescence (EL) and electronic paper have been developed. In recent years, flexible display structures to be mounted on mobile terminals have been proposed (Patent Documents 1-3), and mobile terminals with such structures are now on sale.

[0003] Foldable smartphones equipped with flexible displays are small enough to fit in one hand when closed, yet offer a large screen comparable to a tablet when unfolded. Some of these smartphones can display a smaller screen area when closed than when unfolded. For example, some smartphones that fold with the large main display facing inward have a smaller secondary display on the outer surface when closed. Other smartphones that fold with the large main display facing outward display the image on the half of the folded main display facing the user when closed. These smartphones also synchronize their screen display between the closed and unfolded states. For example, when the large screen of these smartphones is unfolded, the same app screen that was previously displayed on the small screen when closed is displayed on the large screen. Some smartphones also have a multi-window function that divides the open main display into multiple areas, each displaying a different app's user interface image.

[0004] Furthermore, the device described in Patent Document 4 displays a user interface (hereinafter referred to as UI) image on the entire surface of a foldable touch panel when the panel is opened flat. When the touch panel is folded at a certain angle, a soft keyboard is displayed on one side of the touch panel and a UI image on the other side, providing usability similar to that of a notebook PC.

[0005] In recent years, software agents called AI (artificial intelligence) assistants have become popular. AI assistants, also known as virtual assistants, understand users' instructions, questions, and other input, and provide them with services in response to that input.

[0006] Currently, the mainstream method for AI assistants on mobile devices such as smartphones is to display UI images in the display area (hereinafter referred to as the "app display area") for application programs (hereinafter referred to as "apps"), excluding the area showing the time, communication status, etc., or to display them in full-screen across the entire display area of ​​the display. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Publication No. 2017-510065 [Patent Document 2] European Patent Application Publication No. 003352044 [Patent Document 3] International Publication No. 2018 / 192426 [Patent Document 4] Japanese Patent Application Laid-Open No. 2017-130158 Summary of the Invention [Problem to be solved by the invention]

[0008] In the case where the AI ​​assistant's user interface image is displayed over the entire display area or the entire app display area of ​​the display, the AI ​​assistant's user interface image may be displayed while the user is working with an application program. In this case, the user interface image displayed in the area allocated for use of the application is covered by the user interface image displayed in the area allocated for use of the AI ​​assistant, so when the user wants to return to working with the original application, they must perform an operation to bring the application's user interface image to the foreground again.

[0009] Since foldable mobile devices have a large display area when unfolded, it is possible to display the AI ​​assistant's user interface image in part of the display area. However, if the app's user interface image is obscured by the AI ​​assistant's user interface image, the app's operability will be reduced.

[0010] The present invention aims to enable users to receive visual information from an AI assistant while preventing a decrease in the operability of apps provided by a large-area display device of an information processing device having an openable / closable display mechanism. [Means for solving the problem]

[0011] The invention of claim 1 is an information processing device comprising an openable / closable display mechanism having a first display device and a second display device having a display area equal to or smaller than the display area of ​​the first display device, and a processor, wherein the processor executes display control to display a second image showing provided information on the second display device when there is provided information provided by an AI assistant while a first image is displayed within a display area allocated to a first application program on the first display device.

[0012] The invention of claim 2 is the information processing device described in claim 1, characterized in that the display control does not display the second image when the second display device is not facing the user.

[0013] The invention of claim 3 is an information processing device as described in claim 1 or 2, characterized in that the processor further executes a notification that the provided information is available if the provided information is available while the first image is being displayed on the first display device.

[0014] The invention of claim 4 is the information processing device described in claim 3, characterized in that the notification is a process of displaying an image indicating the presence of the provided information, which image is smaller than the second image, on the first image displayed on the first display device.

[0015] The invention of claim 5 is an information processing device described in any one of claims 1 to 4, wherein the display control displays the second image on the second display device when the amount of provided information is less than a threshold, and displays the second image on the first display device when the amount of provided information is greater than the threshold.

[0016] The invention of claim 6 is the information processing device described in claim 5, characterized in that in the display control, when the second image is displayed on the first display device, an image showing the display content of the first image is displayed on the second display device.

[0017] The invention of claim 7 is an information processing device as described in claim 5 or 6, wherein, in the display control, if the amount of provided information is greater than the threshold, a query image inquiring whether or not to display the provided information on the first display device is displayed on the second display device, and if a user gives an instruction to display the query image on the first display device, the second image is displayed on the first display device.

[0018] The invention of claim 8 is a program for operating a computer having an openable / closable display mechanism including a first display device and a second display device having a display area equal to or smaller than that of the first display device, such that, while a first image is displayed in a display area allocated to a first application program on the first display device, if there is information provided by an AI assistant, a second image showing the information is displayed on the second display device. [Effects of the Invention]

[0019] According to the invention of claim 1 or 8, it is possible to receive visual information from an AI assistant while preventing a decrease in the operability of apps obtained by a large-area display device of an information processing device having an openable / closable display mechanism.

[0020] According to the invention of claim 2, the risk of information leakage from the second image displayed on the second display device that the user is not looking at can be reduced compared to when the second image is displayed regardless of the orientation of the second display device.

[0021] According to the invention of claim 3, it is possible to give the user an opportunity to look at the second display device.

[0022] According to the invention of claim 4, the amount of the hidden part of the first image can be reduced compared to when the second image is displayed on top of the first image.

[0023] According to the invention of claim 5, the number of operations required for the user to switch image displays, such as scrolling, to view the entire provided information can be reduced compared to when the second image is always displayed on the second display device regardless of the amount of provided information.

[0024] According to the invention of claim 6, by displaying the second image on the first display device, the effort required by the user to view the display of the first image can be reduced compared to when the first image is left undisplayed.

[0025] According to the invention of claim 7, it is possible to prevent the first image displayed on the first display device from becoming invisible due to the display of the second image without the user's intention. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 10 is a diagram illustrating normal display control of a foldable information processing device. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of an information processing device. [Figure 3] 10A to 10C are diagrams for explaining display control of information provided by an AI assistant in an information processing device according to an embodiment. [Figure 4] FIG. 10 is a diagram illustrating a procedure for controlling the display of information provided by an AI assistant. [Figure 5] FIG. 10 is a diagram illustrating another example of display control of information provided by an AI assistant. [Figure 6] FIG. 10 is a diagram illustrating yet another example of display control of information provided by an AI assistant. [Figure 7] FIG. 10 is a diagram illustrating yet another example of display control of information provided by an AI assistant. DETAILED DESCRIPTION OF THE INVENTION

[0027] An example of display control of a folding information processing device 10 having a large flexible display will be described with reference to Fig. 1. The information processing device 10 is, for example, a portable information terminal (e.g., a smartphone or a tablet terminal), but is not limited to this.

[0028] The information processing device 10 illustrated in FIG. 1 includes a first body 12A and a second body 12B interconnected so as to be openable and closable. A flexible main display 14 is provided on one surface of the first body 12A and the second body 12B, spanning both bodies. The main display 14 has a display area approximately equal to the combined surface area of ​​the first body 12A and the second body 12B. In this example, a sub-display 16 is provided on the surface of the first body 12A opposite to the main display 14. The sub-display 16 has a display area that fits within the surface of the first body 12A and is smaller than that of the main display 14. The sub-display 16 does not need to be flexible. The main display 14 and the sub-display 16 are configured as touch panel displays, for example. In this example, the main display 14 corresponds to the first display device, and the sub-display 16 corresponds to the second display device. Furthermore, the structure including the main display 14, the sub-display 16, and the first body 12A and the second body 12B that support them corresponds to an openable / closable display mechanism.

[0029] The left side of FIG. 1 shows (a) the information processing device 10 in a closed state, and the right side shows (b) the information processing device 10 in an open state.

[0030] The closed state refers to a state in which the display mechanism is closed to the extent that the display area of ​​the main display 14 is not fully visible from the outside. In the closed state, the display area of ​​the main display 14 is not fully visible from outside the information processing device 10, but the display area of ​​the sub-display 16 is clearly visible. In the closed state illustrated in FIG. (a), the surfaces of the first body 12A and the second body 12B facing the main display 14 are in contact or are close enough to be in contact. In this state, the angle formed by the surfaces of the main display 14 facing the first body 12A and the second body 12B (i.e., the opening angle of the display mechanism) is approximately 0 degrees. Note that defining an opening angle of approximately 0 degrees as the closed state is merely an example. Alternatively, a threshold value greater than 0 degrees may be set, and the opening angle within the range of 0 degrees to the threshold value may be defined as the closed state.

[0031] The open state is a state in which the display mechanism is open enough that the display area of ​​the main display 14 is fully visible. In the open state, the display area of ​​the sub-display 16 is not fully visible from outside the information processing device 10, but the display area of ​​the main display 14 is clearly visible. In the figure, as an example, the state in which the display mechanism is opened 180 degrees is shown as (b) open state, but this is merely an example. For example, a second threshold value less than 180 degrees may be defined, and the open state may be defined as an angle within a range of 180 degrees from the second threshold value.

[0032] When the display mechanism is in the closed state, the information processing device 10 displays the screen in the display area of ​​the sub-display 16. In the closed state, the screen display on the main display 14 may be stopped (for example, the main display 14 may be turned off). Conversely, when the display mechanism is in the open state, the information processing device 10 displays the screen in the display area of ​​the main display 14. In the open state, the screen display in the display area of ​​the sub-display 16 may be stopped.

[0033] In the illustrated example, a user interface image (hereinafter referred to as UI image) 17 showing information about a contact created by a contact book application is displayed in a display area (area surrounded by a dashed line in 17) allocated to the contact book application on the closed sub-display 16. This UI image 17 displays information such as the contact's name, address, and phone number, as well as an icon 20 for calling a phone application and an icon 22 for calling an email application. When the user opens the display mechanism from this state to the open state, the information processing device 10 starts displaying the main display 14. At this time, a UI image 15 showing information about the same contact in the same contact book application as the UI image 17 displayed in the closed state is displayed in the display area (area surrounded by a dashed line in 15) allocated to the contact book application on the main display 14. Here, because the main display 14 has a larger display area than the sub-display 16, the application program is configured to display more display content in the UI image 15 than in the UI image 17. For example, in addition to the display content that was displayed in UI image 17, UI image 15 also displays an image of a face photo of a contact user that was not displayed in UI image 17, an icon 24 for calling up an SNS (social media network) application, etc. UI image 15 of an application displayed on main display 14 is an example of a first UI image, and the display content in UI image 15 (for example, consisting of a character string, an image of a face photo, and a group of icons) is an example of the display content of the first UI image. In the illustrated example, the UI image 15 of the contacts app spans the entire display area of ​​the main display 14. However, this is merely an example. Depending on the settings of the information processing device 10 or the OS (operating system) that controls it, a predetermined area within the display area of ​​the main display 14, such as the top or bottom, may be reserved for displaying the time, communication status, notifications, or app icons. In such cases, the UI image of the app opened by the user is displayed in the remaining area of ​​the display area of ​​the main display 14 excluding the reserved area. This area in which the UI image of the app is displayed corresponds to the "display area allocated to the first application program." Note that the term "display area" "allocated to the first application program" does not necessarily mean that multiple apps each have their own display area assigned to them. It also includes the case in which the same display area is commonly assigned to multiple apps. In this case, when one of the apps is launched, the UI image of that app is displayed in the remaining common display area of ​​the display area of ​​the main display 14 excluding the reserved area.

[0034] In this specification, an application refers to an application program executed by the information processing device 10.

[0035] FIG. 2 shows an example of the main hardware configuration of information processing device 10. This hardware configuration includes elements typically found in a computer, such as processor 102, memory 104 such as random access memory, auxiliary storage device 106 such as flash memory, display control circuit 108, and input control circuit 110. It also includes a communication circuit 112 and a sensor 114 for detecting the open / closed state of the display mechanism. In the illustrated example, processor 102, memory 104, auxiliary storage device 106, display control circuit 108, input control circuit 110, communication circuit 112, and sensor control circuit 115 for controlling sensor 114 are connected to bus 120. Processor 102 exchanges data with memory 104, auxiliary storage device 106, display control circuit 108, input control circuit 110, communication circuit 112, and sensor control circuit 115 via bus 120. Note that, in the example of FIG. 2, all elements from processor 102 to sensor control circuit 115 are connected in a flat manner to the same bus 120, but this is merely an example. Alternatively, a hierarchical structure may be used in which some of the elements (e.g., a group of elements including processor 102) are integrated into a single chip, such as a System-on-a-Chip (SoC), and the remaining elements are connected to an external bus to which the chip is connected.

[0036] Here, processor 102 refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0037] Furthermore, the operations of the processor 102 may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the operations of the processor 102 are not limited to the order described in the following embodiments, and may be changed as appropriate.

[0038] The auxiliary storage device 106 is installed with the OS of the information processing device 10, various applications, and a program that interfaces with an AI assistant (to be described later) or an AI assistant on an external server. The auxiliary storage device 106 also has installed therein a program for controlling the display of information images provided by the AI ​​assistant in this embodiment (to be described later).

[0039] The display control circuit 108 controls the screen display of the main display 14 and the sub-display 16. The input control circuit 110 detects and processes touch inputs on the touch panels provided on the main display 14 and the sub-display 16, and transmits the processing results to the processor. The communication circuit 112 executes signal processing for the information processing device 10 to communicate with external network devices wirelessly or otherwise.

[0040] The sensor 114 for detecting the open / closed state of the display mechanism may be, for example, a Hall element. In this case, a magnet is provided on one of the first body 12A or the second body 12B, and a Hall element is provided on the other. When the magnet and the Hall element approach each other as the display mechanism is closed, the Hall element detects magnetism and detects that the display mechanism is in the closed state. The sensor for detecting the open / closed state of the display mechanism is not limited to a Hall element. This sensor 114 may be any of various sensors conventionally used to detect the open / closed state of an openable / closable device, such as an infrared sensor, an illuminance sensor, an acceleration sensor, a gyro sensor, a proximity sensor, etc.

[0041] The information processing device 10 may also have built-in output devices such as a speaker or a vibration mechanism.

[0042] The information processing device 10 has installed therein a program for displaying information provided by an AI assistant.

[0043] An AI assistant is a software agent that understands user input and provides the user with services according to the input. In this embodiment, we focus on a service that provides a visual display of information (hereinafter referred to as "provided information") that is generated and provided by the AI ​​assistant in response to user input, among the services provided by the AI ​​assistant.

[0044] The AI ​​assistant starts up in response to an explicit start-up instruction from the user and accepts input from the user after startup. When a UI image of an app that is set to use the AI ​​assistant is displayed on the main display 14 or the secondary display 16, the AI ​​assistant receives information entered by the user into the app from the app.

[0045] There are various methods for user input to an app or an AI assistant, such as input from a keyboard, touch input on a screen, gesture input using the user's body such as hands or fingers, voice input, image input from a camera, etc. Furthermore, the AI ​​assistant may use the location of the information processing device 10 carried by the user, which is determined using a GPS or an indoor positioning system, as input from the user.

[0046] The AI ​​assistant searches for information related to the user's input from within the information processing device 10 or from a server on a network (e.g., the Internet) to which the AI ​​assistant is connected, and generates information to be provided to the user based on the searched information. Of the information to be provided by the AI ​​assistant, the visual information to be provided that is of interest in this embodiment includes text, images, videos, videos with audio, and the like.

[0047] The AI ​​assistant may be installed in the information processing device 10 itself, or may be installed in a server on a network accessible from the information processing device 10. In the latter case, the information processing device 10 is equipped with a program that performs a process of transmitting a user's input to the AI ​​assistant on the server and a process of displaying the provided information returned from the AI ​​assistant in response to the transmission.

[0048] Consider a case where the information processing device 10 provides a user with visual information generated by an AI assistant while displaying a UI image of an application currently being executed (hereinafter referred to as the "executing application") on the main display 14. For example, when the active application is set to use an AI assistant, the AI ​​assistant generates information to be provided related to information (e.g., text information) input by the user to the UI image. Assume that the user is inputting information to the UI image to create a document, for example.

[0049] In such a case, if an image displaying information provided by the AI ​​assistant (hereinafter referred to as "provided information image") is displayed on the main display 14 so as to completely or partially cover the UI image of the running app, the UI image will become partially or completely invisible, interfering with the user's input work. Also, even if the provided information image is provided with a button for erasing the image so that the UI image of the running app is displayed when the user presses the button, the user will still have to go through the trouble of pressing the button.

[0050] Therefore, in such a case, the information processing device 10 of this embodiment displays an information image provided by the AI ​​assistant on the sub-display 16 while keeping the UI image of the running app displayed on the main display 14. The UI image of the running app displayed on the main display 14 corresponds to the first user interface image displayed on the first display device, and the information image displayed on the sub-display 16 corresponds to the second user interface image displayed on the second display device.

[0051] Such display control of the provided information image will be further described with reference to a specific example shown in FIG.

[0052] In the example of FIG. 3, a UI image 15a of a document creation application is displayed on the main display 14 of the display mechanism in the open state. This document creation application is the currently running application. The UI image 15a is a UI image for creating a document, and displays text, images, etc. input by the user. It is assumed that the document creation application is previously set to use an AI assistant.

[0053] The AI ​​assistant monitors the data (text in the illustrated example) entered by the user in the UI image 15a displayed on the main display 14, searches for information related to that data from sources such as the Internet, and generates information to be provided based on the information obtained as a result of the search. In the illustrated example, the user is creating a document titled "Report: Growth of Each Country" and has written two and a half lines of the main text. The AI ​​assistant monitors the user's input (text data in this example) in the UI image 15a, searches for information related to the input, and edits the searched information to generate information to be provided to the user. For example, the AI ​​assistant has learned to search for and generate information evaluated as useful for the input content and uses the results of this learning to generate information to be provided. In the illustrated example, the AI ​​assistant uses known keyword extraction technology to extract the keywords "United States," "2019," and "nominal GDP" from the text in the UI image 15a and searches for information closely related to these keyword pairs. This search results in information on the United States' nominal GDP in 2019 and a graph showing the closely related changes in the United States' GDP by year. The AI ​​assistant edits the information obtained by the search to generate provided information or information on the provided information image 17a that displays the provided information.

[0054] When the AI ​​assistant generates the provided information, the information processing device 10 performs display control to display a provided information image 17a indicating the provided information on the sub-display 16. This display control function may be incorporated, for example, into the OS of the information processing device 10, the AI ​​assistant installed in the information processing device 10, or a program installed in the information processing device for interfacing with an external AI assistant. Alternatively, an independent program that performs the display control function may be installed in the information processing device 10.

[0055] If the provided information image 17a were displayed on the sub-display 16 immediately upon receiving provided information from the AI ​​assistant, the provided information would be displayed on the sub-display 16 when the user is not looking at it, unnecessarily increasing the risk of information leakage. Therefore, the information processing device 10 displays the provided information image 17a on the sub-display 16 only when it is determined that there is a high probability that the user is looking at the sub-display 16. That is, the information processing device 10 displays the provided information image 17a on the sub-display 16 when it is determined that the sub-display 16 is facing the user, i.e., roughly toward the user's face. Examples of situations in which it is determined that there is a high probability that the user is looking at the sub-display 16 include when the user folds the display mechanism of the information processing device 10 or when the user turns the open information processing device 10 over and faces the sub-display 16 toward the user. In such a situation, the display control function turns on the sub-display 16 so that the provided information image 17a on the sub-display 16 is visible. Whether the user has folded the display mechanism is determined from the signal output by the sensor 114 described above. Furthermore, whether the user has turned the information processing device 10 over may be determined based on one or more signals from an acceleration sensor, a camera, a facial authentication sensor, or the like built into the information processing device 10. For example, if it is determined that the sub-display 16 is facing upward based on the direction of gravitational acceleration detected by an acceleration sensor, it is determined that the user has turned the information processing device 10 over and is now looking at the sub-display 16. It is also determined that the user is now looking at the sub-display 16 when the user's face is detected in an image captured by a camera provided on the same side as the sub-display 16, or when the user's face is recognized by a facial authentication sensor provided on that surface. Information about the user's face to be compared with the camera image or the facial authentication sensor signal may be registered in advance in the information processing device 10.

[0056] Furthermore, simply folding the display mechanism does not necessarily mean that the sub-display 16 faces the user. Therefore, in one example, when folding is detected, it is further determined whether the sub-display 16 faces the user based on a signal from an acceleration sensor or the like. If it is determined that the sub-display 16 faces the user, provided information may be displayed on the sub-display 16.

[0057] Here, the display control function of the information processing device 10 may be configured so that, when it detects that the display has been turned over and displays the provided information on the sub-display 16, the UI image 15a of the document creation application on the main display 14 is not visible. This can be achieved, for example, by turning off the main display 14. This reduces the risk that information displayed on the main display 14 will be secretly viewed by others when the user is not looking at it.

[0058] Furthermore, the information processing device 10 (particularly, the function of controlling the display of provided information) may notify the user when provided information generated by the AI ​​assistant is available. This notification may be made by vibrating the information processing device 10 in a predetermined pattern using a vibration function built into the information processing device 10. Alternatively, the notification may be made by outputting a predetermined sound from a speaker of the information processing device 10 or an audio output device such as earphones connected to the information processing device 10 via a wired or wireless connection. Alternatively, the notification may be made by lighting or blinking a lamp such as an LED (light-emitting diode) provided in the information processing device 10. As another example of the notification mode, as illustrated in FIG. 5, an icon 20 of a predetermined shape indicating the availability of provided information from the AI ​​assistant may be displayed within the UI image 15a on the main display 14. This icon 20 has a small area so as not to obscure the display content of the UI image 15a as much as possible. If the icon 20 has a smaller area than the provided information image 17a, the area of ​​the UI image 15a that is obscured is smaller than when the provided information image 17a is displayed on the main display 14. The notification may also be made by a combination of two or more of vibration, sound, lighting of a lamp, display of an icon 20 on the UI image 15a, and the like.

[0059] The user is informed by the notification that there is provided information generated by the AI ​​assistant, and closes, i.e., folds, or turns over the information processing device 10. This allows the user to view the provided information image 17a displayed on the sub-display 16.

[0060] After the user checks the content of the information provided by the AI ​​assistant by looking at the provided information image 17a, the user opens the information processing device 10 or turns it over again to look at the UI image 15a on the main display 14 again and return to the document creation work.

[0061] FIG. 4 illustrates a processing procedure executed by the information processing device 10 for display control of the image of the provided information described above.

[0062] Assume that the information processing device 10 (particularly the processor 102) is in an open state and is displaying a UI image 15a of a current app on the main display 14. Furthermore, the AI ​​assistant running in the background of the app analyzes the content of the user's input to the UI image 15a, and if it finds useful information to provide to the user based on the analysis results, it generates information to be provided that includes that information.

[0063] The information processing device 10 waits for the AI ​​assistant to generate information to be provided (S10). When it detects that the AI ​​assistant has generated information to be provided (the determination result in S10 is Yes), the information processing device 10 prepares to display the provided information image 17a, which displays the information to be provided, on the sub-display 16 (S12). For example, the provided information image 17a is drawn on the video memory for the sub-display 16 so that the provided information image 17a can be displayed at any time. At this point, the sub-display is turned off (i.e., not displayed) or is displaying a predetermined standby image, and the provided information image 17a has not yet been displayed. Next, the information processing device 10 notifies the user in a predetermined manner, such as by vibrating, sounding, or displaying an icon 20 on the screen, that information to be provided by the AI ​​assistant is available (S14).

[0064] Thereafter, the information processing device 10 waits until the display mechanism is closed or until the display mechanism is turned over while remaining in the open state (S16, S18).

[0065] When detecting that the display mechanism has entered the closed state (Yes in S16) or that the display mechanism has been turned over while still in the open state (Yes in S18), the information processing device 10 displays the provided information image 17a prepared in S12 on the sub-display 16 (S20). Here, for example, the provided information image 17a that has been rendered in advance in the video memory for the sub-display 16 is displayed on the sub-display 16. At this time, the information processing device 10 also turns off the main display 14, for example, to prevent the UI image 15a of the application that has been displayed up until then from being visible (S22). The display information for the UI image 15a of the application remains stored in the video memory for the main display 14, and the UI image 15a remains in a state in which it can be displayed on the main display 14 at any time.

[0066] Next, the information processing device 10 waits for the display mechanism to be in the open state or to be turned over (S24, S26). When it detects that the display mechanism is in the open state (Yes in S24) or that the display mechanism has been turned over (Yes in S26), the information processing device 10 displays the UI image 15a of the application on the main display 14 by, for example, turning on the main display 14. At this time, the information processing device 10 also makes the provided information image 17a invisible by, for example, turning off the sub-display 16 (S28). Thereafter, the processing of the information processing device 10 returns to S10.

[0067] Next, another example of display control of information provided by the AI ​​assistant will be described with reference to FIG.

[0068] In the example of Fig. 5, an icon 20 indicating that information provided by the AI ​​assistant is available is displayed within a UI image 15a of a document creation application displayed on the main display 14 of the display mechanism in the open state. The icon 20 is displayed near a portion of the input content in the UI image 15a that the AI ​​assistant analyzed to generate the information to be provided. The icon 20 may be displayed in a flashing display or in a conspicuous color to strongly attract the user's attention.

[0069] When the user sees the icon 20 and knows that there is information to be provided by the AI ​​assistant, the user closes the display mechanism or turns the entire information processing device 10 over so that the sub-display 16 faces the user. The information processing device 10 then displays a provided information image 17a indicating the provided information on the sub-display 16.

[0070] 5, the icon 20 is displayed, but the display for notifying the user of the existence of provided information is not limited to the icon 20. For example, instead of this, a part of the provided information may be displayed in a window smaller than the provided information image 17a on the sub-display 16, and displayed on the UI image 15a as a display for the notification.

[0071] In the example of FIG. 5, a dialog 30 is displayed in the provided information image 17a, inquiring whether or not to insert the information in the provided information image 17a into the UI image 15a of the application displayed on the main display 14. In the illustrated example, the dialog 30 inquires whether or not to use the graph 19 displayed in the provided information image 17a in the UI image 15a of the original application. When the user presses the "Yes" button 32 in the dialog 30, an instruction to add the graph to an empty space near the icon 20 in the UI image 15a is sent from the information processing device 10 to the document creation application. In response to the instruction, the document creation application updates the display content of the UI image 15a, and the updated display content is reflected in the video memory for the main display 14.

[0072] When the user has finished checking the provided information image 17a on the sub-display 16 and opens the display mechanism or turns the open display mechanism over again so that the main display 14 faces the user, the main display 14 is turned on again. At this time, the graph 19 has been inserted into the UI image 15a of the document creation application displayed on the main display 14 in response to the user's instruction to the dialog 30.

[0073] As described above, in the example of Figure 5, a mechanism is provided for adding the provided information displayed in the provided information image 17a on the secondary display 16 or a part of it to the input content in the UI image 15a of the running application.

[0074] Next, with reference to FIG. 6, yet another example of display control of information provided by the AI ​​assistant will be described.

[0075] The example in FIG. 6 is intended to explain control when the amount of provided information generated by the AI ​​assistant is equal to or greater than a threshold. For example, if the amount of provided information does not fit within the area of ​​the sub-display 16, the user will need to perform operations such as scrolling the provided information image 17a or turning pages displayed on the provided information image 17a in order to read the entire provided information. Such operations impair usability for the user. Furthermore, even if scrolling and page turning are permitted to a certain extent, the need to perform a large amount of scrolling or page turning to view the entire provided information places a heavy operational burden on the user.

[0076] Therefore, in this example, a threshold is set for the amount of information to be provided that is tolerable in terms of the user's operational burden. If the amount of information to be provided generated by the AI ​​assistant exceeds the threshold, the information processing device 10 displays a provided information image 17b indicating the information to be provided on the main display 14. Displaying the provided information image 17b on the main display 14 makes it possible to display a large amount of information to be provided without scrolling or turning pages, or even if an operation such as scrolling is required to display all of the information to be provided, the amount of operation required can be kept to a minimum. Conversely, if the amount of information to be provided is equal to or less than the threshold, the information processing device 10 displays a provided information image 17a on the sub-display 16, as in the example of FIG. 3.

[0077] However, if the provided information image 17b is suddenly displayed on the main display 14, the UI image 15a of the app will disappear, which may confuse the user. Therefore, in the example of Fig. 6, when there is provided information from the AI ​​assistant, the provided information image 17a is first displayed on the sub-display 16, and a dialog 40 is displayed within the provided information image 17a, asking whether or not to display the provided information on the main display 14.

[0078] When the user presses the "Yes" button in this dialog 40, the information processing device 10 displays the provided information image 17b, which displays the provided information, on the main display 14. In this display processing flow, the information processing device 10, for example, draws the provided information image 17b that matches the size of the main display 14 in the video memory for the main display 14 and prepares to display it. When the user opens a closed display mechanism or turns the originally open display mechanism over so that the main display 14 faces the user, the information processing device 10 displays the prepared provided information image 17b on the main display 14.

[0079] If the user presses the "No" button in the dialog 40, the dialog 40 disappears from the provided information image 17a on the sub-display 16, and the provided information is displayed in the provided information image 17a in a scrolling or page-turning manner.

[0080] Furthermore, when the provided information image 17b is displayed on the main display 14, a UI image 15b indicating the display content of the running application that was previously displayed on the main display 14 may be displayed on the sub-display 16. In this case, if the UI image 15b of the running application on the sub-display 16 is horizontally long, the UI image 15b may be displayed horizontally on the sub-display 16 as shown in the figure. When a user who has been viewing the provided information using the provided information image 17b on the main display 14 wants to check the display content of the UI image 15b of the running application, he or she closes or turns over the display mechanism to view the display on the sub-display 16.

[0081] Next, with reference to FIG. 7, yet another example of display control of information provided by the AI ​​assistant will be described.

[0082] In this example, the information processing device 10 has displays 14a and 14b, each of which is approximately the same size as the main display 14, on both sides of the display mechanism in the open state. When the display mechanism is in the closed state, the outer half of the displays 14a and 14b in the folded state may function as a sub-display.

[0083] In this example, when the UI image 15a of the running application is displayed on the display 14a and provided information is provided from the AI ​​assistant, the information processing device 10 displays the provided information image 17c on the rear display 14b. Instead of immediately displaying the provided information image 17c, the information processing device 10 may be controlled to display the provided information image 17c when it is detected that the display 14b has turned toward the user.

[0084] In this example, the information provided by the AI ​​assistant is also displayed in a larger size.

[0085] Although the embodiment of the information processing device 10 has been described above, the embodiment described above is merely illustrative and various modifications and improvements are possible within the scope of the technical concept of the present invention.

[0086] The above embodiment has described a mechanism for controlling the display of provided information, taking as an example the provided information generated by the AI ​​assistant in response to a user's input to a running app (e.g., the "document creation app" in the example of FIG. 3). However, the display control of the above embodiment can also be applied to a case where provided information generated by the AI ​​assistant in response to input directly entered by the user to the AI ​​assistant, rather than to a running app, is displayed. For example, if a user opens the UI image 15a of the running app on the main display 14 and works while activating the AI ​​assistant with a predetermined call phrase and vocally inputs a question or request, the AI ​​assistant generates provided information corresponding to the question or request. In this case, the provided information image displaying the provided information is displayed on the sub-display 16, not on the main display 14. This prevents or reduces the display of the UI image 15a from being hidden. [Explanation of symbols]

[0087] 10 Information processing device, 12A First body, 12B Second body, 14 Main display, 15a, 15b UI images (of running applications), 16 Sub-display, 17a, 17b, 17c Provided information images.

Claims

1. an openable / closable display mechanism having a first display device and a second display device having a display area equal to or smaller than that of the first display device; a processor; Equipped with The processor: and executing display control for displaying, on the second display device, a second image indicating provided information provided by an AI assistant, while a first image is displayed in a display area allocated to a first application program on the first display device.

1. An information processing device comprising:

2. 2 . The information processing apparatus according to claim 1 , wherein the display control does not display the second image when the second display device is not facing the user.

3. The processor further comprises: If there is provided information while the first image is being displayed on the first display device, a notification that there is provided information is executed.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. 4. The information processing device according to claim 3, wherein the notification is a process of displaying an image indicating that the provided information is present on the first image displayed on the first display device, the image being smaller than the second image.

5. The information processing device according to any one of claims 1 to 4, wherein the display control displays the second image on the second display device when the amount of the provided information is equal to or less than a threshold, and displays the second image on the first display device when the amount of the provided information is greater than the threshold.

6. 6. The information processing device according to claim 5, wherein, in the display control, when the second image is displayed on the first display device, an image showing the display content of the first image is displayed on the second display device.

7. 7. The information processing device according to claim 5 or 6, wherein, in the display control, if the amount of the provided information is greater than the threshold, a query image inquiring whether or not to display the provided information on the first display device is displayed on the second display device, and if a user gives an instruction to display the query image on the first display device, the second image is displayed on the first display device.

8. A computer including an openable / closable display mechanism having a first display device and a second display device having a display area equal to or smaller than that of the first display device, A program for causing the device to operate in such a way that, while a first image is displayed in a display area allocated to a first application program on the first display device, if there is provided information provided by an AI assistant, a second image showing the provided information is displayed on the second display device.

Images