Add-in program, information processing device, control method

Abstract

To make it easier to create prompts for the generation AI to generate objects. [Solution] The add-in program of the present invention acquires at least one of the object information of a first object selected by the user on the application's operation screen and the area information of an area selected by the user on the operation screen, and further generates a prompt for generating a second object based on the acquired information and the operation request entered by the user, and sends the generated prompt to the server that generates the second object.

Description

Technical Field

[0001] This proposal relates to an add-in program for generating a prompt and sending the prompt to a generative AI.

Background Art

[0002] Systems for easily creating presentation materials are known. For example, in Patent Document 1, a method is proposed for generating appropriate slides by narrowing down from design candidates prepared based on information such as input text and the age and gender of the user, and reflecting the text in the design.

[0003] In recent years, there has been an increase in AI assistant tools that assist in creating presentation materials using generative AI. For example, in Microsoft's PowerPoint (trademark), when a natural language request such as "Please add a slide about the history of women's soccer" is made to the AI assistant Copilot, the slide is created and added.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology described in Patent Document 1 allows users to edit slides according to their intentions by changing the text content and input locations, following the format of design candidates, but the available designs are limited. In contrast, the technology that uses natural language to have a generative AI (AI assistant) create slides makes it possible to create slides that are not limited to default designs. Furthermore, giving instructions to the generative AI in natural language has the advantage that even rough instructions from the user can be interpreted by the generative AI and used to create slides. However, because the generative AI determines the user's intentions, slides containing objects that the user did not intend may be created. In this case, even if the user tries to instruct the generative AI to change some of the multiple objects already placed on the slide, it is difficult to identify the objects to be changed using only natural language instructions. [Means for solving the problem]

[0006] The add-in program of the present invention is an add-in program that is added to an application, and is characterized in that it causes a computer to function as follows: a first acquisition means for acquiring object information of a first object selected by a user on the operation screen of the application and area information of an area selected by the user on the operation screen of the application; a second acquisition means for acquiring an operation request entered by the user; a prompt generation means for generating a prompt for generating a second object based on the information acquired by the first acquisition means and the operation request acquired by the second acquisition means; and a transmission means for sending the generated prompt to a server that generates the second object. [Effects of the Invention]

[0007] According to the present invention, a prompt for generating a second object can be easily created based on at least one of the first objects or areas selected by the user on the application's operation screen. [Brief explanation of the drawing]

[0008] [Figure 1] Network Configuration Diagram of This Embodiment [Figure 2] Client PC Hardware Configuration Diagram [Figure 3] Hardware configuration diagram of the generation AI server [Figure 4] Software configuration diagram according to this embodiment [Figure 5] An example of an application's user interface with an add-in program applied. [Figure 6] An example of the operation screen when the first text contained in an object is selected. [Figure 7] An example of the operation screen when selecting an area where no objects exist as the first object. [Figure 8] An example of an operation screen when displaying suggested operation requests. [Figure 9] An example of an operation screen when an AI assistant operation screen does not exist. [Figure 10] An example of a sequence diagram illustrating this embodiment. [Figure 11] An example of the process for generating an operation request confirmation message. [Figure 12] An example of prompt generation process [Figure 13] Example of object information [Figure 14] An example of a prompt generated by an add-in program [Modes for carrying out the invention]

[0009] The embodiments for carrying out the present invention will be described below with reference to the drawings. Note that the following embodiments are not intended to limit the invention as defined in the claims, and not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] (First Embodiment) <System Configuration> As shown in FIG. 1, the network configuration diagram according to this embodiment includes a computer 1000 which is a terminal device, an application 2000, an add-in program 3000, a generative AI server 4000, and the Internet 5000.

[0011] The computer 1000 is, for example, placed in an office and is connected to the external Internet 5000 via an in-house network (LAN: not shown) and a router (not shown). Here, the computer 1000 is an example of a user terminal (an information processing device used by a user), and the generative AI server 4000 is an example of an information processing device (server) that provides a generative AI service using a large language model (learning model).

[0012] The application 2000 is an application that operates on the computer 1000 and refers to something that enables the use of an AI assistant function by the add-in program 3000. This AI assistant function refers to something that receives instructions in natural language form from a user, communicates with the generative AI server 4000, and generates and outputs an answer using the generative AI. The add-in program 3000 is a program added to the application 2000 and called from the application 2000, and has a function of communicating with the generative AI server 4000 and providing the product of the generative AI server 4000 to the application 2000.

[0013] The generative AI server 4000 is communicably connected via the Internet 5000 to the application 2000 and the add-in program 3000 operating on the computer 1000. The generative AI server 4000 is a server managed by an operator that provides the add-in program 3000 or an operator that provides the generative AI service.

[0014] In this embodiment, the application 2000, the add-in program 3000, and the generative AI server 4000 may be collectively referred to as an "AI assistant system". Note that the hardware constituting the generative AI server 4000 and the computer 1000 may be separate or may exist on the same hardware as a single entity. Also, the application 2000 may operate on a computer or may be a web application connected via the Internet 5000. When the application 2000 is realized as a web application, the add-in program 3000 may take the form of an optional component selectable in the web application.

[0015] <Hardware Configuration> Referring to FIGS. 2 and 3, the hardware configuration of each device constituting the AI assistant system according to this embodiment will be described. FIG. 2 shows the hardware configuration of the computer 1000. FIG. 3 shows the hardware configuration of the generative AI server 4000.

[0016] As shown in FIG. 2, the computer 1000 includes a display unit 1010, an operation unit 1020, a storage unit 1030, a control unit 1040, and a network communication unit 1050, which are communicably connected to each other. The computer 1100 is not particularly limited, and for example, a desktop or laptop personal computer, a tablet terminal, a smartphone, etc. can be applied. The control unit 1040 includes a CPU (Central Processing Unit) 1041 and a memory 1042, and controls the entire computer 1000.

[0017] The display unit 1010 is composed of a display such as an LCD panel and can display images, etc. The operation unit 1020 is composed of a mouse, keyboard, etc. and can accept input operations from the user. The storage unit 1030 is composed of a storage medium such as a hard disk or SSD and stores various programs (software) necessary for the operation of the computer 1000. Programs are loaded into memory 1042 as needed and executed by the CPU 1041. Programs also include applications 2000 and add-in programs 3000. The CPU 1041 realizes various functions described later by executing various programs. Note that these programs are not limited to those stored in the computer 1000. Programs may be stored in both the computer 1000 and the generation AI server 4000, or they may be distributed and stored in both the computer 1000 and the generation AI server 4000. The network communication unit 1050 performs data input and output with external devices via an external network.

[0018] The generation AI server 4000 has a display unit 4010, an operation unit 4020, a storage unit 4030, a control unit 4040, and a network communication unit 4050, which are connected to each other so as to be able to communicate with one another. The control unit 4040 is composed of a CPU 4041, a memory 4042, and a GPU (Graphics Processing Unit) 4043, and controls the entire generation AI server 4000. As the hardware configuration of the generation AI server 4000 is almost the same as that of computer 1000, a detailed explanation will be omitted.

[0019] <Software Configuration> Referring to Figure 4, the software configuration of each device constituting the AI ​​assistant system according to this embodiment will be described.

[0020] As shown in Figure 4, the add-in program 3000 is a program that provides the application 2000 with an operation request acquisition function 3100, a prompt generation function 3200, a generation AI server communication function 3300, and a response output function 3400.

[0021] Application 2000 is, for example, presentation software, which creates presentation slides by arranging objects such as diagrams, photographs, tables, and text boxes on the object manipulation screen 2200 based on user instructions. The object information management function 2300 stores and manages information about the placed objects. The menu processing function 2100 displays a pop-up menu when the mouse cursor is on the object manipulation screen 2200 and a right-click operation is performed with the mouse. At this time, if the add-in program 3000 is added, an option to use the AI ​​assistant is added to the pop-up menu. Note that the operation to display the pop-up menu is not limited to a right-click with the mouse; for example, if it is a touch panel, the pop-up menu may be displayed when the selected object is long-pressed (touch and hold operation). The AI ​​assistant processing function 2400 displays, for example, the AI ​​assistant operation screen to receive operation requests from the user to the AI ​​assistant and displays responses from the AI ​​assistant. The operation request management function 2500 holds the operation requests entered by the user and passes these held operation requests to the prompt generation function 3200 of the add-in program. The object editing function 2600 receives objects generated by the generation AI server via the add-in program and outputs them to a specified area on the object manipulation screen. Note that the applications 2000 to which the add-in program 3000 is applicable are not limited to presentation software. For example, it can also be used with word processing software and design editing software, and is applicable to any application equipped with an AI assistant processing function that can work in conjunction with the add-in program 3000.

[0022] <Operation Request Acquisition Function> The operation request acquisition function 3100 includes a menu display process 3110, a first object information acquisition process 3120, an operation request confirmation statement generation process 3130, an AI assistant display process 3140, and an operation request confirmation statement output process 3150. In this embodiment, an operation request refers to a statement indicating the process that the user wants to have executed using the AI ​​assistant system based on the object (or area) selected by the user, such as "list (the selected object) in bullet points."

[0023] The menu display process 3110 provides the menu processing function 2100 of the application 2000 with a function to display the menu according to this embodiment. This menu is, for example, a pop-up menu that appears when the user right-clicks the mouse while an object or area is selected on the object manipulation screen 2200. In this embodiment, an option to use the AI ​​assistant is provided in this pop-up menu, allowing the user to activate and use the AI ​​assistant by selecting this option. Note that an object refers to anything displayed on the object manipulation screen 2200, such as a figure, photograph, table, or text box. The user may also select any range of text within a text box displayed on the object manipulation screen as an object. Furthermore, in this embodiment, the menu display process 3110 can also be called by selecting any area on the object manipulation screen 2200 (an area specified by specifying a coordinate position on the screen) instead of an object such as a figure, photograph, table, or text box. In addition, non-objects such as slides in a presentation system or layers in design editing software may also be selected.

[0024] The first object information acquisition process 3120 provides a function to acquire object information from the object information management function 2300 regarding the object selected by the user when the user selects to use the AI ​​assistant from the pop-up menu displayed by the menu display process 3110. Hereafter, the object selected by the user on the object operation screen 2200 will be referred to as the first object, and the object newly generated by the generation AI server 4000 will be referred to as the second object. Multiple objects may be selected as the first object. Object information refers to information used in application 2000 to process objects, such as the ID, type, size, coordinates, and file path of the object selected by the user. The first object information acquisition process 3120 may also acquire the text contained in the first object. This text is represented as a pair of text content and text information. Of these, text information includes formatting information such as the language setting, size, expansion setting, color, and indentation of the text, and text content refers to the content of the written text (i.e., the string) itself.

[0025] The operation request confirmation statement generation process 3130 provides a function to generate an operation request confirmation statement when a user requests the use of the AI ​​assistant from a pop-up menu. For example, this operation request confirmation statement refers to wording intended to confirm the process that the user wants to perform on the object selected by the user using the AI ​​assistant system, such as "Please tell me what you want to do with this object." This operation request confirmation statement may be a pre-prepared template in the AI ​​assistant system, or it may be wording that is changed according to the object information and area information acquired in the first object information acquisition process 3120. The operation request confirmation statement generation process 3130 may also generate the operation request confirmation statement using the generation AI server 4000, and in that case, it may refer to the log information held by the AI ​​assistant processing function 2400.

[0026] The AI ​​assistant display process 3140 provides a function to display the AI ​​assistant operation screen using the AI ​​assistant processing function 2400 when requested by the user. The AI ​​assistant operation screen may be displayed within the object operation screen 2200, or it may be displayed as a separate window pop-up. If the AI ​​assistant operation screen is already open and contains information such as already executed operation requests, the AI ​​assistant operation screen may be initialized to display the executed information as a new AI assistant operation screen.

[0027] The operation request confirmation message output process 3150 provides a function that, when a user instructs the AI ​​assistant to use the AI ​​assistant from a pop-up menu, outputs the operation request confirmation message generated by the operation request confirmation message generation process 3130 to the AI ​​assistant operation screen using the AI ​​assistant processing function 2400.

[0028] <Prompt generation function> The prompt generation function 3200 includes an operation request acquisition process 3210 and a prompt generation process 3220.

[0029] The operation request acquisition process 3210 provides a function to acquire operation requests entered by the user on the AI ​​assistant operation screen from the operation request management function 2500.

[0030] The prompt generation process 3220 provides a function to generate a prompt for input to the generation AI server 4000 based on the object information and area information obtained by the first object information acquisition process 3120 and the operation request obtained by the operation request acquisition process 3210. The prompt generated in this embodiment is a statement generated by combining information that identifies the selected object or area with the operation request. For example, if the user selects the object "text box with object ID=1" and the operation request is "make it a bulleted list", the prompt generation process 3220 combines these to generate the prompt "Please change the text contained in the text box with object ID=1 to a bulleted list." In particular, with generation AIs that handle large-scale language models, the accuracy of the generated response can vary greatly depending on how the instructions are given, so it is important to input prompts that are easy for the generation AI to understand. For this reason, when generating a prompt, it is also possible to format the prompt into a format that is easy for the generation AI to understand, such as Markdown. Alternatively, elements other than object information and operation requests may be added, such as the processing policy, output method, and prohibited actions that the generating AI will perform.

[0031] Furthermore, it may be possible to create a prompt to cause the AI ​​generation server to generate content by referencing an image file that is not placed on the object manipulation screen. For example, suppose an area is selected on the object manipulation screen, and the user inputs an operation request to the AI ​​assistant such as "remake an image from another file to have a brighter atmosphere and place it in the selected area." In this case, it is necessary to reference a different file from the object already placed on the object manipulation screen, so the operation request confirmation message output process 3150 should be used to display a user interface and message on the screen that prompts the user to specify the other file. Then, the prompt generated by the prompt generation process 3220 and the other file specified in the user interface should be passed to the prompt transmission process 3310 and sent to the AI ​​generation server.

[0032] <Generating AI Server Communication Function> The generation AI server communication function 3300 includes a prompt sending process 3310 and a response receiving process 3320.

[0033] The prompt transmission process 3310 provides a function to send the prompt generated by the prompt generation process 3220 to the prompt receiving function 4100 of the generation AI server 4000 to request content generation.

[0034] The AI ​​server's response statement generation function 4200 and second object generation function 4300 interpret the prompt received by the prompt reception function 4100 and generate a response statement and a second object. The response transmission function 4400 then sends the response statement generated by the response statement generation function 4200 and the second object generated by the second object generation function 4300 to the add-in program's response reception process 3320.

[0035] The response receiving process 3320 of the add-in program provides a function to receive responses such as response statements and second objects generated by the generation AI server 4000. This response may include two types of content: a response statement displayed on the AI ​​assistant operation screen, such as "The selected object has been modified into a bulleted list," and a second object displayed on the object operation screen 2200. Alternatively, this response may be parameters necessary to obtain file content including the second object, such as a link to storage where the file of the second object generated by the generation AI server 4000 is saved. Note that the second object generated by the generation AI server is not limited to images or text, but may also be code or macros. In that case, for example, the code or macro received by the response receiving process 3320 may be embedded in the presentation being created in application 2000.

[0036] <Response output function> The response output function 3400 includes a response statement output process 3410 and a second object output process 3420.

[0037] The response output processing unit 3410 provides a function to display the response received from the AI ​​server 4000 by the response reception processing unit 3320 on the AI ​​assistant operation screen using the AI ​​assistant processing function 2400.

[0038] The second object output process 3420 provides a function to output the second object received from the generation AI server 4000 by the response reception process 3320 to the application's object editing function 2600, and output it within a specified area on the object operation screen. Alternatively, the second object received from the generation AI server 4000 by the response reception process 3320 may be temporarily displayed on the AI ​​assistant operation screen for user confirmation. In that case, the second object may be pasted to a specified location on the object operation screen in response to an instruction from the user to apply the confirmed second object. Furthermore, if the response reception process 3320 has received a storage link, it may also provide a function to obtain the second object from the received link and output it to the application 2000. Furthermore, if a tabular second object is generated based on a prompt such as "Convert the contents of this text box to a tabular format," the second object output process 3420 may instruct the object editing function 2600 to replace the text box (first object) with the tabular second object, thereby directly updating the object information managed by the object information management function 2300.

[0039] The above is a description of the software configuration of each device constituting the AI ​​assistant system according to this embodiment.

[0040] <Example of operation screen> Referring to Figure 5, an example of an operation screen provided to application 2000 by the add-in program 3000 according to this embodiment will be described.

[0041] In the object manipulation screen 2200 displayed by application 2000, for example, if the first object 2210 is selected by user operation and a right-click operation is performed with the mouse, a pop-up menu (menu bar) 2110 is displayed. At this time, the add-in program 3000 controls the menu bar 2110 so that an option (menu 2111) for using the AI ​​assistant is displayed. Furthermore, the add-in program 3000 also provides a menu button 2121 for calling up the AI ​​assistant on the menu bar 2120 displayed by application 2000. That is, instead of the user right-clicking on the selected object and specifying the option 2111 from the displayed pop-up menu, the user can also call up the AI ​​assistant by selecting object 2210 and then specifying the menu button 2121.

[0042] In the example in Figure 5, "Text Box 2" is shown as the first object 2210. However, the selection target is not limited to "Text Boxes"; it could also be the text box of "Title 1" or the drawing in "Figure 3." In Figure 5, each object is simply represented by a rectangle, but in reality, it should be a string of text, a drawing, etc. Also, in the example in Figure 5, the entire text box is shown as the first object. However, this is not the only option; it may be possible to select only a portion of the text within the text box as the first object. For example, as shown in Figure 6, when text is contained within text box 2210, the user may be able to select any portion of that text 2211 as the first object.

[0043] Furthermore, as shown in Figure 7, to display a pop-up menu by selecting any area 2212 on the object manipulation screen 2200, the user simply needs to specify a rectangular area 2212 using a mouse or the like and right-click on that area. In this case, the prompt generation process 3220 will generate a prompt based on the operation request and information such as the coordinates and size of the selected area (hereinafter referred to as area information).

[0044] Then, as shown in Figure 5, when the user executes menu 2111 or menu button 2121, the add-in program 3000 launches the AI ​​assistant operation screen 2410 and outputs an operation request confirmation message 2411 to the AI ​​assistant operation screen 2410. At this time, the operation request confirmation message 2411 will display a message corresponding to the selected object or area, as will be described later. When the user enters an operation request in the AI ​​assistant input field 2420 and confirms it by pressing the return key or the like, the add-in program 3000 displays the entered operation request 2412 on the AI ​​assistant operation screen 2410. Then, based on the entered operation request and the information of the selected object or area, the add-in program 3000 generates a prompt and sends the generated prompt to the generating AI server.

[0045] As shown in Figure 8, when displaying the operation request confirmation statement 2411 on the AI ​​assistant operation screen 2410, a list of operation request candidates 2421 may be displayed depending on the type of first object 2210 selected by the user. The user may then select a candidate from the displayed list and input it as an operation request into the input field 2420. In this case, the content of the operation request 2412 displayed on the AI ​​assistant operation screen 2410 may be the same as the description displayed in candidate 2421, or it may include a more detailed description as shown in Figure 8 (with the description shown in parentheses added, as in operation request 2412 in Figure 8). These operation request candidates are prepared in advance by the add-in program 3000, and it is preferable that they be displayed according to the type of selected object or the type of selected area.

[0046] When the add-in program 3000 sends a prompt to the generation AI server 4000, it receives a response from the generation AI server. The response received from the generation AI server 4000 includes the second object and response statement generated by the generation AI server 4000 based on the prompt. Once the add-in program 3000 receives the response from the generation AI server, it outputs the response statement 2413 to the AI ​​assistant operation screen 2410 and updates the first object 2210 with the second object generated by the second object generation function 4300.

[0047] Furthermore, in the case of an application 2000 that does not have an area for the AI ​​assistant operation screen 2410 as shown in Figure 5, an example of an operation screen when the add-in program 3000 provides the AI ​​assistant function will be explained with reference to Figure 9. In Figure 9, when the first object 2210 on the object operation screen 2200 is selected, the add-in program 3000 displays the menu 2111 when the pop-up menu (menu bar) 2110 is displayed by right-clicking with the mouse. Furthermore, when the mouse cursor is pointed over the menu 2111, the operation request input field 2112 is displayed. This operation request input field 2112 has the same function as the AI ​​assistant input field 2420 in Figure 5, and allows the user to input an operation request. Alternatively, the add-in program 3000 may provide an operation request input button 2122 on the menu bar 2120. The operation request input button 2122 also has the same function as the AI ​​assistant input field 2420, and allows the user to input an operation request.

[0048] <Sequence> Referring to Figure 10, the sequence of the AI ​​assistant system implemented between the user, application 2000, add-in program 3000, and generation AI server 4000 in this embodiment will be described. The numbers following "S" below indicate the step numbers in the sequence.

[0049] First, in S6001, application 2000 changes the status of an object or area to a selected state according to the user's selection operation. This selected state is the status that is transitioned to when the user clicks an object on the object manipulation screen 2200. For example, the background color of a selected object is changed so that the user can identify that it is selected. In the case of selecting an area, for example, the user can specify the top-left and bottom-right coordinates of the desired area on the object manipulation screen using the mouse pointer or the like to make it selectable.

[0050] Next, in S6002, as explained in Figure 5, when the user right-clicks the mouse on the selected object or area, application 2000 displays a pop-up menu. When the application detects that the user has selected the "Operate with AI Assistant" menu from the pop-up menu, in S6003, it starts the add-in program 3000. Note that the processing in S6003 is in accordance with the specifications of application 2000, but if add-in program 3000 is already running, it is sufficient to simply notify the running add-in program that it has been called.

[0051] In S6004, the add-in program 3000 obtains object information for the first selected object or area information for the selected area from application 2000.

[0052] In S6005, the add-in program 3000 generates an operation request confirmation statement. Preferably, this operation request confirmation statement is worded according to the selected object or area. Details of the operation request confirmation statement generation process will be described later with reference to Figure 11. Also, specific examples of object information obtained in S6004 will be described later with reference to Figure 13, but the operation request confirmation statement may be a standard phrase. Alternatively, for example, wording that exemplifies the executable operations pre-configured based on the type of the first object may be added, such as "What operation would you like to perform on this text box? For example, you can highlight important text, modify the text into a bulleted list, summarize the text, etc."

[0053] In S6006, the add-in program 3000 instructs application 2000 to launch the AI ​​assistant operation screen. For example, as shown in Figure 5, an area for giving instructions to the AI ​​assistant in a chat format (AI assistant operation screen 2410) is displayed within the window displayed by application 2000. Note that the AI ​​assistant operation screen 2410 is not limited to being displayed within the application window; it may also be displayed as a separate window. If the AI ​​assistant operation screen 2410 is already launched, the process in S6006 may be skipped.

[0054] In S6007, the add-in program 2000 outputs and displays an operation request confirmation message on the AI ​​assistant screen 2410 displayed by application 2000.

[0055] Note that the processing described above in S6004, S6006, and S6007 is just an example, and the method of information exchange may be modified as appropriate according to the specifications of Application 2000.

[0056] After processing S6007, the user may select a different object or area without entering an operation request, and then select "Operate with AI Assistant" from the pop-up menu. In this case, while retaining the previously acquired object and area information, the system will acquire additional object information for the newly selected object and area information for the other area. In this case, processing S6001 to S6007 will be repeated.

[0057] In S6008, application 2000 receives an operation request from the user. Subsequently, in S6009, add-in program 3000 retrieves the input operation request from application 2000.

[0058] Then, in S6010, the add-in program generates a prompt to instruct the generation AI server based on the operation request obtained in S6009 and the object information and area information of the first object obtained in S6004. Details of the generation of this prompt will be described later using Figure 12. A specific example of the prompt generated in S6010 will also be described later using Figure 14.

[0059] Furthermore, when analyzing operation requests and creating prompts, it may be determined that the user has made a request to reference an external file. In that case, S6011 will request the application to display a screen for specifying the external file to be referenced, and obtain information about the external file specified by the user through that screen.

[0060] In S6012, the add-in program sends the generated prompt to the generation AI server 4000. This prompt sent to the generation AI server includes object information and area information obtained in S6004, as well as information about external files obtained in S6011. The detailed communication procedure for sending the prompt in S6012 may be modified as appropriate in accordance with the specifications of the generation AI server 4000 to which the prompt is sent.

[0061] Upon receiving a prompt, the generating AI server 4000 inputs the prompt into its learning model (generating AI), thereby generating a response statement (S6013) and a new object (second object) (S6014). Then, in S6015, the generating AI server 4000 returns the generated response statement and second object to the add-in program 3000 as a response to the prompt.

[0062] In S6016, the add-in program 3000 outputs the response text contained in the response received in S6015 to the AI ​​assistant operation screen of application 2000.

[0063] Furthermore, if the information acquired in S6004 includes area information (i.e., an area was specified by the user), the prompt generated in S6010 instructs the system to generate a new object corresponding to the size of that area information. Therefore, the new object (second object) generated in S6014 by the generation AI server is an object suitable for that area. Accordingly, in S6017, the add-in program 3000 outputs and displays the second object in the specified area on the object operation screen.

[0064] Furthermore, if the information acquired in S6004 does not include area information (i.e., the user did not specify an area and only the first object was selected), the prompt generated in S6010 will not include area information. Therefore, the new object (second object) generated in S6014 by the generating AI server is based on the object information of the first object, and is generated based on the size, type, and content of the first object. Accordingly, in S6018, the add-in program 3000 outputs and displays the second object on the AI ​​assistant operation screen. The AI ​​assistant operation screen then allows the user to choose whether to replace the first object in the object operation screen with the second object, or to add the second object to the object operation screen. If application 2000 does not have the function to display the AI ​​assistant operation screen, it may be displayed in a predetermined position (e.g., the center) within the object operation screen. Whether to display the second object on the AI ​​assistant operation screen or the object operation screen in S6018 should be determined appropriately according to the specifications of application 2000.

[0065] Finally, the sequence in Figure 10 ends after the above processing, but if the user has other operation requests, the same sequence will be executed again from S6001. In that case, when generating a prompt in S6010, the AI ​​assistant processing function 2400 may retrieve the previous chat log and append it to the prompt. This chat log includes, for example, the operation request information previously entered by the user, the object information and area information of the previously selected first object, and the object information of the generated second object. Alternatively, the object information of the generated second object may be converted into an image format and saved, so that the saved second object can be recalled at any time based on the user's instructions.

[0066] Referring to Figure 11, the details of the generation and output processes for the operation request confirmation statements S6005 to S6007 will be explained. Note that the process in Figure 11 is just one example and is not limited to this method.

[0067] In S1101, the add-in program determines whether the information obtained in S6004 includes object information. If the processes S6001 to S6007 are repeated, multiple pieces of information have already been obtained via S6004, so the program determines whether any of them contain object information.

[0068] Furthermore, in S1102, the add-in program determines whether the information obtained in S6004 includes area information.

[0069] If the results of the S1101-S1102 check indicate that the acquired information includes object information but does not include area information, then in S1103, a message prompting the user to either input an operation request or select an additional area is generated and output.

[0070] If the results of the S1101-S1102 checks that the acquired information includes both object information and area information, then in S1104, a message prompting the user to enter an operation request is generated and output.

[0071] If the results of the S1101-S1102 checks that the acquired information does not include object information (but includes area information), then in S1105, a message prompting the user to either input an operation request or select an additional object is generated and output.

[0072] Note that in S1004, only a message prompting the user to enter an operation request is displayed, but it does not force the user to enter an operation request; at this point, it is up to the user to decide whether or not to enter an operation request. Therefore, in S1106, it is determined whether an operation request has been entered by the user or whether additional objects or areas have been selected. If an operation request has been entered, the add-in program retrieves the operation request and generates a prompt in S6009-S6010. If the user does not enter an operation request and further objects or areas are selected, the processes in S6001-S6007 are repeated, and the process shown in Figure 11 is also repeated.

[0073] Next, with reference to Figure 12, the details of the prompt generation process for S6010 will be explained. Note that the process in Figure 12 is just one example and is not limited to this method.

[0074] In S1201, the add-in program determines whether the information obtained in S6004 includes both object information and area information. If it determines that it includes both, it proceeds to S1203; otherwise, it proceeds to S1202.

[0075] In S1202, the add-in program determines whether the information obtained in S6004 consists only of object information or only of area information. If it determines that the information is object information, it proceeds to S1204; otherwise, it proceeds to S1205.

[0076] In S1203, the add-in program generates a prompt to create a second object based on the object information of the first object obtained in S6004 and the instructions given in the operation request obtained in S6009, with a size corresponding to the area information obtained in S6004. For example, if the object information of the "text box with object ID=2" and the area information of area 2212 are obtained in S6004, and the operation request instruction "make it tabular" is obtained in S6009, the program generates a prompt that reads, "Please change the text contained in the text box with object ID=2 to a tabular format so that it fits within the size of area 2212."

[0077] In S1204, the add-in program generates a prompt to create a second object based on the object information of the first object obtained in S6004 and the instructions given in the operation request obtained in S6009. For example, if the object information of the "text box with object ID=2" in Figure 5 is obtained in S6004, and the operation request instruction "make it a bulleted list" is obtained in S6009, the program generates a prompt that says, "Please change the text contained in the text box with object ID=2 to a bulleted list."

[0078] In S1205, the add-in program generates a prompt to create a second object based on the instructions given in the operation request obtained in S6009, with a size corresponding to the area information obtained in S6004. For example, if the area information for area 2212 in Figure 5 is obtained in S6004, and the operation request instruction "Create a penguin illustration" is obtained in S6009, the program will generate a prompt that says, "Create a penguin illustration with a size that fits within [the size of area 2212]."

[0079] When generating prompts in S1203-S1205, the operation request is analyzed, and the prompt is generated by combining the analysis results with the acquired first object information and area information. Operation request analysis involves, for example, preparing several candidate prompt formats for an example operation request, and comparing their similarity to the operation request actually entered by the user to determine which format should be applied. The method of operation request analysis is not limited to this; for example, natural language analysis may be performed using an external language analysis device. Furthermore, when analyzing the operation request, a determination may also be made as to whether an external file reference is necessary. If it is determined that an external file reference is necessary, the process in S6011 is performed.

[0080] <Example of object information> Referring to Figure 13, an example of object information acquired by the add-in program 3000 in the first object information acquisition process 3120 according to this embodiment will be described. For example, as in the object operation screen 2200 in Figure 5, when three objects, "Title 1," "Text Box 2," and "Figure 3," are displayed, the object information definition file 7000 contains information from 7100 to 7500. This object information definition file is data managed by the object information management function 2300, and refers to a file managed in units such as a list of objects displayed on a single slide in a presentation system. However, the object information does not need to be completed in a single object information definition file 7000; for example, it may refer to other files in the form of links or paths. Note that the description in Figure 13 has been partially modified for illustrative purposes and does not conform to the format of a specific object information definition file that actually exists.

[0081] First, description 7100 is the format information for the object information definition file. This format information includes, for example, information such as the version of the Office Open XML format or the Illustrator format. The second object generation function 4300 in the generation AI server 4000 is configured to generate a second object in a format that can be inserted into a file of such a format.

[0082] In this example <p:csld>The tag-separated range 7200 indicates a description of object information. There is one range 7200 within the object information definition file 7000, and within the range of this tag, the object information for each object managed by this object information definition file 7000 is described. Subsequently, ranges 7300 to 7500 describe the object information for each object described in the object information definition file 7000.

[0083] First, in this example... <p:sp>The tagged range 7300 shows the simplest example of object information. <p:objpr>The tagged range 7310 describes object information about the target object. This object information includes parameters such as an ID to uniquely distinguish the object, a type indicating the object's type, and a unique name assigned to the object. While the description method for range 7310 is not limited to this example, it must include at least information that uniquely identifies the object in order for the generating AI server 4000 to determine which is the first object selected by the user. Furthermore, <a:xfrm>The tagged range 7311 contains information indicating the area where the object is displayed on the object manipulation screen 2200. For example, in this example, it shows the x / y coordinates of the reference point for a rectangular object called "title". <a:off>The tag and the x / y lengths of this object are shown. <a:ext>It is managed with the tag. Note that the content of range 7311 varies depending on the type of object, and is not necessarily <a:xfrm>It is not managed by tags.

[0084] In this example <p:sp>The tagged range 7400 shows an example where the object contains text. First, <p:objpr>The range 7410, separated by tags, contains information about the target object, similar to range 7310. And, <p:textbody>The text contained within the target object is described in the range 7420, separated by tags. In this example, <a:p>The text contains two different pieces of text separated by tags: range 7421 and range 7424. Each piece of text contains: <a:textpr>The text information described in the ranges 7422 and 7425 separated by tags, <a:textcnt>The text content is contained within the ranges 7423 and 7426, which are separated by a bracket. In this example, the text information includes parameters such as "lang" indicating language information, "size" indicating font size, "bold" indicating font expansion settings, "color" indicating font color, and "indent" indicating font indentation. However, the text information may not always contain unique information such as an ID to identify each piece of text. Therefore, as shown in Figure 6, when specifying an arbitrary range of text contained within an object instead of the object itself, information to uniquely identify the first piece of text that acts as a substitute for an ID may be used, for example, in the format "from the Ath character to the Bth character in the object with id=2".

[0085] In this example <p:pic>The tag-separated range 7500 is an example of specifying an external file that exists outside of the object information definition file 7000, such as an image or video. In this example, <p:picpr>The tag-separated range 7510 contains parameters "cnt_type" which indicates the type of external file, and "pass" which indicates the file path to the external file.

[0086] The above is merely an example, and the description of object information is not limited to the example in Figure 13. There are no formatting restrictions other than the requirement that information that uniquely identifies the object, such as an ID, be obtainable.

[0087] <Example of a prompt> Referring to Figure 14, an example of a prompt generated by the add-in program 3000 in the prompt generation process 3220 according to this embodiment will be described. For example, prompt 8000 contains information from 8100 to 8500. In this example, prompt 8000 is in Markdown format to improve the recognition accuracy of the generating AI, but it does not necessarily have to be written in this format.

[0088] First, 8100 describes the role that the generating AI should play. For example, 8100 describes the processing policies, output methods, and prohibited actions of the generating AI. Using delimiters like [] to explicitly specify locations within the prompt is effective in improving the generating AI's accuracy.

[0089] 8200 contains the operation request entered by the user on the AI ​​assistant operation screen 2410. While 8200 may contain the user's operation request as is, actions such as summarizing the user's request using the generation AI or extracting important words are also effective in improving generation accuracy. This operation can be performed by either the add-in program 3000 or the generation AI server 4000.

[0090] Then, lines 8300 to 8500 describe the object information obtained in the first object information acquisition process 3120. First, line 8300 describes information that uniquely identifies the first object. This includes the object's ID or a phrase such as "the Ath to Bth characters in the object with ID=2". Line 8400 describes the file format information that defines the object information obtained in the first object information acquisition process 3120. Then, line 8500 describes the entire contents of the file containing the object information, for example, as explained in Figure 13. Note that Figure 14 shows an example where the entire contents of the file to be processed are described, but this is not the only option; you may also describe only the parts related to the selected object.

[0091] The above is merely an example; prompt 8000 only needs to describe a prompt that is generated based on object information and an operation request for at least the first object, including information that uniquely identifies the first object selected by the user.

[0092] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be realized by combining circuits (for example, ASICs or processors dedicated to image processing) that implement some or all of the functions of the above-described embodiments.

[0093] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.< / p:picpr> < / p:pic> < / a:textcnt> < / a:textpr> < / a:p> < / p:textbody> < / p:objpr> < / p:sp> < / a:xfrm> < / a:ext> < / a:off> < / a:xfrm> < / p:objpr> < / p:sp> < / p:csld>

Claims

1. An add-in program that is added to an application, A first acquisition means for acquiring at least one of the object information of a first object selected by the user on the operation screen of the application, and the area information of an area selected by the user on the operation screen of the application. A second acquisition means for acquiring an operation request entered by the user, A prompt generation means that generates a prompt for generating a second object based on the information obtained by the first acquisition means and the operation request obtained by the second acquisition means. A transmission means that sends the generated prompt to the server that generates the second object. An add-in program that enables a computer to function.

2. The add-in program according to claim 1, characterized in that, if the information acquired by the first acquisition means includes both the object information of the first object and the area information, the prompt generating means generates a prompt instructing the creation of the second object by performing the processing specified in the operation request on the object information of the first object based on the size specified in the area information.

3. The add-in program according to claim 1, characterized in that the prompt generation means generates a prompt instructing the generation of the second object by executing the process specified in the operation request on the object information of the first object if the information acquired by the first acquisition means includes object information of the first object but does not include area information.

4. The add-in program according to claim 1, characterized in that the prompt generation means generates a prompt instructing the generation of the second object by executing the process specified in the operation request based on the size specified in the area information, if the information acquired by the first acquisition means does not include the object information but includes the area information.

5. The add-in program further controls the computer, Receiving means that receives the second object generated in the server based on the prompt from the server, A first output means that outputs the received second object to the application. The add-in program according to claim 1, characterized in that it functions as such.

6. The add-in program according to claim 5, characterized in that, if the information acquired by the first acquisition means includes the area information, and the prompt generated by the prompt generation means is a prompt that instructs the generation of the second object based on the size specified by the area information, the first output means controls to output the second object to the area specified by the area information on the operation screen of the application.

7. The add-in program according to claim 5, characterized in that, if the information acquired by the first acquisition means does not include the area information, the first output means controls to output the second object to a predetermined position on the operation screen of the application or to the AI ​​assistant operation screen.

8. The add-in program further controls the computer, A confirmation message generation means generates a message indicating the next operation the user should perform, based on the information already acquired by the first acquisition means. A second output means that controls the output of the generated message to the AI ​​assistant operation screen of the application. To make it function as, The messages generated by the confirmation message generation means are different depending on whether the information acquired by the first acquisition means includes both the object information and the area information of the first object, includes the object information of the first object but does not include the area information, or does not include the object information of the first object but includes the area information. The add-in program according to feature 1.

9. An information processing device that executes an add-in program that is added to an application, wherein the information processing device is A first acquisition means for acquiring at least one of the object information of a first object selected by the user on the operation screen of the application, and the area information of an area selected by the user on the operation screen of the application, A second acquisition means for acquiring an operation request entered by the user, A prompt generation means that generates a prompt for generating a second object based on the information obtained by the first acquisition means and the operation request obtained by the second acquisition means, A transmission means for sending the generated prompt to the server that generates the second object, An information processing device characterized by having the following features.

10. A method for controlling an information processing device that executes an add-in program that is added to an application, A first acquisition step of acquiring at least one of the object information of a first object selected by the user on the operation screen of the application, and the area information of an area selected by the user on the operation screen of the application, A second acquisition step of obtaining an operation request entered by the user, A prompt generation step that generates a prompt for generating a second object based on the information obtained in the first acquisition step and the operation request obtained in the second acquisition step, A transmission step of sending the generated prompt to the server that generates the second object, A control method characterized by including

Images