Information processing device and program

Abstract

This application aims to facilitate the representation of the data to be edited as a string of natural language characters, thereby facilitating the generation of AI to perform data editing. [Solution] The information processing device in this application comprises: a receiving means for receiving a selection of an object contained in data and a natural language string from a user; a identifying means for identifying a natural language string corresponding to the object once the object has been selected; and an input means for inputting the data, the identified string, and the received natural language string to a generating AI, wherein, upon input of the identified string and the received natural language string to the generating AI, the generating AI performs processing on the selected object based on the received natural language string.

Description

Technical Field

[0004] ,

[0006] , , , , , ,

[0005] , , ,

[0003] , , , , , ,

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] Conversational AIs such as chatbots and generative artificial intelligence (Generative AI) have been developed, and various services using AI are provided. Patent Document 1 discloses a system in which an image of a car is preview-displayed on a display, and an edited preview image of the car is displayed by inputting a natural language instruction (prompt) for changing the body color in a chat to a generative AI.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When instructing a generative AI to edit data, the user needs to express an object to be edited included in the data, such as an image or a text box, as a natural language string and instruct the edit. Therefore, for example, a user who cannot express the object to be edited as a natural language string has difficulty instructing data editing.

[0005] The present application has been made in view of the above problems, and an object thereof is to facilitate instructing a generative AI to edit data by expressing an object to be edited as a natural language string.

Means for Solving the Problems

[0006] In view of the above issues, the information processing device in this application comprises: a receiving means for receiving a selection of an object included in the data and a natural language string from a user; a identifying means for identifying a natural language string corresponding to the object once the object has been selected; and an input means for inputting the data, the identified string, and the received natural language string to a generating AI, wherein, upon input of the identified string and the received natural language string to the generating AI, the generating AI performs processing on the selected object based on the received natural language string. [Effects of the Invention]

[0007] According to the present invention, it becomes easy to represent the data to be edited as a string of natural language characters and instruct the generating AI to edit the data. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows an example of the overall configuration of this system. [Figure 2] This diagram shows an example of the computer hardware configuration of this system. [Figure 3] This diagram shows an example of the hardware configuration of the AI ​​generation server for this system. [Figure 4] This diagram shows an example of the printer hardware configuration in this system. [Figure 5] This diagram shows an example of the software configuration of this system. [Figure 6] This diagram shows an example of the print application screen in this system. [Figure 7] This diagram shows an example of the print application screen when an image is selected in the preview area of ​​the print application screen. [Figure 8] This diagram shows an example of the print application screen after selecting an image in the preview area of ​​the print application and executing the conversion process. [Figure 9]This diagram shows an example of the print application screen when an area is manually selected in the preview area of ​​the print application screen. [Figure 10] This diagram shows an example of the print application screen when a region is manually selected in the preview area of ​​the print application screen and the conversion process is executed. [Figure 11] This sequence diagram shows an example of the process of converting and printing images via chat in this system. [Figure 12] A flowchart illustrating an example of object recognition processing in this system. [Figure 13] This figure shows an example of the results of object recognition processing in this system. [Figure 14] This diagram shows an example of the structure of historical data in this system. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] <Embodiment> <System Configuration> The following describes a first embodiment of the present invention. First, the network configuration of the printing system according to this embodiment will be described with reference to Figure 1.

[0011] As shown in FIG. 1, this printing system includes a computer 1000 which is a terminal device, a printer 2000 configured to be printable, and a generation AI server 3000. The computer 1000 and the printer 2000 are arranged, for example, within an office and are communicably connected to each other via a network 4000. The network 4000 is connected to an external Internet 5000 via a router (not shown). Thereby, the generation AI server 3000 connected to the Internet 5000, the computer 1000, and the printer 2000 are communicably connected to each other.

[0012] Here, the computer 1000 is an example of an information processing device, a user terminal, and a terminal device, the printer 2000 is an example of an image processing device, an image forming device, and an MFP. Note that MFP is an abbreviation for Multi Function Peripheral. The generation AI server 3000 provides a generation AI service 3100. Also, a printing application 1100 described later is executed and provided by the computer 1000 or the printer 2000.

[0013] <Hardware Configuration> Referring to FIGS. 2, 3, and 4, an example of the hardware configuration of each device constituting the printing system according to this embodiment will be described. FIG. 2 shows an example of the hardware configuration of the computer 1000. FIG. 3 shows an example of the hardware configuration of the generation AI server 3000. FIG. 4 shows an example of the hardware configuration of the printer 2000.

[0014] As shown in FIG. 2, the computer 1000 includes a CPU 111, a ROM 112, a RAM 113, an HDD 114, and a network I / F 115. Note that CPU is an abbreviation for Central Processing Unit, ROM is an abbreviation for Read Only Memory, RAM is an abbreviation for Random Access Memory, HDD is an abbreviation for Hard Disk Drive, and I / F is an abbreviation for interface.

[0015] The CPU 111 controls the overall operation by reading control programs stored in the ROM 112 and HDD 114 and executing various processes. The RAM 113 is used as the CPU 111's main memory and temporary storage area such as the work area. The HDD 114 is a large-capacity storage unit that stores image data and various programs. The network I / F 115 is an interface for connecting to the internet. It receives processing requests from other devices and services via the network I / F 115 and sends and receives various information. In this embodiment, the computer 1000 is described as having an operation unit (display means) that is not shown.

[0016] As shown in Figure 3, the hardware configuration of the generation AI server 3000 is almost the same as that of the computer 1000. Therefore, in this embodiment, the hardware configuration of the generation AI server 3000 will not be described. It is preferable that the generation AI server 3000 has a GPU (Graphics Processing Unit) 316, which is a processing unit composed of an image processing processor.

[0017] As shown in Figure 4, the printer 2000 is composed of a control unit 210, an operation unit 220, a printer unit 221, a scanner unit 222, and an authentication device 223. The control unit 210 is composed of the following units 211 to 219 and controls the operation of the entire printer 2000. The CPU 211 reads the control program stored in the ROM 212 and executes and controls various functions of the printer 2000, such as reading, printing, and communication. The RAM 213 is used as the main memory of the CPU 211, a temporary storage area for the work area, etc.

[0018] In this embodiment, one CPU 211 uses one memory (RAM 213 or HDD 214) to execute each process shown in the flowchart described later, but this is not limited to this. For example, multiple CPUs or multiple RAMs or HDDs may work together to execute each process.

[0019] HDD214 is a large-capacity storage unit that stores image data and various programs. The operation interface (I / F215) connects the operation unit (220) and the control unit (210). The operation unit (220) is equipped with a touch panel and keyboard, and accepts user operation, input, and instructions. The printer interface (I / F216) connects the printer unit (221) and the control unit (210). Image data for printing is transferred from the control unit (210) to the printer unit (221) via the printer interface (I / F216) and printed onto the printing medium.

[0020] The scanner I / F 217 is an interface that connects the scanner unit 222 and the control unit 210. The scanner unit 222 reads a document placed on a document glass (not shown) or an ADF (Auto Document Feeder), generates image data, and inputs it to the control unit 210 via the scanner I / F 217.

[0021] The printer 2000 can print (copy) image data generated by the scanner unit 222 from the printer unit 221, as well as send files or emails. The authentication device I / F 218 is an interface that connects the authentication device 223 and the control unit 210. The authentication device 223 is, for example, a card reader that reads IC cards or a fingerprint authentication device that reads fingerprints, and is used when a user authenticates in order to use the printer.

[0022] The network interface 219 connects the control unit 210 (printer 2000) to the LAN (Local Area Network). The printer 2000 uses the network interface 219 to send image data and information to various services on the Internet and to receive various types of information. The operation unit 220 is an interface consisting of a touch panel and keyboard, which displays information to the user and accepts user input.

[0023] <Software Configuration> Referring to Figure 5, an example of the software configuration of each device included in the printing system according to this embodiment will be described.

[0024] As shown in Figure 5, the computer 1000 has a print application 1100. The print application 1100 is an application that allows users to preview prints, change print settings, and send print jobs to the printer 2000. The print application 1100 may be an application independent of the OS, or it may be an application integrated with the print driver application built into the OS.

[0025] The printing application 1100 includes a request control unit 1101, a chat control unit 1102, a data management unit 1103, and a print job control unit 1104.

[0026] The request control unit 1101 is in a state of readiness to receive requests from the computer 1000, printer 2000, and generation AI server 3000, and in response to the requests, it causes each part of the printing application 1100 to execute processing.

[0027] The chat control unit 1102 sends prompts and print data entered by the user to the generation AI server 3000. The chat control unit 1102 also displays the data received from the generation AI server 3000.

[0028] The data management unit 1103 manages the data used by the printing application 1100. The data managed includes, for example, prompts entered into the chat control unit 1102, unconverted print data sent to the generation AI service 3100, converted print data received from the generation AI service 3100, and information obtained from the print data. Application settings used by the printing application 1100 are also saved and managed.

[0029] The print job control unit 1104 converts the print data generated by the print application 1100 and the generation AI server 3000 into a format that can be printed by the printer 2000, and sends it to the printer 2000 as a print job. Alternatively, the print data may be in a format that can be printed by the printer 2000 without conversion by the print job control unit 1104. The preview control unit 1105 displays the received print data and data received from the generation AI server 3000.

[0030] The printer 2000 has a request control unit 2101, a print control unit 2102, and a chat control unit 2103. The request control unit 2101 is in standby mode, ready to receive requests from the computer 1000, the printer 2000, and the generation AI server 3000.

[0031] The print control unit 2102 executes the printing of the print job received from the computer 1000. Specifically, it prints an image based on the image data contained in the received print job onto a printing medium such as paper. The printer 2000 may print the received print job immediately after receiving it without taking any instructions from the user, or it may hold it until it receives a print instruction from the user and then print it.

[0032] The chat control unit 2103 sends prompts and print data entered by the user to the generation AI server 3000. The chat control unit 2103 also displays the data received from the generation AI server 3000.

[0033] The generation AI server 3000 provides the generation AI service 3100. The generation AI service 3100 includes a request control unit 3101, a chat control unit 3102, a data management unit 3103, and a layout conversion unit 3104. In this embodiment, a prompt is a string of natural language that serves as an instruction for the user to specify a function to be executed by the generation AI function operating on the generation AI server 3000.

[0034] The request control unit 3101 has functions for interpreting and processing generated AI and for changing the layout of print data. The request control unit 3101 is in standby mode to receive requests from the computer 1000, printer 2000, and generated AI server 3000, and in response to the requests, it causes each part of the generated AI service 3100 to execute processing.

[0035] The chat control unit 3102 has a function for interpreting the generated AI. The chat control unit 3102 interprets the received prompts and print data written in natural language, and determines the response and layout conversion process for the print data.

[0036] The layout conversion unit 3104 receives and executes the layout conversion process interpreted by the chat control unit 3102. That is, it executes the conversion process when a natural language string is input. As described later, since the target of the processing is specified for the natural language string, it performs processing based on the input natural language string on the target of the processing.

[0037] Furthermore, the layout conversion unit 3104 performs object recognition processing to recognize objects in the print data. In this embodiment, the chat control unit 3102 and the layout conversion unit 3104 are described as separate processing units, but they may be processed together as a single conversion processing unit.

[0038] The data management unit 3103 stores and manages the history of received prompts, received print data, and converted print data, linking them to information that identifies the user who issued the command. An example of the history data stored here will be described later in Figure 14.

[0039] <Example of a print application screen> Referring to Figure 6, the print application screen 600 displayed on computer 1000 or printer 2000 is described. The print application screen 600 has a preview area 601, a chat area 610, and other controls. The print application screen 600 is an example of a screen provided by the print application 1100.

[0040] The preview area 601 is an area that displays a preview image of the print data that the print application 1100 instructs to print. Objects (images, diagrams, text, etc.) included in the print data displayed in the preview area 601 can be selected using the mouse cursor, etc. When the user selects an object, the name of the selected object is entered into the chat input area 611. In other words, when an object is selected, a natural language string corresponding to that object is identified and displayed in the chat input area 611.

[0041] Furthermore, users can freely select areas. An example of the screen when an area is selected will be shown later. In addition, the selected object will be highlighted and its name will be displayed when the mouse hovers over it. Mouse hover refers to the operation of placing the mouse cursor over an object. Object name display 607 shows an example where the name of the object "Image 1" is displayed when the mouse cursor hovers over the image at the top of the preview area 601. Details of object processing will be shown later.

[0042] The print data preview function allows users to check the printed output before it is produced by Printer2000, and to select objects or areas they want to modify.

[0043] The chat area 610 displays the interaction between the generating AI and the user in a chat format. Specifically, it displays natural language strings (prompts) entered by the user and the generating AI's responses to them. In addition, the preview area 601 and the chat area 610 are updated each time a response is received from the AI ​​server 3000. Note that, for example, if the processing cannot be determined from the user's prompt, the generating AI may not perform any processing on the print data and may instead display inquiries or error messages to the user, in which case it is not necessary for the preview area 601 to be updated.

[0044] As mentioned earlier, the print application screen 600 is displayed on the control panel (not shown) of either the computer 1000 or the printer 2000, but the following explanation will use the case where it is displayed on the control panel of the computer 1000 as an example.

[0045] In this embodiment, first, a preview image of print data such as a file or website is displayed in the preview area 601. Next, the user enters a layout conversion instruction for the print data in natural language strings in the chat input area 611. The preview area 601 then displays a preview image of the data converted by the AI ​​server 3000 based on the layout conversion instruction, and the screen is updated.

[0046] The preview area 601 includes a preview page number display 602, page turning buttons 603, a print header area 604, a preview conversion toggle button 605, and a header / footer setting button 606.

[0047] The preview page number display 602 shows the total number of pages when printing the print data and the preview page number currently displayed in the preview area 601. When the page forward button 603 is pressed, the currently displayed preview page number is either moved back or forward, changing the preview page displayed in the preview area.

[0048] The print header area 604 displays a preview of the header that will be printed during printing. The header can include, for example, the date and time of printing, and the name of the file to be printed, such as the file name or URL. Furthermore, if the setting to print that the document has been edited by the generation AI is enabled, and the print data has been edited by the generation AI service 3100, a string indicating that the document was edited by the generation AI can be included in the header. Note that this string may be previewed and printed not in the print header area, but for example, in the print footer, other margins of the print, or within the print body.

[0049] The preview conversion toggle button 605, when pressed, switches the display of the preview area between the edited preview image and the unedited preview image. It also simultaneously switches the display of the preview page number indicator 602.

[0050] At this time, the print application screen will display whether it is the first preview or the latest preview so that the user can see. Furthermore, instead of the image displayed in the preview area 601 switching when the preview conversion switch button 605 is pressed, a screen displaying the unedited preview and the edited preview side-by-side may be displayed. Alternatively, these configurations may be configured to switch between each other.

[0051] The chat area 610 includes a chat input area 611. The chat input area 611 accepts natural language string input from the user and accepts conversion instructions for print data. The print application 1100 sends the input conversion instructions along with the print data to the generation AI service 3100.

[0052] The printing application 1100 displays the converted string received from the generation AI service 3100 as a chat response string in the chat area 610. In Figure 6, the chat area 610 displays the user's instruction, "Please remove the image in the sidebar," and the response displays a converted result indicating that the image has been successfully deleted. Note that image deletion is just one example of a layout conversion instruction, and is not limited to this. Other layout conversion instructions may include, for example, instructions to change the color of an image (object), resize it, move it, or rotate it. Similar to the image deletion instruction described below, the generation AI executes instructions corresponding to these layout conversion instructions.

[0053] The print application screen 600 includes, as other controls, a print execution button 620, a history display button 630, and a settings screen display button 631. The print execution button 620, when pressed, starts the process of sending a print job to the printer 2000, causing the print application 1100 to print the current print data.

[0054] The history display button 630 is a button that, when pressed, displays the history display screen (not shown) of the printing application 1100, which shows the conversion history of past print data.

[0055] The settings screen display button 631 is a button that, when pressed, displays the settings screen (not shown) of the print application 1100. The settings screen includes settings related to printing, such as a printer selection control for selecting the printer to print to, a copy control for specifying the number of copies, and a color mode control for specifying the print color, and allows users to configure these settings. It may also include settings related to the conversion of print data by the generation AI server.

[0056] For example, this includes enabling / disabling the history management of conversion results, enabling / disabling the process of embedding the instruction strings used in the conversion of print data, and enabling / disabling the process by which the print application 1100 automatically performs the conversion process using the embedded instruction strings.

[0057] The object display slider 632 controls the level of detail of objects that can be selected in the preview area 601. Users can change the granularity of selectable objects by sliding the object display slider 632. The area selection toggle switch 633 switches the method of area selection in the preview area 601 between selecting on an object-by-object basis or allowing selection of any area.

[0058] <Example of a print application screen - Object specification> Refer to Figures 7 and 8 to see an example screen when an object of print data is selected in the preview area 601 of the print application 1100. Figure 7 shows an example of the display when an object named "Image 2" is selected in the preview area 601.

[0059] Object name display 608 shows an example of how the object name is displayed when the mouse cursor hovers over the image in the middle of the preview area 601. Additionally, the object name "Image 2," corresponding to the object selected by the user in the preview area 601, is identified and entered into the chat input area 611. The user can add instructions for "Image 2" using natural language strings via an input device such as a keyboard (not shown).

[0060] Figure 8 shows an example of how the display looks when a user adds an editing instruction to "Image 2" saying "Please delete this." In this way, when a user wants to make changes to print data, they can easily change the target object by selecting the object they want to change and adding the change instructions.

[0061] <Example of print application screen - specifying area> Referring to Figures 9 and 10, an example screen is shown when a user manually selects an area on the print data displayed in the preview area 601 of the print application 1100. Figure 9 shows an example display when the area selection toggle switch 633 is set to area selection mode and the user selects a dotted area in the preview area 601. When an area is selected, coordinates indicating the area, "Coordinates: Top left (100, 300), Bottom right (200, 500)", are entered in the chat input area 611.

[0062] Users can add instructions in natural language for the area "coordinates: top left (100,300), bottom right (200,500)". Figure 10 shows an example of how it looks when a user adds the instruction "I want this removed" to the area indicated by "coordinates: top left (100,300), bottom right (200,500)". In this way, users can specify the area to be changed by specifying the region.

[0063] <Processing flow> Referring to Figure 11, the sequence of layout conversion processing in the generation AI service 3100 of this embodiment will be described. The sequence of processing that takes place between the computer 1000, the printer 2000, and the elements constituting the generation AI service 3100 will be described. The numbers following "S" below indicate the step numbers in the sequence.

[0064] As mentioned above, the printing application 1100 may operate under the control of the CPU 111 of the computer 1000, or under the control of the CPU 211 of the printer 2000. Below, we will describe an example in which the CPU 111 of the computer 1000 controls various parts of the printing application 1100 to realize its functions.

[0065] The CPU 111 controls the request control unit 1101, chat control unit 1102, data management unit 1103, and preview control unit 1105, which constitute the print application 1100, and executes subsequent processing. In addition, the CPU 311 of the generation AI server 3000 controls the request control unit 3101, chat control unit 3102, data management unit 3103, and layout conversion unit 3104, which constitute the generation AI service 3100, and executes subsequent processing.

[0066] The request control unit 1101 of the print application 1100 detects a print request from the computer 1000 (S101). Note that this is just an example, and printing is not mandatory, but in this embodiment, the data is described on the premise that it will be printed and will be referred to as print data. The print request includes the print data to be printed. Note that the print data may be sent from the printer 2000 instead of the computer 1000.

[0067] The request control unit 1101 sends an object recognition request to the request control unit 3101 of the generation AI service 3100 (S102). This object recognition request includes the print data to be printed.

[0068] The request control unit 3101 sends an object recognition request to the layout conversion unit 3104 (S103). The object recognition request includes the print data to be printed.

[0069] The layout conversion unit 3104 performs object recognition processing on the received print data (S104) and sends the acquired object information to the request control unit 3101 (S105).

[0070] Details of the object recognition process will be described later. This process provides object information for objects (images, diagrams, text, strings, etc.) within the print data. The object information includes the object type, a name that uniquely identifies the object, and object area information such as coordinates.

[0071] The request control unit 3101 sends object information to the request control unit 1101 of the print application 1100 (S106).

[0072] The request control unit 1101 transmits the received print data to the preview control unit 1105, causing the preview image to be displayed on an unillustrated control panel (S107). As a result, the print data is previewed in the preview area 601 of the print application screen 600. Each object included in the previewed image is associated with the object information acquired in S106.

[0073] When an object is selected by the user in the preview area 601, the preview control unit 1105 sends a request to the chat control unit 1102 of the print application 1100 for input of the object name corresponding to the selected object (S108). In other words, the preview control unit 1105 accepts the selection of objects to be included in the data from the user. The request for input of the object name includes the object name corresponding to the selected object.

[0074] If a region is selected by the user, an input request containing the coordinates corresponding to the selected region is sent. This causes the object's name or coordinates to be displayed in the chat input area 611.

[0075] The chat control unit 1102 inputs the object name or coordinates received from the preview control unit 1105 in S108 into the chat input area 611 (S109). In other words, when the user selects an object, the object selection is accepted, a natural language string corresponding to the object is identified, and this is displayed in the chat input area 611.

[0076] The chat control unit 1102 detects that a conversion request has been entered by the user (S110). Specifically, it detects that in addition to the name or coordinates of the object displayed in the chat input area 611 in S108, the user has entered a natural language string into the chat input area 611. In other words, the chat control unit 1102 receives a natural language string from the user.

[0077] The chat control unit 1102 sends a conversion request to the request control unit 1101 when the execution button included in the chat input area 611 is pressed (S111). The conversion request is an instruction to the generating AI consisting of a natural language string corresponding to an object or area selected by the user and a natural language string received by being entered into the chat input area 611. In this embodiment, an example including print data is also described as necessary. The execution button may be a send button or a post button.

[0078] The request control unit 1101 sends a conversion request to the request control unit 3101 of the generation AI service 3100 (S112). Specifically, it sends the natural language string corresponding to the selected object or area, the natural language string received by the user as input into the chat input area 611, and the print data to the generation AI service 3100 of the generation AI server 3000.

[0079] The request control unit 3101 sends a conversion request to the chat control unit 3102 (S113).

[0080] The chat control unit 3102 interprets the received conversion request and print data written in natural language, and determines the layout conversion process for the print data (S114).

[0081] The chat control unit 3102 transmits the layout conversion processing request determined in S114 to the layout conversion unit 3104 (S115).

[0082] The layout conversion unit 3104 executes the layout conversion process included in the layout conversion request on the print data included in the layout conversion request (S116). At this time, the layout conversion is performed using the object information obtained in S104. Specifically, in accordance with the change request, the specified processing is executed on the object or area in the received print data that corresponds to the change request.

[0083] The layout conversion unit 3104 performs object recognition on the conversion result (S117). The method of object recognition may differ from that used in S104. For example, if the print data is a web page, the method may be changed to suit the form of the print data, such as performing object recognition using a generation AI.

[0084] The layout conversion unit 3104 returns the conversion result to the chat control unit 3102 (S118). At this time, the conversion result includes the converted print data and object information.

[0085] The chat control unit 3102 saves the information before and after conversion as history data to the data management unit 3103 (S119). An example of the saved history data is shown in Figure 14.

[0086] The chat control unit 3102 generates conversion result text indicating the success or failure of the conversion from the conversion result received in S118 (S120).

[0087] The chat control unit 3102 returns the conversion request result to the request control unit 3101 (S121). The request control unit 3101 returns the conversion request result to the request control unit 1101 of the print application 1100 (S122). The conversion request result includes the converted print data and its object information returned by the layout conversion unit 3104 in S118, and the converted result text generated in S120.

[0088] The request control unit 1101 transmits the print data included in the received conversion request result to the preview control unit 1105, causing the preview image to be displayed on the operation unit (not shown) (S123). As a result, the print data including the objects that have undergone conversion (editing) processing is displayed as a preview in the preview area 601 of the print application screen 600. Each object included in the previewed image contains the object information acquired in S117.

[0089] The request control unit 1101 sends the converted text included in the received conversion request result to the chat control unit 1102, which then displays it in the chat area 610 (S124).

[0090] The request control unit 1101 saves the received conversion result to the data management unit 1103 of the print application 1100 (S125). Subsequently, by repeating steps S108 to S125 any number of times, print data with the user's desired layout can be generated. Note that repeating these processes is not mandatory, and the process may be completed in a single step.

[0091] In this embodiment, the print data is described as being sent each time, but it may be configured not to send it each time. In that case, the print data sent in S102 may be stored in the data management unit 3103 of the generation AI service 3100, and from the second time onward, the conversion process (S116) may be performed using the latest converted print data stored in the data management unit 3103. This reduces the amount of communication and alleviates the economic burden on users of the generation AI service, which is normally charged according to the amount of data transmitted.

[0092] Once the AI ​​generation process produces the print data that the user expects, the user inputs a print execution command. The request control unit 1101 detects the user's print command and sends the print execution command and print data to the request control unit 2101 of the printer 2000 (S126).

[0093] Under the control of the CPU 211 of the printer 2000, the request control unit 2101 executes printing using the print control unit 2102 (S127). This allows the print data processed by the generation AI to be printed.

[0094] The request control unit 1101 of the print application 1100 discards the conversion results that are no longer needed from the data management unit 1103 (S128). Note that this process is not mandatory.

[0095] By performing the above process, users can easily instruct the generating AI to edit data by selecting objects in the previewed image and representing the target for editing in natural language strings.

[0096] <Object Recognition Process> The flow and results of the object recognition process for print data will be explained with reference to Figures 12 and 13. The numbers following "S" below indicate the step numbers in the flowchart. This process is executed by the CPU 311 of the generation AI server 3000, which controls each part, and is initiated when print data is sent to the generation AI server 3000.

[0097] First, in S201, the CPU 311 reads the print data received from the request control unit 3101. The print data can take various forms, such as image data or web pages, but here we will show an example where image data is received and read. Image 1301 in Figure 13 shows an example of a read image.

[0098] Next, in S202, the CPU 311 performs region identification (object recognition) on the loaded image 1301. Region identification is performed, for example, by image region separation processing described in Japanese Patent Publication No. 2011-76575 or region identification processing described in Japanese Patent Publication No. 2003-30584. This provides the type and coordinate information of each region in the image 1301. Figure 13 shows the identification result 1302, which is the result of the region identification processing.

[0099] Next, in S203, the CPU311 assigns a unique object name to each region information obtained in S202. This makes it possible to identify each region by its object name.

[0100] Table 1303 in Figure 13 shows an example of object information obtained through the above series of processes. Alternatively, character recognition processing may be performed on the area of ​​object type "text" to obtain information for each character contained within the area and include it in the object information. Table 1304 in Figure 13 shows an example of information for each character contained in the object named "text1".

[0101] The above process illustrates a flow for image data, but if the print data is a web page, similar information can be obtained, for example, by the structured document extraction method described in Japanese Patent Publication No. 2014-81945.

[0102] Alternatively, as another means of obtaining the above object information, one may input the print data and processing details (performing region identification and outputting the type and coordinate information of each region) into the generating AI, regardless of the format of the print data, and obtain the above information.

[0103] <Historical Data Structure> Referring to Figure 14, the structure of the historical data stored in the data management unit 3103 of the generation AI service 3100 is shown. The historical data DB 300 includes the following items of historical data: prompt 301, pre-conversion data 302, post-conversion data 303, pre-conversion object data 304, post-conversion object data 305, job identifier 306, and date and time 307.

[0104] Each row in the history data DB300 represents one piece of history data. Prompt 301 is a prompt received by the generation AI service 3100 as a conversion request. Pre-conversion data 302 is the print data before the conversion process is executed by the prompt. Post-conversion data 303 is the print data after the generation AI service 3100 has executed the conversion process by the prompt.

[0105] The pre-conversion object data 304 is the object information obtained when object recognition was performed before the conversion process was executed. The post-conversion object data 305 is the object information obtained when object recognition was performed after the conversion process was executed.

[0106] The pre- and post-conversion data may be stored in external storage rather than directly in the history data DB300, and the history data DB300 may store information to identify the files in the external storage, such as a URL. The job identifier 306 stores, for example, the file name or URL of the file to be printed. The date and time 307 is the date and time when the conversion was performed by the prompt.

[0107] The contents stored in the history data DB300 may be deleted once the target print data has been printed. Alternatively, they may remain even after printing without being deleted.

[0108] As described above, the printing system in this application performs object recognition on print data and makes objects selectable in the preview area. When an object is selected, an object name that uniquely identifies the object is automatically entered into the chat input area, and user instructions can also be entered, allowing the user to modify the print data with appropriate natural language chat instructions and enable printing.

[0109] In this embodiment, the user can also specify multiple objects or regions. Furthermore, if the user obtains information to identify an object, such as the name of an object, through actions like mouseover, the user may input that identification information as a natural language string without selecting the object and issue a conversion instruction to the generation AI service 3100.

[0110] In this embodiment, Figure 8 shows an example where "Image 2" is generated when the user selects an object, and the user inputs the instruction "Please erase" as a natural language string. This demonstrates an example where the generation AI service 3100 performs processing on the object corresponding to Image 2 based on the instruction "Please erase".

[0111] However, if, for example, the user wants to perform this process on image 1 in addition to image 2, the user selects an object to select "image 2," and the user inputs a natural language string instruction saying "delete image 1 and image 2." In this case, as shown in S114, the generation AI service 3100 interprets the prompt and performs processing based on the instruction "delete image 1 and image 2" on the objects corresponding to image 2 and image 1.

[0112] In this embodiment, an example of sending print data from the computer 1000 to the generation AI server 3000 has been described, but the system is not limited to this. For example, the printer 2000 may acquire image data by scanning a document, and this data may then be sent from the printer 2000 to the generation AI server 3000.

[0113] In this case, both the printer 2000 and the computer 1000 log in to the generation AI service 3100 account using the same authentication information. Next, the image data acquired by the printer 2000 is sent to the generation AI server 3000 along with the account information. Subsequently, by logging into the generation AI service 3100 from the computer 1000 with the same account information, a preview image of the image data can be viewed on the computer 1000. As a result, the image data acquired by the printer 2000 is viewed on the computer 1000, and by executing the above process, the user can obtain the desired print data.

[0114] Furthermore, if the printing application 1100 is operated by the CPU 211 of the printer 2000, it is possible to implement this system using only the printer 2000 and the generation AI server 3000, without using the computer 1000.

[0115] In this embodiment, an example has been described in which object recognition is performed by the generation AI server 3000, but the configuration is not limited to this. Alternatively, object recognition may be performed by the computer 1000 or printer 2000 that sends print data to the generation AI server 3000, and the object information may be sent to the generation AI server 3000 along with the print data.

[0116] In this application, "generative AI" refers to a technology that automatically generates various types of content, such as text, images, music, and videos, similar to those created by humans, by utilizing deep learning and machine learning techniques.

[0117] In this embodiment, the printing system may have servers such as the generation AI server 3000 located outside of Japan, while the terminal device, the computer 1000 (hereinafter referred to as "terminal device"), is located within Japan. Even in this case, files and data can be transmitted from each server to the terminal device, and the terminal device can receive these files and data.

[0118] Even if each server is located outside of Japan, the transmission and reception (sending and receiving) of files and data within this system are performed as a single integrated process. Furthermore, since the system functions by receiving these files and data on terminal devices located within Japan, it can be considered that such transmission and reception took place domestically.

[0119] In this system, even if, for example, the server is located outside of Japan and the terminal device is located within Japan, the terminal device can perform the main functions of the system, and the effects of those functions can be realized within Japan. For example, even if the server is located outside of Japan, if the terminal device constituting this system is located within Japan, it is possible to use the system within Japan using that terminal device. Furthermore, the use of such a system may affect the economic interests of, for example, the patent holder.

[0120] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

Claims

1. A means of receiving the selection of objects included in the data and natural language strings from the user, A means for identifying a natural language string corresponding to the object, upon selection of the object, The system includes input means for inputting the aforementioned data, the identified string, and the received natural language string into a generating AI. When the identified string and the received natural language string are input to the generating AI, the generating AI performs processing on the selected object based on the received natural language string. An information processing device characterized by the following:

2. It further has a means of display, The information processing apparatus according to claim 1, characterized in that the display means displays an object on which the processing has been performed by the generating AI.

3. It further has a receiving means, The receiving means receives a natural language string obtained by the input means inputting the data to the generating AI, which is a natural language string corresponding to the object contained in the data. The information processing apparatus according to claim 1, characterized in that the identifying means identifies a natural language string corresponding to the object contained in the data based on the received natural language string.

4. It further has recognition means, The recognition means recognizes the object contained in the data and associates the natural language string with the recognized object. The information processing apparatus according to claim 1, characterized in that the identifying means identifies a natural language string corresponding to the object contained in the data based on the natural language string associated by the recognition means.

5. The information processing apparatus according to claim 1, characterized in that the object is an image and the data is image data including the image.

6. The information processing apparatus according to claim 1, characterized in that the object is a string and the data is image data including the string.

7. The information processing device according to claim 1, characterized in that the natural language string corresponding to the object is the name of the object.

8. A means for receiving data from the user, including the selection of a data area and a natural language string, A means for identifying a natural language string corresponding to the selected region, The system includes input means for inputting the aforementioned data, the identified string, and the received natural language string into a generating AI. When the identified string and the received natural language string are input to the generating AI, the generating AI performs processing based on the received natural language string in the selected region. An information processing device characterized by the following:

9. It further has a means of display, The information processing apparatus according to claim 8, characterized in that the display means displays objects included in the region where the processing was performed by the generating AI.

10. It further has a receiving means, The receiving means receives a natural language string obtained by inputting the data and the selected region into the generating AI, the natural language string corresponding to the selected region included in the data. The information processing apparatus according to claim 8, characterized in that the identifying means identifies a natural language string corresponding to the selected region contained in the data based on the received natural language string.

11. It further has recognition means, The recognition means recognizes the region selected by the user and associates the natural language string with the recognized region. The information processing apparatus according to claim 8, characterized in that the identifying means identifies a natural language string corresponding to the selected region included in the data based on the natural language string associated by the recognition means.

12. The information processing apparatus according to claim 8, characterized in that the region is a region in the image indicated by the data.

13. A process for receiving data, including the selection of objects included in the data and natural language strings from the user, A program that causes a computer to perform the following steps: an identification step of identifying a natural language string corresponding to an object upon selection of the object; and an input step of inputting the data, the identified string, and the received natural language string to a generating AI, When the identified string and the received natural language string are input to the generating AI, the generating AI performs processing on the selected object based on the received natural language string. A program characterized by the following features.

14. The process further includes a display step, The program according to claim 13, wherein the display step displays the object on which the processing was performed by the generating AI.

15. It further includes a receiving process, The receiving step receives a natural language string obtained by inputting the data into the generating AI in the input step, which is a natural language string corresponding to the object contained in the data. The program according to claim 13, wherein the specified step identifies a natural language string corresponding to the object contained in the data based on the received natural language string.

16. It further includes a recognition process, The recognition step recognizes the object contained in the data and associates the natural language string with the recognized object. The program according to claim 13, wherein the identification step identifies a natural language string corresponding to the object included in the data based on the natural language string associated by the recognition step.

17. The program according to claim 13, characterized in that the object is an image and the data is image data including the image.

18. The program according to claim 13, characterized in that the object is a string and the data is image data including the string.

19. The program according to claim 13, characterized in that the natural language string corresponding to the object is the name of the object.

Images