Information processing device, information processing method, and program

Abstract

This invention has been made in view of the above problems, and aims to enable the generation AI to provide instructions that are in line with the user's intent by transforming instruction text or images so that the generation AI can identify the indicated location in the image. [Problem-solving] The program of the present invention is characterized by causing an information processing device to execute an acquisition step of acquiring a document image containing multiple handwritten areas indicating regions and an instruction text entered by a user; an identification step of the multiple handwritten areas to be processed by a generation AI; a conversion step of converting the instruction text acquired in the acquisition step into an instruction text that the generation AI can identify the instruction areas in; and an output step of outputting the instruction text converted in the conversion step and the document image to the generation AI.

Description

Technical Field

[0001] The present invention relates to a technique for giving instructions to generative AI.

Background Art

[0002] In recent years, generative AI that can automatically generate creative content such as images, text, and voice has been rapidly spreading. Along with this, various services using generative AI are being provided. For example, services for reverse engineering legacy programs and services that search the Internet for answers to inquiries in natural language are known.

[0003] Furthermore, in generative AI, not only text data but also multimodal information such as image information can be taken into account. Specifically, for example, when a user inputs a photographic image of "a bouquet with a circle (mark) around a single flower" and a question sentence "What kind of flower is this?", the generative AI explains what kind of flower the single flower is.

[0004] Also, scenes where handwritten notes are added to printed documents and the written documents are digitized (scanned) can still be seen in recent years. The written content includes supplementary explanations written by the user for the purpose of facilitating their own understanding, markings on important parts, markings on parts to be investigated / questioned later, doubts, etc., covering a wide range. There are also multiple writing locations, and the way of writing tends to change according to the written content.

[0005] For example, important parts are marked using a yellow fluorescent pen, and parts to be investigated / questioned later are surrounded with a blue ballpoint pen, etc. The way of writing varies among users. Regarding the digitization of this written document, a service that creates a scanned image with only the handwritten parts emphasized / deleted (hereinafter, handwritten part emphasis service) is known. Also, a technique (Patent Document 1) that performs OCR by cutting out only the area marked or surrounded by the user with a marker is known.

[0006] However, handwritten text highlighting services and the technology described in Patent Document 1 only recognize the parts written by the user by hand and could not be used to "make the generating AI recognize the specified parts." Therefore, users had to think of instructions to "identify the parts to be instructed for the generating AI" from "the content written by the user by hand," which placed a heavy burden on the user in terms of instruction input. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Patent No. 7048275 [Overview of the project] [Problems that the invention aims to solve]

[0008] This invention has been made in view of the above problems, and aims to enable the generation AI to provide instructions that are in line with the user's intent by transforming instruction text or images so that the generation AI can identify the indicated location in the image. [Means for solving the problem]

[0009] The program of the present invention is characterized by causing an information processing device to execute an acquisition step of acquiring a document image containing multiple handwritten areas indicating regions and instruction text entered by a user; an identification step of the multiple handwritten areas to be processed by a generation AI; a conversion step of converting the instruction text acquired in the acquisition step into instruction text that the generation AI can identify the instruction areas; and an output step of outputting the instruction text converted in the conversion step and the document image to the generation AI. [Effects of the Invention]

[0010] According to the present invention, users can give instructions to the generating AI that align with their intentions. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram showing an example of the configuration of an information processing system. [Figure 2] This figure shows an example of the hardware configuration of each device that makes up an information processing system. [Figure 3] This is a sequence diagram of an information processing system. [Figure 4] This is a flowchart showing the process from giving instructions to the generating AI to outputting its response in the information processing system of the first to fifth embodiments. [Figure 5] This figure shows specific examples of document images to be input into an information processing system, and responses from the generating AI. [Figure 6] This figure shows examples of screens used by users to operate and confirm information processing systems in embodiments 1 to 5. [Figure 7] This figure shows an example of a screen used by a user to operate and confirm information in the information processing system in the second embodiment. [Figure 8] This is a flowchart for the information processing system in the second embodiment, showing the process from giving instructions to the generating AI to outputting its response. [Figure 9] This figure shows an example of a screen that a user operates on in the information processing system in the third embodiment. [Figure 10] This figure shows an example of a screen that a user operates on in the information processing system in the fourth embodiment. [Figure 11] This is a flowchart showing the process from giving instructions to the generating AI to outputting its response in the information processing system of the fifth embodiment. [Figure 12] This figure shows an example of a screen used by a user to operate and confirm information in the information processing system in the fifth embodiment. [Figure 13] This is a flowchart showing the process from giving instructions to the generating AI to outputting its response in the information processing system of the sixth embodiment. [Figure 14] This figure shows examples of screens used by a user to operate and confirm information processing systems in embodiments 6 to 9. [Figure 15] FIG. is a diagram showing an example of a screen that a user operates and checks in an information processing system according to the seventh embodiment. [Figure 16] FIG. is a flowchart showing the process from an instruction to a generation AI to the output of its answer in the information processing system according to the seventh embodiment. [Figure 17] FIG. is a diagram showing an example of a screen that a user operates in an information processing system according to the eighth embodiment. [Figure 18] FIG. is a diagram showing an example of a screen that a user operates in an information processing system according to the ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components described in the following embodiments are examples and are not intended to limit the scope of the technology of the present disclosure. For example, each part constituting the present invention can be replaced with any configuration that can exhibit the same function. Also, any component may be added.

[0013] In Embodiments 1 to 5, examples of solving the above problems by converting instruction sentences are shown. And in Embodiments 6 to 9, examples of solving the above problems by converting images are shown.

[0014] [Embodiment 1] <Information Processing System> FIG. 1 is a block diagram showing a configuration example of an information processing system 100 of the present invention.

[0015] As shown in FIG. 1, the information processing system 100 includes an information processing device 101, a generation AI server 102, and an information processing server 103. For example, the information processing device 101 and the information processing server 103 are connected to each other via a network 104.

[0016] In this configuration, the information processing device 101 and the information processing server 103 may be configured to have multiple connections to the network 104, rather than a single connection. For example, the information processing server 103 may consist of a first server device having high-speed computing resources and a second server device having large-capacity storage, and these may be connected to each other via the network 104.

[0017] Network 104 is connected to the external internet 105 via a router (not shown). The generation AI server 102 is connected to the information processing device 101 and information processing server 103 via the internet 105 and network 104, enabling communication between them.

[0018] The information processing device 101 can be implemented in, for example, a Multi-Function Peripheral (MFP) with multiple functions such as printing, scanning, and faxing, a personal computer, a smartphone, or a tablet terminal. The information processing device 101 has an image acquisition unit 151, an instruction text acquisition unit 152, and a display unit 158 ​​as its functional units.

[0019] The image acquisition unit 151 optically reads, for example, an original document 111 printed on a storage medium such as paper, generates a document image 113 by performing predetermined scan image processing, and transmits it to the information processing server 103. The image acquisition unit 151 also receives, for example, fax data 112 sent from a fax transmitter (not shown), generates a document image 113 by performing predetermined fax image processing, and transmits it to the information processing server 103.

[0020] Furthermore, the information processing device 101 may be implemented not only as an MFP equipped with the aforementioned scanning and faxing functions, but also as a PC (Personal Computer), etc. Specifically, for example, a document image 113 such as a PDF or JPEG, generated using a document creation application running on the PC acting as the information processing device 101, may be sent to the information processing server 103.

[0021] Furthermore, the information processing device 101 may be a smartphone or tablet device. In this case, images taken with the attached camera may be used.

[0022] The instruction text acquisition unit 152 transmits, for example, the instruction text 114 entered by the user via the display unit 158 ​​(described later) to the information processing server 103. At this time, the instruction text entered by the user may be one that has been prepared in advance by an engineer or user, one that has been modified or added to by the user or system, or one that has been entered directly by the user or system from scratch. The image acquisition unit 151 and the instruction text acquisition unit 152 are examples of acquisition means of the present invention.

[0023] The display unit 158 ​​displays information received from the information processing server 103 on the display of the display device 210 (see Figure 2). The display unit 158 ​​displays, for example, the instruction setting / confirmation screen 600 (see Figure 6), which will be described later. The display unit 158 ​​may also display on the display of display device 267 (see Figure 2) instead of display device 210 (see Figure 2). Furthermore, the display unit 158 ​​may also display on a display unit (not shown) of a PC or mobile terminal connected via the Internet 105 and network 104.

[0024] The generation AI server 102 is a server managed by the provider of the generation AI service. The generation AI server 102 is accessed via an API (Application Programming Interface) and outputs the response results to the instruction text and instruction image received from the information processing server 103.

[0025] In this context, the generation AI server 102 may be a combination of plugins that implement additional functions developed by a service provider utilizing the generation AI service. Furthermore, the generation AI server 102 may exist as a directly connected entity via the network 104, or it may be configured to exist on a separate system from the same vendor as the information processing system 1000.

[0026] Furthermore, the functions of the generation AI server 102 may be configured to reside within the information processing server 103, or some functions or devices may reside within the information processing server 103.

[0027] The information processing server 103 functions as a document image analysis unit 154, an instruction text analysis unit 159, an instruction content generation unit 155, and a storage unit 157. The information processing server 103 takes the document image 113 as input and transmits the results of processing it via the generation AI server 102 to the information processing device 101.

[0028] First, the document image analysis unit 154 performs a process to recognize handwritten areas on the document image 113 received from the information processing device 101, and detects handwritten areas. The document image analysis unit 154 is an example of a specific means of the present invention. Known techniques are used for the method of recognizing handwritten areas. Known techniques include, for example, the technique of classifying a document image into printed areas, handwritten areas, and blank areas using the concept of semantic segmentation.

[0029] A classifier applying known technology may be accessed externally via an API, or it may exist as a learning device (not shown) via network 104. In addition, the document image analysis unit 154 may perform OCR on the document image 113 in addition to detecting handwritten portions. The OCR may target the entire document image 113, or it may target the printed and handwritten areas within the document image 113 separately.

[0030] Next, the instruction analysis unit 159 detects descriptions in the instruction 114 received from the information processing device 101 that indicate which part of the document image 113 the instruction refers to. Specifically, for example, it uses well-known natural language processing techniques to detect words that indicate a part, region, or location within the document image 113.

[0031] Well-known natural language processing techniques include, for example, detecting and identifying demonstrative pronouns such as "here" through coordinating analysis, and extracting specific keywords such as "region" or "location."

[0032] Next, the instruction content generation unit 155 generates an instruction (not shown) that identifies the parts to be instructed to the generating AI, based on the handwritten parts of the document image 113 and the instruction text 114. The instruction text that identifies the parts to be instructed to the generating AI is, for example, text in which the instruction words in the instruction text 114 are replaced with specific notations such as a surrounding line. The instruction content generation unit 155 then finalizes the instruction text that identifies the parts to be instructed to the generating AI and the document image 113 as the instruction content. The instruction content generation unit 155 is an example of the conversion means of the present invention.

[0033] Alternatively, the instruction content generation unit 155 may generate an instruction text (not shown) with the OCR results obtained by the document image analysis unit 154 added, and then finalize this instruction text and the document image 113 as the instruction content.

[0034] Next, the information processing server 103 transmits the instruction content generated and confirmed by the instruction content generation unit 155 to the generation AI server 102. Furthermore, the information processing server 103 receives the response result for the instruction content generated and confirmed by the instruction content generation unit 155 from the generation AI server 102 and stores it in the storage unit 157.

[0035] Network 104 is implemented using a LAN or WAN, and is a communication unit that connects the information processing device 101, the generation AI server 102, and the information processing server 103 to each other for sending and receiving data between the devices. Network 104 may be connected by a wired connection or a wireless connection.

[0036] <Device configuration> Figure 2 shows an example of the hardware configuration of the information processing device 101, the generation AI server 102, and the information processing server 103 included in the information processing system 100.

[0037] Figure 2(a) shows the hardware configuration of the information processing device 101. In this specification, an example in which an MFP (Multi-Function Printer / Peripheral) is used as the information processing device 101 will be described.

[0038] As shown in Figure 2(a), the information processing device 101 includes a CPU 201, ROM 202, RAM 204, printer device 205, scanner device 206, document transport device 207, and storage 208. Furthermore, it includes an input device 209, a display device 210, and an external interface 211. Each part is connected to the others via a data bus 203.

[0039] The CPU 201 is a control unit that controls the overall operation of the information processing device 101. The CPU 201 starts the system of the information processing device 101 by executing a startup program stored in the ROM 202, and then executes a control program stored in the storage 208 to realize the functions of the information processing device 101, such as printing, scanning, and faxing.

[0040] ROM202 is a storage unit implemented with non-volatile memory, and stores a startup program for starting the information processing device 101.

[0041] The data bus 203 is a communication unit for sending and receiving data between devices that make up the information processing device 101.

[0042] RAM204 is a memory unit implemented using volatile memory and is used as work memory when the CPU201 executes control programs.

[0043] The printer device 205 is an image output device that prints and outputs document images onto a storage medium such as paper. The scanner device 206 is an image input device that optically reads a storage medium such as paper on which text, diagrams, etc., are printed. The data obtained by the scanner device 206 is acquired as a document image.

[0044] The document transport device 207 is implemented as an ADF (Automatic Document Feeder), detects documents placed on the document glass, and transports the detected documents one by one to the scanner device 206.

[0045] Storage 208 is a storage unit implemented using an HDD (Hard Disk Drive) or the like, and stores the aforementioned control program and document images.

[0046] The input device 209 is an operation unit implemented as a touch panel or hard keys, which receives operation input from a user using the information processing device 101. The display device 210 is a display unit implemented as a liquid crystal display, which displays the settings screen of the information processing device 101 to the user. The display device 210 displays, for example, the setting / confirmation screen 600 (see Figure 6) of the instruction content, which will be described later, as explained in the display unit 158 ​​(see Figure 1).

[0047] The external interface 211 is an interface that connects the information processing device 101 and the network 104, and transmits document images to the information processing server 103, and transmits document images and instruction texts (prompts) to the generation AI server 102. The external interface 211 is an example of the output means of the present invention.

[0048] Figure 2(b) shows the hardware configuration of the generation AI server 102. As shown in Figure 2(b), the generation AI server 102 has a CPU 231, ROM 232, RAM 234, storage 235, input device 236, display device 237, external interface 238, and GPU 239. The various parts of the generation AI server 102 are connected to each other via a data bus 233.

[0049] The CPU 231 is a control unit that controls the overall operation of the generation AI server 102. The CPU 231 starts up the generation AI server 102 system by executing a startup program stored in the ROM 232 and then executes a control program stored in the storage 235.

[0050] The control program executed here utilizes a Large Language Model (LLM) capable of inputting multimodal data, including at least images and text. This control program then outputs the results of a conversion based on text-based instructions (prompts).

[0051] ROM232 is a storage unit implemented with non-volatile memory, and stores a startup program for starting the generation AI server 102.

[0052] The data bus 233 is a communication unit for sending and receiving data between devices that make up the generation AI server 102.

[0053] RAM234 is a memory unit implemented using volatile memory and is used as work memory when the CPU231 executes control programs.

[0054] Storage 235 is a memory unit implemented as an HDD (Hard Disk Drive) or the like, and stores the aforementioned control program, large-scale language model, document images, instruction statements (prompts), etc.

[0055] The input device 236 is an operation unit implemented by a mouse and keyboard, etc., which receives operation input to the generation AI server 102 from a user utilizing the generation AI server 102.

[0056] The display device 237 is a display unit implemented as a liquid crystal display or the like, which displays the settings screen of the generation AI server 102 to the user using the generation AI server 102.

[0057] The external interface 238 is an interface that connects the generation AI server 102 and the network 104, and receives document images and instruction texts (prompts) from the information processing server 103. The external interface 238 also transmits the output results from the large-scale language model to the information processing server 103.

[0058] The GPU239 is a processing unit composed of image processing processors. For example, the GPU239 performs calculations to transform input image and text data using a large-scale language model, according to control commands given by the CPU231.

[0059] Figure 2(c) shows the hardware configuration of the information processing server 103. As shown in Figure 2(c), the information processing server 103 consists of a CPU 261, ROM 262, RAM 264, storage 265, input device 266, display device 267, and external interface 268, which are connected to each other via a data bus 263.

[0060] The CPU 261 is a control unit that controls the overall operation of the information processing server 103. The CPU 261 starts up the information processing server 103 system by executing a startup program stored in the ROM 202, and then executes a control program stored in the storage 265 to realize various functions such as displaying document images and inputting instructions to the generation AI.

[0061] ROM262 is a storage unit implemented with non-volatile memory and stores the startup program for starting the information processing server 103. Data bus 263 is a communication unit for sending and receiving data between devices that make up the information processing server 103. RAM264 is a storage unit implemented with volatile memory and is used as work memory when the CPU 261 executes the control program.

[0062] Storage 265 is a storage unit implemented using an HDD (Hard Disk Drive) or the like, and stores the aforementioned control program and document images.

[0063] The input device 266 is an operation unit implemented as a mouse or keyboard, and receives operation input to the information processing server 103 from a user using the information processing server 103 or from an engineer controlling the information processing server 103.

[0064] The display device 267 is a display unit implemented using a liquid crystal display or the like. The display device 267 displays the settings screen of the information processing server 103, the input screen of the generation AI server 102, etc., to users of the information processing server 103, or to engineers who control the information processing server 103. For example, as explained in the display unit 158 ​​(see Figure 1), the display device 267 may display the setting / confirmation screen 600 (see Figure 6), which will be described later, instead of the display device 210.

[0065] The external interface 268 is an interface that connects the information processing server 103 and the network 104, and receives document images from the information processing device 101 and sends instruction messages (prompts) to the generation AI server 102.

[0066] <Usage Sequence> Figure 3 shows the sequence of the information processing system 100. The symbol "S" in the description of each process indicates a step in the sequence, and the same applies to subsequent flowcharts. For ease of explanation, user operations will also be described using steps.

[0067] Figure 3 is a sequence diagram illustrating the process from issuing instructions to the generation AI server 102 using the document image 113 and instruction text 114 acquired by the information processing device 101, to outputting the response. A detailed explanation of how the information processing server 103 in Figure 3 handles the process from issuing instructions to the generation AI to outputting the response will be described later using Figure 4.

[0068] In S311, a user of the information processing system 100 places a document, such as paper, on the document transport device 207 of the information processing device 101 and instructs the scanning of the document by pressing the scan execution button using the input device 209.

[0069] In S312, the information processing device 101 transmits the document image 113 obtained by scanning the original document 111 to the information processing server 103.

[0070] In S313, a user of the information processing system 100 inputs an instruction statement 114 to the information processing device 101 using the input device 209 to instruct the generation AI on the document image 113. For example, the text 603 in Figure 6(a) is an example of the instruction statement 114. Here, the instruction statement 114 may be one that has been prepared in advance by an engineer or user, one that has been modified or added to by the user or system, or one that has been directly input by the user or system from scratch.

[0071] In S314, the information processing device 101 transmits the instruction statement 114, which was input by a user of the information processing system 100, to the information processing server 103.

[0072] In S315, the information processing server 103 recognizes handwritten portions from the document image 113 received in S312. Handwritten portions refer to all handwritten text within the original document 111. Subsequently, the information processing server 103 extracts handwritten portions that represent specific areas from the recognized handwritten portions. This will be explained using Figures 5(a) and 5(b).

[0073] Figure 5(a) is a specific example of document image 113, and Figure 5(b) is an excerpt of only the handwritten portions from document image 500 shown in Figure 5(a).

[0074] The handwritten areas refer to the enclosed line 510 surrounding text 501, the marker area 512, and the marker area 513. Furthermore, the handwritten depictions representing areas also refer to the enclosed line 510, the marker area 512, and the marker area 513. In other words, the handwritten depictions representing areas are those where the start and end points are clearly defined.

[0075] In addition to areas enclosed by lines and marked with markers, handwritten areas indicating regions may also include areas enclosed in parentheses, underlined areas, and so on.

[0076] Although not covered in this embodiment, handwritten text is categorized as a handwritten area that is not a handwritten drawing area representing a region.

[0077] In S316, the information processing server 103 detects a description indicating which part of the document image 113 the instruction refers to in response to the instruction text 114 received in S314. Subsequently, the information processing server 103 compares the detected description with the handwritten drawing area representing the region extracted in S315. This allows the information processing server 103 to identify the instruction area for the generating AI from the handwritten drawing area representing the region extracted in S315.

[0078] In S317, the information processing server 103 converts the instruction text 114 received in S314 into text in which the handwritten portion identified in S316 can be identified as an instruction portion for the generating AI. The information processing server 103 confirms the converted instruction text and the document image 113 received in S312 as the instruction content for the generating AI. Alternatively, the information processing server 103 may use an instruction text that reflects both the OCR-processed text of the document image 113 and the converted instruction text as the instruction content for the generating AI, instead of the converted instruction text.

[0079] In S331, the information processing server 103 transmits the instructions for the generating AI, which were finalized in S317, to the information processing device 101.

[0080] In S332, the information processing device 101 presents the instructions received in S331 to the user of the information processing system 100. This presentation prompts the information processing device 101 to confirm whether the instructions are as intended.

[0081] In S333, the user of the information processing system 100 inputs the result of confirming the instructions presented in S332.

[0082] In S334, the information processing device 101 sends the confirmation result entered in S333 to the information processing server 103. In S318, the information processing server 103 sends the instruction content received in S314 to the generating AI server 102 based on the confirmation result received in S334.

[0083] In S319, the generation AI server 102 sends back the response to the instructions received in S318 to the information processing server 103.

[0084] In S320, the information processing server 103 sends the response received in S319 to the information processing device 101. Alternatively, the information processing server 103 may display the response on the display device 267 instead of sending it to the information processing device 101. In this case, the information processing system 100 omits the processing in S321, which will be described later. In S321, the information processing device 101 displays the response received in S320 to the user of the information processing system 100 via the display device 210.

[0085] <Instructions for generating AI> Figure 4 is a flowchart illustrating the process described in Figure 3, from issuing instructions to the generation AI server 102 to outputting its response. The series of processes shown in the flowchart of Figure 4 are explained assuming that the CPU 261 of the information processing server 103 loads the program code stored in ROM 262 or storage 265 into RAM 264 and executes it.

[0086] In S401, the CPU 261 acquires a document image obtained by scanning a paper document or other original using the information processing device 101. In S402, the CPU 261 acquires an instruction text for the generating AI, which has been received as user input by the information processing device 101. S401 and S402 are examples of the "acquisition process" of the present invention.

[0087] An example of an instruction is text 603 in Figure 6(a). Text 603 is an instruction that instructs text 501 in Figure 5 to be summarized. Other instructions include translation into other languages, conversion to image information, retrieval of supplementary information such as background data and meaning, and correction of typographical errors. In addition, in S402, the CPU 261 has a function to determine whether the acquired instruction contains words that indicate a part, region, or location within the document image. For example, it makes this determination using well-known natural language processing methods such as "enclosed region" or "here".

[0088] In S403, the CPU 261 recognizes handwritten areas from the document image acquired in S401 using known techniques. The CPU 261 also determines whether any of the recognized handwritten areas contain handwritten drawings that represent a region.

[0089] As explained in S315, handwritten sections refer to all handwritten text within the document, while handwritten descriptive sections that indicate an area refer to handwritten sections where the beginning and end of the area are clearly identifiable.

[0090] Subsequently, if CPU261 finds a handwritten drawing area representing a region (Yes), it proceeds to process S404. If CPU261 does not find a handwritten drawing area representing a region (No), it proceeds to process S407.

[0091] Known techniques for recognizing handwritten areas include, for example, classifiers that cluster printed and handwritten areas, and extractors that extract pixels containing handwritten areas from an image. Examples of printed areas include text 501-503 in document image 500 in Figure 5(a). Examples of handwritten areas include the border line 510, marker area 512, and marker area 513 in document image 500.

[0092] When using these known technologies, the information processing system 100 may be provided as a learning device, or it may utilize one that exists on an external server via an API. In addition, the CPU 261 may not recognize the handwritten areas, but instead directly recognize the handwritten drawing areas that represent the regions.

[0093] In S404, the CPU 261 determines the type of shape of the handwritten drawing area that represents the region, which was recognized in S403. S404 is an example of the "determination step" in the present invention. This will be explained using Figure 6(a). Figure 6(a) is an example of a screen for the user to set and confirm the instructions.

[0094] As explained in S315, the handwritten areas that represent regions are the border lines 510 and the marker areas 512-513. The types of shapes are predetermined by the engineers or users, and include, for example, list boxes 601-602 and 604-605.

[0095] CPU261 clusters the handwritten areas representing regions to determine which of the list boxes 601-602 and 604-605 they correspond to. As a result of the clustering, the enclosure line 510 is assigned to the cluster of the closed region of list box 601, and the marker regions 512 and 513 are assigned to the cluster of the marker portion of list box 602.

[0096] Furthermore, known technologies will be used for the clustering method. The clustering process may be implemented on the information processing system 100, or it may be performed using a system located on an external server via an API.

[0097] In this embodiment, the location of the instruction text within the document image is determined by a process (not shown) that judges the relationship between the handwritten drawing and the document image. Alternatively, determination may be made using other factors such as color or thickness, or a combination thereof. Examples of determination results include colored marker areas, double underlines, and thick dashed lines.

[0098] In S405, the CPU 261 determines whether there is a candidate notation among the shape types determined in S404 that indicates "the location indicated by the instruction statement obtained in S402". If the CPU 261 finds a candidate notation indicating "the indicated location" among the shape types determined in S404 (Yes), it proceeds to process S413. If the CPU 261 finds no candidate notation indicating "the indicated location" among the shape types determined in S404 (No), it proceeds to process S407.

[0099] The "specified area" refers to which region of the document image acquired in S401 the instruction is directed to. Furthermore, if the instruction is not specific enough to define the "specified area," for example, if it simply says "summarize this section" instead of "summarize this section," the CPU 261 proceeds to process S407. Also, if the "specified area" in the instruction is not detected, even though it actually exists, such as in the case of the instruction "summarize the enclosed area," the CPU 261 proceeds to process S407.

[0100] Here, we will explain in detail, using Figure 6(a), how to determine whether there are candidate notations to indicate the "location being instructed." Figure 6(a) is an example screen for users to set and confirm the instructions.

[0101] Text 603 is an instruction to the generating AI for document image 500, entered by the user using cursor 612. The "here" (text 609) in text 603 is an instruction indicating a specific location within the document image, and the user's intention was to refer to text within a closed region, such as the border line 510 drawn on the document image. In other words, the instruction to the generating AI is to refer to the entire text area enclosed by a border line, such as the border line 510, on the document image.

[0102] Candidates for the notation indicating the "pointing location" are handwritten markings that represent an area, which "here" (text 609) in text 603 may refer to. If there are multiple handwritten markings that represent an area and have the same shape, they are treated as a single candidate.

[0103] As explained in S404, there are two types of shapes for the handwritten drawing areas that represent regions: closed regions (enclosing lines 510) and marker areas (marker regions 512-513). In other words, there are two candidates for the notation that indicate a specific "indicated area": ​​the enclosing line (510) and the marker regions (512-513). Therefore, CPU 261 determines that there are candidates for the notation that indicate a "indicated area".

[0104] Although not described in this embodiment, for example, in S404, if the cluster is identified as having markings such as stars or arrows, making it unclear what is being referred to, then the "here" (text 609) in text 603 becomes unclear. Therefore, in this case, CPU 261 determines that there are no candidate notations indicating the "pointing location".

[0105] In S413, CPU261 determines whether there is only one candidate for the notation indicating the "location being indicated" determined in S405. If there is only one candidate for the notation indicating the "location being indicated" (Yes), CPU261 proceeds to process S408. If there are multiple candidates for the notation indicating the "location being indicated" (No), CPU261 proceeds to process S406.

[0106] Here, we will explain how to determine whether there is only one candidate for the notation indicating the "instructed location" using Figure 6(a). As explained in S405, there are two candidates for the notation indicating the "instructed location" of instruction statement 603: the enclosed line (510) and the marker area (512-513). Therefore, CPU 261 determines that there are multiple candidates for the notation indicating the "instructed location".

[0107] For example, if the candidate for the notation indicating the "indicated location" is a marker area (512-513), then because they have the same marker shape, they are considered as one type (marker area), and the CPU 261 determines that there is only one candidate for the notation indicating the "indicated location". Alternatively, the CPU 261 may combine S405 and S413 into a single determination. Also, if there is only one candidate for the notation indicating the "indicated location" (Yes), the CPU 261 may omit S408 and proceed to the processing of S411.

[0108] In S406, the CPU 261 presents the user with candidate notations indicating the location of the instruction to the generating AI, which were detected during the judgment in S405, and accepts a selection from the candidates (an example of the acceptance process). The CPU 261 identifies the instruction location from the notation indicating the location of the instruction to the generating AI selected by the user (an example of the identification process). The user's selection will be described later in the chapter "User Interface Regarding Instructions to the Generating AI" using Figure 6.

[0109] In S407, CPU261 accepts input from the user regarding the location of the instructions for the generating AI. After that, CPU261 proceeds to processing in S408. The method of inputting the instructions can be, for example, by drawing them freehand on the image, by using pre-prepared rectangles or circles, or by tracing the text of the instruction location with a finger.

[0110] Alternatively, users may specify the entire document image as the area to be instructed to be generated by the AI. Methods for specifying the entire document image include, for example, placing a radio button representing the entire document on the screen that accepts input for the area to be instructed, or drawing a line around the entire document.

[0111] In S408, the CPU 261 converts the instruction obtained in S402 into an instruction that clearly identifies the instruction location specified in S406 or S407 (an example of the conversion process). The conversion of the instruction will be explained using Figure 6(a).

[0112] As explained in S405, in document image 500, there are two candidates for the notation indicating the indicated area: the enclosure line (510) and the marker area (512, 513). Now, suppose the user selects the closed area of ​​list box 601. That is, suppose the user selects the enclosure line (510) as the notation indicating the indicated area. At this time, the CPU 261 generates an instruction statement in which text 609 in instruction statement 603 is replaced with text 659 indicating the closed area of ​​list box 601.

[0113] Subsequently, CPU 261 adds text 643 indicating that the checkbox 619, which takes surrounding information into account, is checked, and generates instruction statement 643. Surrounding information refers to a set of information surrounding the instruction location to the left, right, and front of it, and is auxiliary information to prevent the instruction location from being interpreted in other ways. For example, if part of a paragraph is the instruction location, the surrounding information refers to the entire paragraph excluding that instruction location.

[0114] CPU261 may generate an instruction statement without adding text 649, or it may generate an instruction statement with text 649 added without providing checkbox 619, or it may add text 649 to the instruction statement at the time of regeneration by the generation AI.

[0115] In S411, the CPU 261 presents the document image acquired in S401 and the instruction text converted in S408 to the user of the information processing system 100, prompting the user to confirm whether they have understood the information. One way to prompt confirmation is to present the document image and instruction text on a confirmation screen to the user of the information processing system 100, and have them press the "OK" button if there are no problems, or the "Modify" button if corrections are needed.

[0116] In S412, the CPU 261 uses the verification result obtained in S411 to determine whether the document image obtained in S401 and the instruction text converted in S408 match the intent of the user using the information processing system 100. If it matches the intent of the user using the information processing system 100 (Yes), the process proceeds to S409. If it does not match the intent of the user using the information processing system 100 (No), the process proceeds to S407.

[0117] For example, if the CPU 261 detects that the "OK" button has been pressed in S411, it determines that this matches the intention of the user using the information processing system 100. Conversely, if the CPU 261 detects that the "Modify" button has been pressed in S411, it determines that this does not match the intention of the user using the information processing system 100.

[0118] In S409, the CPU 261 finalizes the document image acquired in S401 and the instruction text converted in S408 as the instruction content for the generating AI. The CPU 261 then sends the finalized instruction content to the generating AI server 103 (an example of the output process). The CPU 261 may use an instruction text obtained by modifying the instruction text acquired in S408 as the instruction content for the generating AI, instead of the instruction text acquired in S408.

[0119] Here, the processed instruction text will be explained using Figure 5(d). Figure 5(d) shows the OCR result for the document image 500 in Figure 5(a). The instruction text converted in S408 is assumed to be text 643 in Figure 6(b).

[0120] In this case, the modified instruction text refers to text that, for example, is divided into two chapters, "Instructions" and "OCR Results," with text 643 containing the preface "Referring to the OCR results," inserted into the chapter showing the "Instructions," and text 530 containing the OCR results inserted into the chapter showing the "OCR Results." This can sometimes improve the processing performance of the generation AI by adding the OCR results, compared to simply inputting the document image into the generation AI and having it process it.

[0121] In S410, the CPU 261 obtains a response from the generation AI server 102 in response to the instructions entered in S409. The CPU 261 then presents the obtained response to the user. Examples of presentation to the user are explained using Figures 5(c) and 6(b). Figure 5(c) shows the result of summarizing text 501. Figure 6(b) is an example of a screen where the user confirms the result of giving instructions to the generation AI.

[0122] The CPU 261 displays the summary result 521 on the screen 620 along with the document image 500 and the instruction text 643 input to the generation AI. At this time, the text 521 may be processed and changed to a format that is easy for the user to understand. Alternatively, the CPU 261 may display the instruction text that has been processed from the instruction text 643 and input to the generation AI, or it may display only the summary result 521. In this embodiment, the summary result is displayed on the screen, but it may also be possible to output it in the form of metadata or comments for the document image.

[0123] <User interface for giving instructions to the generating AI> Figure 6 illustrates the user interface for giving instructions to the generating AI. Figure 6 shows an example of a screen on the display device 210 of the information processing system 100 where the user operates and confirms.

[0124] First, let's explain how to determine the location of the instructions for the generating AI. Figure 6(a) is an example of a screen for the user to set and confirm the instructions. Screen 600 consists of a document image 500 entered by the user, list boxes 601-602 and 604-605 for selecting the location of the instructions, a pull-down button 611, an instruction text 603, a cursor 612 for entering the instruction text, and a "Set" button 613 for confirming the instructions. In addition, screen 600 has a checkbox 619 that indicates whether to take surrounding information into account as well as the location of the instructions for the generating AI.

[0125] As explained in S406, the information processing system 100 presents the user with a bounding box (510) and marker areas (512-513) as candidates for notation indicating the location of instructions to be given to the generating AI. In this embodiment, the closed area of ​​list box 601 corresponds to the option for the bounding box (510), and the marker area of ​​list box 602 corresponds to the option for the marker areas (512-513).

[0126] When the information processing system 100 detects that a user has pressed the pull-down button 611, it displays the list boxes 602, 604, and 605, which represent the unselected options. Note that other formats, such as radio buttons, are acceptable instead of list boxes, as long as they fulfill the function of selection.

[0127] Furthermore, as explained in S402, the information processing system 100 accepts the input of instruction sentences 603 for the generating AI from the user. In this embodiment, the method of inputting instruction sentences is implemented using a cursor 612, but it may also be in the form of selecting and editing instruction sentences prepared in advance by the engineer or user, editing template instruction sentences, or selecting and modifying from past input history. In addition, the information processing system 100 may save the instruction location and instruction sentence as a history of the content of instructions to the generating AI, and when a similar handwritten location is recognized, it may recommend displaying past instruction sentences.

[0128] Furthermore, when the information processing system 100 detects that the user has pressed the setting button 613, it executes the processing as described in S401 to S408 to determine the location and wording of the instructions for the generating AI. At this time, when the information processing system 100 detects that the user has made a selection operation in the list boxes 601 to 602, 604 to 605, or the checkbox 619, it may interactively convert the wording of the instructions and replace wording 603 with wording 643.

[0129] Next, we will explain how to check the response from the generating AI to the instructions. Figure 6(b) is an example of a screen in which the user checks the results of instructions given to the generating AI. Screen 620 consists of a document image 500, instructions to the generating AI 643, the response result from the generating AI 521, a "Modify Instructions" button 621, and an "OK" button 623.

[0130] As explained in S410, the information processing system 100 presents the user with the response from the generating AI. When the information processing system 100 detects that the user has pressed the "Modify Instructions" button 621, it transitions to screen 600. Furthermore, when the information processing system 100 detects that the user has pressed the "OK" button 623, it considers that the response from the generating AI is complete, closes screen 600, and terminates the system.

[0131] As described above, Embodiment 1 converts the instruction text entered by the user into an instruction text that allows the AI ​​to identify the specific part of the instruction to be given. Therefore, it is possible to reduce the effort required for the user to think of an instruction text while providing instructions that align with the user's intentions.

[0132] [Embodiment 2] In Embodiment 1, when there was only one type of notation indicating instructions to the generating AI, the instructions to the generating AI were executed. On the other hand, in Embodiment 2, when there are multiple types of notations indicating instructions to the generating AI, the instructions to the generating AI are switched according to each type. Embodiment 2 will be explained mainly by replacing Figure 6 in Embodiment 1 with Figure 7 and Figure 4 with Figure 8. In Embodiment 2, all other points are the same as in Embodiment 1, so the explanation will be omitted.

[0133] <Instructions for generating AI> Figure 8 is a flowchart illustrating the flow from giving instructions to the generating AI server 102 to outputting its response in Embodiment 2. Of the processes shown in the flowchart of Figure 8, those with the same reference numerals as those in the flowchart of Figure 4 are basically the same as in Embodiment 1, so their explanation is omitted. However, for processes with the same reference numerals as those in the flowchart of Figure 4 that differ from those in Embodiment 1, only the differences will be explained.

[0134] In S402, the CPU 261 obtains the instruction text for the generating AI, which has been received as user input by the information processing device 101. At this time, the CPU 261 obtains one instruction text for each type of notation indicating an instruction for the generating AI. This will be explained using Figure 7(a). Figure 7(a) is an example of a screen for the user to set and confirm the content of the instruction.

[0135] In this embodiment, there are two instruction statements (text 603 and text 703). Text 603 is an instruction statement to the generating AI entered by the user using cursor 612 for text 501 in document image 500 (text within the closed region of list box 601). Text 703 is an instruction statement to the generating AI entered by the user using cursor 612 for texts 722-723 in document image 500 (text in the marker portion of list box 602). Figure 7 will be explained in detail in the chapter "User Interface Regarding Instructions to the Generating AI".

[0136] In S801, the CPU 261 executes the processes from S405 to S802 for each instruction statement obtained in S402. For example, in the example in Figure 7(a), as explained in S402, there are two instruction statements (text 603 and text 703), so the CPU 261 executes the processes from S405 to S802 for each of text 603 and text 703. In other words, the CPU 261 executes the processes from S405 to S802 twice.

[0137] In S802, the CPU 261 obtains the response from the generating AI server 102 to the instructions entered in S409.

[0138] In S803, the CPU261 determines whether it has finished processing all the instructions obtained in S801, and repeats the processing from S405 to S803 until it is determined that processing has finished for all the instructions.

[0139] In S804, CPU261 compiles the responses obtained in S802 for each instruction and presents them to the user. An example of how the information is presented to the user is explained using Figure 7(b). Figure 7(b) is an example of a screen in which the user confirms the results of instructions given to the generating AI.

[0140] First, CPU261 displays document image 500, which is common to both instruction sentences (643 and 751), on screen 720. Then, as the first answer, CPU261 displays the answer result 521 on screen 720 along with instruction sentence 643, which was input to the generation AI.

[0141] Furthermore, the CPU 261 displays the response results 742-743 on screen 720 along with the instruction 751 input to the generating AI as the second response. At this time, the CPU 261 may process response results 521 and 742-743 to change them into a format that is easy for the user to understand. Alternatively, the CPU 261 may display the instruction 643 and 751 that have been processed and input to the generating AI instead of the instruction 643 and 751, or it may display only response results 521 and 742-743.

[0142] In this embodiment, the results are displayed on the response screen, but the output may be made in the form of metadata or comments for the document image, or it may be made to be printed on paper from the information processing device 101. Furthermore, the CPU 261 may process the document image 500 for each instruction into an image that visually indicates which part of the document the instruction refers to, and then display that processed image for each instruction.

[0143] In this embodiment, CPU 261 executes S411 to S802 for each instruction statement, but it may also execute them all at once. Conversely, CPU 261 may execute S804 for each instruction statement or instruction location.

[0144] <User interface for giving instructions to the generating AI> Figure 7 shows an example of a screen on the display device 210 of the information processing system 100 in Embodiment 2, which is operated and confirmed by the user.In Figure 7, the processing of the same reference numerals as in Figure 6 is the same as in Embodiment 1, so its explanation is omitted.

[0145] First, let's explain how to determine the location of the instructions for the generating AI. Figure 7(a) is an example of a screen in Embodiment 2 for the user to set and confirm the content of the instructions. Screen 700 has a list box and a pull-down button for selecting the type of notation that indicates instructions for the generating AI, and an input field for the instruction text, for each of the two types of notations that indicate instructions for the generating AI within the document image 500. The first location is the same as in Embodiment 1, so we will omit the explanation and will explain only the second location below.

[0146] All the types of shapes to be identified in S404 are displayed in list boxes 601-602 and 604-605 as options for notation indicating instructions to the generating AI. In the first instance, the closed region of list box 601 is selected by the user as the notation indicating instructions to the generating AI. Therefore, in the second instance, the information processing system 100 presents the user with list boxes 702-704-705, which have the closed region removed, as options for notation indicating instructions to the generating AI.

[0147] When the information processing system 100 detects that the user has pressed the pull-down button 711, it displays the list boxes 704 to 705, which are the unselected options. As explained in Embodiment 1, other formats such as radio buttons are also acceptable instead of list boxes, as long as they can perform the function of selection. Also, as explained in S402, the information processing system 100 accepts the input of an instruction sentence 703 for the generating AI from the user. The method of inputting the instruction sentence is the same as in Embodiment 1, so the explanation is omitted.

[0148] Furthermore, when the information processing system 100 detects that the user has pressed button 713, it adds new input fields for notation and instruction text indicating instructions to the generating AI. The maximum number of fields that can be added is equal to the number of shapes to be identified in S404.

[0149] When the system detects that the user has pressed the setting button 613, it executes the process described in S401 to S408 to determine the location and wording of the instructions for the generating AI. At this time, when the information processing system 100 detects that the user has made a selection operation in the list boxes 601 to 602, 604 to 605, and the checkbox 619, it may interactively convert the wording of the instructions and replace wording 603 with wording 643.

[0150] Finally, when the information processing system 100 detects that the user has pressed button 714, it executes the processing as described in S401 to S408, and obtains the instruction location and instruction text for all notations indicating instructions to the generating AI. In this embodiment, since the first notation indicating an instruction to the generating AI is a closed area of ​​list box 601, the first instruction location is obtained as the text area 501 of the enclosed line 510, and the first instruction text is obtained as text 603.

[0151] Furthermore, since the notation indicating the second instruction to the generating AI is in list box 701, the second instruction location is obtained as text area 722-723 in marker area 512-513, and the second instruction text is obtained as text 703. Subsequently, as explained in S408, the information processing system 100 converts the confirmed instruction location and instruction text into instruction texts that clearly indicate the instruction location to the generating AI.

[0152] In other words, in this embodiment, the first instruction after conversion is text 643 in Figure 7(b), which reflects the information of the closed area of ​​list box 601 and checkbox 619 in text 603. The second instruction after conversion is text 751 in Figure 7(b), which reflects the information of the marker portion and check of list box 702 in text 703.

[0153] Next, we will explain how to check the response from the generating AI to the instructions. Figure 7(b) is an example of a screen in Embodiment 2 where the user checks the results of instructions given to the generating AI. Screen 720 is an additional screen to screen 620 in Figure 6(b), with a second instruction sentence 751 for the generating AI and the response results 742-743 from the generating AI to the second instruction sentence.

[0154] As explained in S804, the information processing system 100 presents the user with the response from the generating AI for each notation indicating an instruction to the generating AI. At this time, the information processing system 100 presents the document image 500 used to give instructions to the generating AI, and then presents the instruction text (643, 751) and the response from the generating AI (521, 742~743) as a set for each notation indicating an instruction to the generating AI.

[0155] Furthermore, the information processing system 100 may display a corresponding instruction and response when it detects a notation indicating an instruction location or a user selection for an instruction location. For example, after detecting a user's selection of the enclosure line 510 or marker areas 512-513, the information processing system 100 may display an instruction and response to the generating AI corresponding to the detected handwritten drawing area representing the area. In this case, the information processing system 100 may also explicitly indicate the notation indicating the instruction to the generating AI in text.

[0156] As described above, according to the present invention, even when there are multiple types of notations indicating instructions to the generating AI, the content of the instructions to the generating AI can be switched according to the type of notation, thereby enabling instructions that match the user's intentions.

[0157] [Embodiment 3] In Embodiment 1, when a user of the information processing system 100 sets the instruction content, all types of shapes to be identified in S404 of Figure 4 are presented as options for the instruction location to be given to the generating AI. On the other hand, in Embodiment 3, only the types of shapes contained in the document image acquired in S401 of Figure 4 are presented as options. Embodiment 3 will be explained by replacing Figure 6(a) in Embodiment 1 with Figure 9. In Embodiment 3, all other points are the same as in Embodiment 1, so the explanation will be omitted.

[0158] <User interface for giving instructions to the generating AI> Figure 9 shows an example of a screen on the display device 210 of the information processing system 100 in Embodiment 3, for which the user sets and confirms instructions. Of the processes in Figure 9 that have the same reference numerals as those in Figure 6(a) are the same as in Embodiment 1, so their explanation is omitted.

[0159] Screen 900 is a version of Screen 600 in which the options for notation indicating the indicated location are limited to list boxes 601 to 602, which are notations present within the document image. As explained in S404 of Figure 4, for the document image 500, the border line 510 is assigned to the closed region cluster of list box 601, and the marker areas 512 and 513 are assigned to the marker cluster of list box 602.

[0160] In other words, document image 500 contains only two of the clusters prepared in advance by engineers or users. Therefore, the information processing system 100 presents the closed region of list box 601 and the marker portion of list box 602 to the user of the information processing system 100 as options for notation indicating the location of instructions to be given to the generating AI.

[0161] In this embodiment, when showing options for notation indicating the location of instructions to the generating AI, these options are displayed as text. However, they may also be displayed as images of handwritten drawings representing the area. For example, the information processing system 100 may display a reduced and conceptualized image of the border line 510 instead of the text "closed area (enclosure, etc.)" in the list box 601.

[0162] Furthermore, in this embodiment, the user of the information processing system 100 indicates the location to be instructed to the generating AI by selecting and pressing the list boxes 601 to 602. However, for example, the user of the information processing system 100 may also indicate the location to be instructed to the generating AI by selecting and pressing the handwritten drawing areas (outline 510, marker areas 512 to 513) that represent regions on the document image 500 on the screen 900.

[0163] When the information processing system 100 detects a selection or press of a handwritten drawing area representing a region by the user, it reflects the selection result on list boxes 601 to 602. That is, for example, when it detects a press of the border line 510, the information processing system 100 selects list box 601. In addition, when the information processing system 100 detects a selection or press of a handwritten drawing area representing a region by the user, it may highlight the selected or pressed drawing area with a highlighter or other means.

[0164] As described above, according to Embodiment 3 of the present invention, the information processing system 100 can present to the user only the portions contained within the received document image as options for the location to be instructed to the generating AI. This reduces the effort required for the user to select a specified location.

[0165] [Embodiment 4] In Embodiment 1, the user of the information processing system 100 had to input or specify the instructions for the generating AI in some form. On the other hand, in Embodiment 4, if there is a handwritten comment near the instruction area for the generating AI in the document image, that comment is reflected in the instructions. Embodiment 4 will be explained by replacing Figure 6(a) in Embodiment 1 with Figure 10, and the document image 500 in Figure 5(a) with the document image 1010 in Figure 10. In Embodiment 4, all other points are the same as in Embodiment 1, so the explanation will be omitted.

[0166] <User interface for giving instructions to the generating AI> Figure 10 shows an example screen on the display device 210 of the information processing system 100 in Embodiment 4, which allows the user to set and confirm instructions. Of the processes in Figure 10 that have the same reference numerals as those in Figure 6(a) are the same as in Embodiment 1, so their explanation is omitted.

[0167] Document image 1010 in Figure 10(a) is the same as document image 500 in Figure 5(a) with handwritten text 1003 added. Screen 1000 is the same as Figure 6(a) but with document image 500 replaced by document image 1010 and instruction text 603 replaced by instruction text 1013.

[0168] The information processing system 100 selects a notation indicating the location of the instruction to be given to the generating AI by the user when a button 601 is pressed. When the instruction location estimation button 1011 is pressed, it performs a search process for surrounding handwritten characters for the shape indicated in 601. Although not shown in the diagram, the search process can be performed as the process S404 in Figure 4.

[0169] As a concrete example, the system searches for the closest handwritten character among those surrounding the enclosed area 510 of the list box 601. The method for searching for surrounding handwritten characters is, for example, to calculate the distance from a point on the enclosed area 510 to handwritten characters other than the "handwritten drawing area representing the area," and select the one with the closest distance. In other words, in this embodiment, the information processing system 100 determines that the handwritten area 1003 is the corresponding area and presents it as a handwritten character surrounding the enclosed area 510.

[0170] Subsequently, the information processing system 100 performs OCR on the handwritten portion (string) 1003 and obtains text 1013 as an OCR result. The information processing system 100 reflects the obtained OCR result text 1013 in the instruction field for the generation AI shown in Figure 10(b). The reflected text 1013 can be modified by the user, as in Embodiment 1. If no corresponding handwritten portion is found nearby, the information processing system 100 operates as in Embodiment 1.

[0171] Furthermore, the information processing system 100 may, for example, present all handwritten characters that are closer than other "handwritten drawing areas representing regions" or within a certain threshold as surrounding handwritten characters. The information processing system 100 may, for example, present all handwritten characters other than "handwritten drawing areas representing regions" in order of proximity to the enclosure line 510, regardless of whether they are surrounding or not.

[0172] Furthermore, in this embodiment, OCR was performed after searching for the nearest handwritten character, but the timing of performing OCR is not limited to this. Also, when the information processing system 100 searches for the nearest handwritten character, it may pre-select the areas containing handwritten characters other than symbols.

[0173] As described above, according to Embodiment 4 of the present invention, handwritten comments near the instruction points for the generating AI can be presented to the user as instruction text. This reduces the effort required for the user to input and modify instruction text.

[0174] [Embodiment 5] In Embodiment 1, the instruction text was transformed in order to identify the parts of the output AI that needed to be instructed. In contrast, in Embodiment 5, in addition to the instruction text, the input document image is also transformed. Embodiment 5 will be explained mainly by replacing Figure 4 in Embodiment 1 with Figure 11 and Figure 6(b) with Figure 12. In Embodiment 5, all other aspects are the same as in Embodiment 1, so the explanation will be omitted.

[0175] <Information Processing System> Figure 1 is a block diagram showing an example configuration of the information processing system 100 of the present invention, as described in Embodiment 1. In Embodiment 5, only the differences from Embodiment 1 will be explained.

[0176] The instruction content generation unit 155 receives the instruction text 114 from the information processing device 101. Based on the handwritten portion of the document image 113 and the instruction text 114, the instruction content generation unit 155 generates an instruction image (not shown) that identifies the portion to be instructed to the generation AI, and an instruction text (not shown) that identifies the portion to be instructed to the generation AI.

[0177] An instruction image is, for example, an image of document image 113 with handwritten text outside the instruction area removed, an image with a semi-transparent mask image superimposed on the area outside the instruction area, or an image that conforms to a predefined notation by the engineer, such as "the instruction area will be represented by a red circle." An instruction sentence that allows the generation AI to identify the instruction area is, for example, text in which the instruction words in instruction sentence 114 have been replaced with specific notations such as "outline" that represent the handwritten area on the document image.

[0178] The instruction content generation unit 155 finalizes the instruction content, which includes the instruction image and an instruction statement that identifies the location to be instructed to the generating AI. Alternatively, the instruction content generation unit 155 may generate an instruction statement (not shown) with the OCR results acquired by the document image analysis unit 154 added, and finalize this instruction statement and the document image 113 as the instruction content. Adding the OCR results can assist the generating AI in acquiring information from the image.

[0179] <Usage Sequence> Figure 3 is a sequence diagram illustrating the process from issuing instructions to the generation AI server 102 using the document image 113 and instruction text 114 acquired by the information processing device 101, as explained in Embodiment 1, to outputting the response. In Embodiment 5, only the differences from Embodiment 1 will be explained.

[0180] In S317, the information processing server 103 converts the document image 113 received in S312 into an image in which the handwritten portions identified in S316 can be identified as instruction locations for the generating AI. At this time, the information processing server 103 may store the document image 113 in storage 265 and perform the conversion process on a copy of the document image 113.

[0181] Furthermore, the information processing server 103 converts the instruction text 114 received in S314 into text in which the handwritten portion identified in S316 can be identified as an instruction portion for the generating AI. Subsequently, the information processing server 103 confirms the converted image and the converted instruction text as the instruction content for the generating AI. The information processing server 103 may also use an instruction text that reflects both the text obtained by OCR on the document image 113 and the converted instruction text as the instruction content for the generating AI, instead of the converted instruction text.

[0182] <Instructions for generating AI> Figure 11 is a flowchart illustrating the flow from giving instructions to the generating AI server 102 to outputting its response in Embodiment 5. Of the processes shown in the flowchart of Figure 11, those with the same reference numerals as those in the flowchart of Figure 4 are basically the same as in Embodiment 1, so their explanation is omitted. However, for processes with the same reference numerals as those in the flowchart of Figure 4 that differ from those in Embodiment 1, only the differences will be explained.

[0183] In S1101, the CPU 261 converts the document image acquired in S401 into an image in which the indicated area identified in S406 or S407 is clearly visible. The image conversion will be explained using Figure 6(a). As explained in S405, in the document image 500, there are two candidates for the notation indicating the indicated area: the border line (510) and the marker area (512, 513). Now, suppose the user has selected a closed area in the list box 601. That is, suppose the user has selected the border line (510) as the notation indicating the indicated area.

[0184] At this time, the CPU 261 generates an image that processes only the outline (510) from the handwritten parts (510, 512, 513) of the document image 500. In other words, the image in which the indicated area is clearly identifiable is an image in which a layer containing only the outline 510, which is the notation indicating the indicated area, is superimposed on the layer containing the printed parts 510 to 512. This superimposed image becomes image 1230 in Figure 12.

[0185] Furthermore, an image clearly showing the indicated area may be generated by overwriting the document image 500, or it may be generated separately from the document image 500. Also, although not described in this embodiment, the CPU 261 may change the notation method for the area indicated to the generating AI, rather than the enclosure line 510 itself. For example, the CPU 261 may draw a red enclosure line, which is predefined as the notation for the area indicated to the generating AI, instead of the enclosure line 510, at the area pointed to by the enclosure line 510.

[0186] In this case, in S408, CPU 261 replaces text 609 in instruction 603 with a predefined text, "the part circled in red," instead of the information in the list box. This converts the instruction into one that clearly indicates the part to be instructed to be generated by the AI. The predefined text can be set by the user or administrator. In this embodiment, handwritten parts other than the notation indicating the part to be instructed are removed from the image input to the AI, but instead of removal, a semi-transparent mask may be drawn.

[0187] In S1102, the CPU 261 finalizes the image converted in S1101 and the instruction text converted in S408 as instructions for the generating AI. The CPU 261 then sends the finalized instructions to the generating AI server 103. Note that the CPU 261 may use an instruction text obtained in S408, or a modified version of the instruction text obtained in S408, as the instructions for the generating AI.

[0188] <User interface for giving instructions to the generating AI> Figure 12 shows an example of a screen in Embodiment 5 where the user can check the results of instructions given to the generating AI. Of the processes in Figure 12 that have the same reference numerals as those in Figure 6(b) are the same as in Embodiment 1, so their explanation is omitted.

[0189] Screen 1220 is the same as screen 620 in Figure 6(b), but with document image 500 replaced by image 1230. In other words, the image input to the generation AI has changed from document image 500 to image 1230.

[0190] As described above, according to Embodiment 5 of the present invention, the generating AI can identify the location to be instructed to generate the AI ​​from both the input image and the instruction text. This reduces the possibility that the generating AI may misrecognize the location specified by the user, and enables it to provide instructions that are more in line with the user's intentions.

[0191] [Embodiment 6] Embodiment 1 shows an example of converting instructions for the generating AI so that the specified location can be identified. Embodiment 6 shows an example of clarifying the specified location by converting the image input to the generating AI.

[0192] The same parts as in Embodiment 1 will not be explained. However, even in the parts where the explanation is omitted, the parts that are explained using Figure 6 should be read as referring to Figure 14.

[0193] <Information Processing System> The configuration of the information processing system 100 is the same as that described in Embodiment 1 using Figure 1. However, in Embodiment 6, the instruction statement analysis unit 159 may be omitted. In this case, the instruction statement 114 acquired by the instruction statement acquisition unit 152 is output directly to the instruction content generation unit 155 (without going through the instruction statement analysis unit 159).

[0194] In this embodiment, the instruction content generation unit 155 receives an instruction statement 114 from the information processing device 101. Based on the handwritten portion of the document image 113 and the instruction statement 114, the instruction content generation unit 155 generates an instruction image (not shown) that identifies the portion of the instruction statement 114. The instruction image is, for example, an image obtained by deleting the handwritten portion other than the instruction portion from the document image 113. The instruction content generation unit 155 then finalizes the instruction statement 114 and the instruction image as the instruction content.

[0195] Alternatively, the instruction content generation unit 155 may generate an instruction text (not shown) containing the OCR results obtained by the document image analysis unit 154 and the instruction text 114, and then finalize this instruction text and instruction image as the instruction content.

[0196] Embodiments 6 to 9 describe a configuration in which the instruction analysis unit 159 is not provided.

[0197] <Device configuration> The hardware configuration example of the information processing device 101, the generation AI server 102, and the information processing server 103 included in the information processing system 100 is the same as that described with reference to Figure 2 in Embodiment 1.

[0198] <Usage Sequence> The process of using the document image 113 and instruction text 114 acquired by the information processing device 101 to send instructions to the generation AI server 102 and output the response will be explained with reference to Figure 3. Here, since everything except S317 is the same as described in Embodiment 1, only the explanation of S317 will be given here.

[0199] In S317, the information processing server 103 converts the document image 113 received in S312 into an image in which the handwritten portions identified in S316 can be identified as instruction locations for the generating AI. At this time, the information processing server 103 may store the document image 113 in storage 265 and perform the conversion process on a copy of the document image 113.

[0200] The information processing server 103 confirms the converted image and the instruction statement 114 received in S314 as the instruction content for the generating AI. Alternatively, the information processing server 103 may use an instruction statement that reflects both the text obtained by OCR on the document image 113 and the instruction statement 114 as the instruction content for the generating AI.

[0201] <Instructions for generating AI> Figure 13 is a flowchart illustrating the process described in Figure 3, from issuing instructions to the generation AI server 102 to outputting its response. The series of processes shown in the flowchart of Figure 13 are explained assuming that the CPU 261 of the information processing server 103 loads the program code stored in ROM 262 or storage 265 into RAM 264 and executes it.

[0202] In addition, the same numbers are used in Figure 13 for parts that were explained using Figure 4 in Embodiment 1, and the explanations here are omitted. However, even in parts where the explanation is omitted, parts that were explained using Figure 6 should be read as being in Figure 14.

[0203] In S1305, the CPU 261 determines whether there is a candidate notation among the shape types determined in S404 that indicates "the location indicated by the instruction statement obtained in S402". If the CPU 261 finds a candidate notation indicating "the indicated location" among the shape types determined in S404 (Yes), it proceeds to process S413. If the CPU 261 finds no candidate notation indicating "the indicated location" among the shape types determined in S404 (No), it proceeds to process S407.

[0204] The "specified area" refers to which region of the document image acquired in S401 the instruction is directed to. Furthermore, if the instruction is not specific enough to define the "specified area," for example, if it simply says "summarize this section" instead of "summarize this section," the CPU 261 proceeds to process S407. Also, if the "specified area" in the instruction is not detected, even though it actually exists, such as in the case of the instruction "summarize the enclosed area," the CPU 261 proceeds to process S407.

[0205] Here, we will explain in detail, using Figure 14(a), how to determine whether there are candidate notations indicating the "location being instructed." Figure 14(a) is an example screen for users to set and confirm the instructions.

[0206] Text 603 is an instruction to the generating AI for document image 500, entered by the user using cursor 612. The word "here" in text 603 is an instruction indicating a specific location within the document image, and the user's intention was to refer to text within a closed region, such as a border line 510 drawn on the document image. In other words, the instruction to the generating AI is to refer to the entire text area enclosed by a border line, such as the border line 510, on the document image.

[0207] Candidates for the notation indicating the "pointing location" are handwritten markings in text 603 that represent an area and may be referred to by "here". If there are multiple handwritten markings that represent an area and have the same shape, they will be treated as a single candidate.

[0208] As explained in S404, there are two types of shapes for the handwritten drawing areas that represent regions: closed regions (enclosing lines 510) and marker areas (marker regions 512-513). In other words, there are two candidates for the notation that indicate a specific "indicated area": ​​the enclosing line (510) and the marker regions (512-513). Therefore, CPU 261 determines that there are candidates for the notation that indicate a "indicated area".

[0209] Although not described in this embodiment, for example, in S404, if a cluster is identified as having markings such as stars or arrows, making it unclear what is being referred to, then the meaning of "here" in text 603 becomes unclear. Therefore, in this case, CPU 261 determines that there are no candidate notations indicating the "pointing location".

[0210] In S413, CPU261 determines whether there is only one candidate for the notation indicating the "location being indicated" determined in S1305. If there is only one candidate for the notation indicating the "location being indicated" (Yes), CPU261 proceeds to process S411. If there are multiple candidates for the notation indicating the "location being indicated" (No), CPU261 proceeds to process S406.

[0211] Here, we will explain how to determine whether there is only one candidate for the notation indicating the "instructed area" using Figure 14(a). As explained in S405, there are two candidates for the notation indicating the "instructed area" of instruction statement 603: the enclosed line (510) and the marker area (512-513). Therefore, CPU 261 determines that there are multiple candidates for the notation indicating the "instructed area".

[0212] For example, if a candidate for the notation indicating the "indicated location" is a marker area (512-513), the CPU 261 will consider it as a single type (marker area) because they share the same marker shape, and will determine that there is only one candidate for the notation indicating the "indicated location." Alternatively, the CPU 261 may treat S1305 and S413 as a single determination.

[0213] In S1306, CPU261 presents the user with candidate notations indicating the location of the instruction to the generating AI, which were detected during the judgment in S1305, and accepts a selection from the candidates. CPU261 identifies the instruction location from the notation indicating the location of the instruction to the generating AI selected by the user. After that, CPU261 proceeds to the processing in S1308.

[0214] In S1308, the CPU 261 converts the document image acquired in S401 into an image in which the specified area identified in S1306 or S407 is clearly visible. The image conversion will be explained using Figure 14(a).

[0215] As explained in S1305, in document image 500, there are two candidates for the notation indicating the indicated area: the border line (510) and the marker area (512, 513). Now, suppose the user selects a closed area in list box 601. That is, suppose the user selects the border line (510) as the notation indicating the indicated area. At this time, the CPU 261 generates an image from the handwritten areas (510, 512, 513) of document image 500, with the border line (510) as the target of processing.

[0216] In other words, an image that clearly shows the indicated area is an image created by overlaying a layer containing only the outline 510, which indicates the indicated area, onto the layer containing the printed text 510-512. This overlaid image is image 1430 in Figure 14(b). Note that the image that clearly shows the indicated area may be generated by overwriting document image 500, or it may be generated separately from document image 500.

[0217] In S1311, the CPU 261 presents the user of the information processing system 100 with the instruction text acquired in S402 and the image converted in S1308, prompting the user to confirm whether it matches their intention. Methods for prompting confirmation include, for example, presenting the instruction text and image on a confirmation screen to the user of the information processing system 100, and having them press the "OK" button if there are no problems, or the "Modify" button if corrections are needed.

[0218] In S1312, the CPU 261 uses the confirmation result obtained in S1311 to determine whether the instruction text obtained in S402 and the image converted in S1308 match the intention of the user using the information processing system 100. If it matches the intention of the user using the information processing system 100 (Yes), the process proceeds to S1309. If it does not match the intention of the user using the information processing system 100 (No), the process proceeds to S407.

[0219] For example, if the CPU 261 detects that the "OK" button has been pressed in S1311, it determines that this matches the intention of the user using the information processing system 100. Conversely, if the CPU 261 detects that the "Modify" button has been pressed in S1311, it determines that this does not match the intention of the user using the information processing system 100.

[0220] In S1309, the CPU 261 confirms the instruction text acquired in S402 and the image converted in S1308 as the instruction content for the generation AI. Then, the CPU 261 sends the confirmed instruction content to the generation AI server 103. Note that the CPU 261 may use an instruction text that is a modified version of the instruction text acquired in S402 as the instruction content for the generation AI, instead of the instruction text acquired in S402.

[0221] Here, we will explain the modified instruction text using Figure 5(d). Figure 5(d) shows the OCR result for the document image 500 in Figure 5(a). Assume that the user inputs the text 603 in Figure 14(a) as an instruction text to the generation AI. In this case, a modified instruction text is, for example, divided into two chapters: "Instructions" and "OCR Results". The chapter indicating "Instructions" contains the text 603 with the preface "Refer to the OCR results, in the image", and the chapter indicating "OCR Results" contains the text with the OCR results 530 inserted.

[0222] Another example is text that includes a preface to the generating AI, such as "Please perform the following instructions on the image below," to clarify the instructions for the generating AI. This can sometimes improve the processing performance of the generating AI by adding the OCR results, compared to simply inputting the document image into the generating AI for processing.

[0223] In S1310, the CPU 261 obtains a response from the generation AI server 102 in response to the instructions entered in S1309. The CPU 261 then presents the obtained response to the user. Examples of presentation to the user are explained using Figures 5(c) and 14(b). Figure 5(c) shows the result of summarizing text 501. Figure 14(b) is an example of a screen where the user confirms the result of giving instructions to the generation AI.

[0224] The CPU 261 displays the summary result 521 on the screen 620 along with the instruction text 603 entered by the user and the image 1430 input to the generation AI. At this time, the text 521 may be processed and changed to a format that is easy for the user to understand. Alternatively, the CPU 261 may display the instruction text that has been processed from the instruction text 603 and input to the generation AI, or it may display only the summary result 521. In this embodiment, the summary result is displayed on the screen, but it may also be possible to output it in the form of metadata or comments for the document image data.

[0225] <User interface for giving instructions to the generating AI> Figure 14 illustrates the user interface for giving instructions to the generating AI. Figure 14 shows an example of a screen on the display device 210 of the information processing system 100 where the user operates and confirms.

[0226] The difference between Figure 14(a) and Figure 6(a) is that Figure 14(a) does not have checkbox 619.

[0227] When the information processing system 100 detects that the user has pressed the setting button 613, it confirms the instruction location and instruction text for the generating AI. In this embodiment, the information processing system 100 confirms the instruction location for the generating AI as a list box 601 and the instruction text as text 603. Subsequently, as described in S1308, the information processing system 100 converts the confirmed instruction location and instruction text into an image in which the instruction location for the generating AI is clearly visible.

[0228] Next, we will explain how to check the response from the generating AI to the instructions given. Figure 14(b) is an example of a screen in which the user checks the results of instructions given to the generating AI. Screen 620 consists of the image 1430 used to give instructions to the generating AI, the instruction text 603 to the generating AI, the response result from the generating AI 521, the "Modify Instructions" button 621, and the "OK" button 623.

[0229] As explained in S1310, the information processing system 100 presents the user with the response from the generating AI. When the information processing system 100 detects that the user has pressed the "Modify Instructions" button 621, it transitions to screen 600. Furthermore, when the information processing system 100 detects that the user has pressed the "OK" button 623, it considers that the response from the generating AI is complete, closes screen 600, and terminates the system.

[0230] As described above, according to Embodiment 6, by converting a document image into an image in which the parts to be instructed to the AI ​​can be identified, the effort required of the user to think of instructions can be reduced, while instructions that are in line with the user's intentions can be given.

[0231] [Embodiment 7] In Embodiment 6, when there was only one type of notation indicating instructions to the generating AI, the instructions to the generating AI were executed. On the other hand, in Embodiment 7, when there are multiple types of notations indicating instructions to the generating AI, the instructions to the generating AI are switched for each type. Embodiment 7 will be explained mainly by replacing Figure 14 with Figure 15 and Figure 13 with Figure 16 in Embodiment 6. In Embodiment 7, all other points are the same as in Embodiment 6, so the explanation will be omitted.

[0232] <Instructions for generating AI> Figure 16 is a flowchart illustrating the flow from giving instructions to the generation AI server 102 to outputting its response in Embodiment 7. Of the processes shown in the flowchart of Figure 16, those with the same reference numerals as those in the flowchart of Figure 13 are basically the same as in Embodiment 6, so their explanation is omitted. However, for processes with the same reference numerals as those in the flowchart of Figure 13 that differ from those in Embodiment 6, only the differences will be explained.

[0233] In S402, the CPU 261 obtains the instruction text for the generating AI, which has been received by the user via the information processing device 101. At this time, the CPU 261 obtains one instruction text for each type of notation indicating an instruction to the generating AI. This will be explained using Figure 15(a). Figure 15(a) is an example of a screen for the user to set and confirm the content of the instruction.

[0234] In this embodiment, there are two instruction statements (text 603 and text 703). Text 603 is an instruction statement to the generating AI entered by the user using cursor 612 for text 501 in document image 500 (text within the closed region of list box 601). Text 703 is an instruction statement to the generating AI entered by the user using cursor 612 for texts 722-723 in document image 500 (text in the marker portion of list box 602). Figure 15 will be explained in detail in the chapter "User Interface Regarding Instructions to the Generating AI".

[0235] In S1601, the CPU 261 executes the processes from S1305 to S1602 for each instruction statement obtained in S402. For example, in the example in Figure 15(a), as explained in S402, there are two instruction statements (text 603 and text 703), so the CPU 261 executes the processes from S1305 to S1602 for each of text 603 and text 703. In other words, the CPU 261 executes the processes from S1305 to S1602 twice.

[0236] In S1602, the CPU 261 obtains the response from the generating AI server 102 to the instructions entered in S1309.

[0237] In S1603, the CPU 261 determines whether it has finished processing all the instructions obtained in S1601, and repeats the processing from S1305 to S1603 until it is determined that processing has finished for all the instructions.

[0238] In S1604, the CPU 261 presents the user with the answers obtained in S1602 for each instruction. An example of how the answers are presented to the user is explained using Figure 15(b). Figure 15(b) is an example of a screen in which the user checks the results of giving instructions to the generation AI. As the first answer, the CPU 261 displays the answer result 521 on screen 720 along with the instruction 603 entered by the user and the image 1430 entered into the generation AI.

[0239] Furthermore, the CPU 261 displays the response results 742-743 on screen 720 as the second response, along with the instruction text 703 entered by the user and the image 1530 entered into the generation AI. At this time, the response results 521 and 742-743 may be modified to change them into a format that is easy for the user to understand.

[0240] Furthermore, the CPU 261 may display instructions that have been processed from instructions 603 and 703 and input into the generation AI, instead of instructions 603 and 703, or it may display only the answer results 521 and 742-743. In this embodiment, the results are displayed on the answer result screen, but they may also be output as metadata or comments for the document image, or they may be output on paper or other media from the information processing device 101.

[0241] In this embodiment, CPU 261 executes S1311 to S1602 for each instruction statement, but it may also execute them all at once. Conversely, CPU 261 may execute S1604 for each instruction statement or instruction location.

[0242] <User interface for giving instructions to the generating AI> Figure 15 shows an example of a screen on the display device 210 of the information processing system 100 in Embodiment 7, which is used by the user for operation and confirmation. The processing of items with the same reference numerals as in Figure 14 in Figure 15 is the same as in Embodiment 6, and therefore its explanation is omitted.

[0243] First, let's explain how to determine the location of the instructions for the generating AI. Figure 15(a) is an example of a screen in Embodiment 7 for the user to set and confirm the content of the instructions. Screen 700 has a list box and a pull-down button for selecting the type of notation that indicates instructions for the generating AI, and an input field for the instruction text, for each of the two types of notations that indicate instructions for the generating AI within the document image 500.

[0244] The first location is the same as in Embodiment 6, so the explanation will be omitted, and only the second location will be explained below. All types of shapes to be identified in S404 are displayed in list boxes 601-602 and 604-605 as options for notation indicating instructions to the generating AI.

[0245] In the first instance, the closed region of list box 601 is selected by the user as the notation indicating instructions to the generating AI. Therefore, in the second instance, for the selection of notation indicating instructions to the generating AI, the information processing system 100 presents the user with list boxes 702, 704, and 705, which have the closed region removed. When the information processing system 100 detects that the user has pressed the pull-down button 711, it displays list boxes 704 to 705, which are the unselected options. As explained in Embodiment 6, other formats such as radio buttons are also acceptable instead of list boxes, as long as they can perform the selection function.

[0246] Furthermore, as explained in S402, the information processing system 100 receives instruction text 703 for the generating AI from the user. The method for inputting the instruction text is the same as in Embodiment 6, so the explanation is omitted.

[0247] Furthermore, when the information processing system 100 detects that the user has pressed button 713, it adds new input fields for notation and instruction text indicating instructions to the generating AI. The maximum number of fields that can be added is equal to the number of shapes to be identified in S404.

[0248] Finally, when the information processing system 100 detects that the user has pressed button 714, it determines the instruction location and instruction text for all notations indicating instructions to the generating AI. In this embodiment, since the first notation indicating an instruction to the generating AI is a closed area of ​​list box 601, the first instruction location is determined as the text area 501 enclosed by the line 510, and the first instruction text is determined as text 603.

[0249] Furthermore, since the notation indicating the second instruction to the generating AI is the list box 701, the second instruction location is confirmed as the text area 722-723 of the marker area 512-513, and the second instruction text is confirmed as text 703. Subsequently, as explained in S1308, the information processing system 100 converts the confirmed instruction location and instruction text into images in which the instruction location to the generating AI is clearly identifiable.

[0250] Next, we will explain how to check the response from the generating AI to the instructions. Figure 15(b) is an example of a screen in Embodiment 7 where the user checks the results of giving instructions to the generating AI. Screen 720 is an additional screen to screen 620 in Figure 14(b), and includes an image 1530 used for the second instruction to the generating AI, the second instruction text 703 to the generating AI, and the response results 742-743 from the generating AI to the second instruction.

[0251] As explained in S1604, the information processing system 100 presents the user with the response from the generating AI for each notation indicating an instruction to the generating AI. At this time, the information processing system 100 presents the images (1430, 1530), instruction texts (603, 703), and responses from the generating AI (521, 742~743) used to give instructions to the generating AI as a single unit for each notation indicating an instruction to the generating AI.

[0252] Furthermore, instead of presenting the user with the response from the generating AI for each notation indicating an instruction to the generating AI, the information processing system 100 may adopt a method of interactively switching the display. For example, the information processing system 100 may use a document image 500 to detect the user's selection of a border line 510 or marker areas 512-513, and then display instructions and responses to the generating AI corresponding to the handwritten drawing areas that represent the detected areas. In this case, the information processing system 100 may also explicitly indicate the notation indicating the instruction to the generating AI in text.

[0253] As described above, according to the present invention, even when there are multiple types of notations indicating instructions to the generating AI, the content of the instructions to the generating AI can be switched according to the type of notation, thereby enabling instructions that match the user's intentions.

[0254] [Embodiment 8] In Embodiment 6, when a user of the information processing system 100 sets the instruction content, all types of shapes to be identified in S404 of Figure 13 are presented as options for the instruction location to be given to the generating AI. On the other hand, in Embodiment 8, only the types of shapes contained in the document image acquired in S401 of Figure 13 are presented as options. Embodiment 8 will be explained by replacing Figure 14(a) in Embodiment 6 with Figure 17. In Embodiment 8, all other points are the same as in Embodiment 6, so the explanation will be omitted.

[0255] <User interface for giving instructions to the generating AI> Figure 17 shows an example screen on the display device 210 of the information processing system 100 in Embodiment 8, which allows the user to set and confirm instructions. Of the processes in Figure 17 that have the same reference numerals as those in Figure 14(a) are the same as in Embodiment 6, so their explanation is omitted.

[0256] Screen 1700 consists of a document image 500 entered by the user, list boxes 601-602 for selecting the instruction location, a pull-down button 611, an instruction text 603, a cursor 612 for entering the instruction text, and a "Settings" button 613 for confirming the instruction content.

[0257] As explained in S404 of Figure 13, for the document image 500, the border line 510 is assigned to the closed region cluster of list box 601, and the marker areas 512 and 513 are assigned to the marker cluster of list box 602. In other words, only two clusters from the clusters prepared in advance by the engineers or users are assigned to the document image 500.

[0258] Therefore, the information processing system 100 presents the closed region of list box 601 and the marker portion of list box 602 to the user of the information processing system 100 as options for notation indicating the location of instructions to be given to the generating AI.

[0259] In this embodiment, when showing options for notation indicating the location of instructions to the generating AI, these options are displayed as text. However, they may also be displayed as images of handwritten drawings representing the area. For example, the information processing system 100 may display a reduced and conceptualized image of the border line 510 instead of the text "closed area (enclosure, etc.)" in the list box 601.

[0260] Furthermore, in this embodiment, the user of the information processing system 100 indicates the location to be instructed to the generating AI by selecting and pressing the list boxes 601 to 602. However, for example, the user of the information processing system 100 may also indicate the location to be instructed to the generating AI by selecting and pressing the handwritten drawing areas (outline 510, marker areas 512 to 513) that represent regions on the document image 500 on the screen 1700.

[0261] When the information processing system 100 detects a selection or press of a handwritten drawing area representing a region by the user, it reflects the selection result on list boxes 601 to 602. That is, for example, when it detects a press of the border line 510, the information processing system 100 selects list box 601.

[0262] Furthermore, when the information processing system 100 detects that a user has selected or pressed a handwritten drawing area representing a region, it may highlight the selected or pressed drawing area using a highlighter or similar method.

[0263] As described above, according to Embodiment 8 of the present invention, the information processing system 100 can present to the user only the portions contained within the received document image as options for the location to be instructed to the generating AI. This reduces the effort required for the user to select a specified location.

[0264] [Embodiment 9] In Embodiment 9, the user of the information processing system 100 had to input or specify the instructions for the generating AI in some form. On the other hand, in Embodiment 9, if there is a handwritten comment near the instruction location for the generating AI in the document image, that comment is reflected in the instructions. Embodiment 9 will be explained by replacing Figure 14(a) in Embodiment 6 with Figure 18, and the document image 500 in Figure 5(a) with the document image 1010 in Figure 18. In Embodiment 9, all other points are the same as in Embodiment 6, so the explanation will be omitted.

[0265] <User interface for giving instructions to the generating AI> Figure 18 shows an example of a screen on the display device 210 of the information processing system 100 in Embodiment 9, which allows the user to set and confirm instructions. Of the processes in Figure 18 that have the same reference numerals as those in Figure 14(a) are the same as in Embodiment 6, so their explanation is omitted.

[0266] Document image 1010 in Figure 18(a) is the same as document image 500 in Figure 5(a) with handwritten text 1003 added. Screen 1800 is the same as Figure 14(a) but with document image 500 replaced by document image 1010 and instruction text 603 replaced by instruction text 1013.

[0267] The information processing system 100 selects a notation indicating the location of the instruction to be given to the generating AI by the user when a button 601 is pressed. When the instruction location estimation button 1011 is pressed, it performs a search process for surrounding handwritten characters for the shape indicated in 601. Although not shown in the diagram, the search process can be performed as the process S404 in Figure 13.

[0268] As a concrete example, the system searches for the closest handwritten character among those surrounding the enclosed area 510 of the list box 601. The method for searching for surrounding handwritten characters is, for example, to calculate the distance from a point on the enclosed area 510 to handwritten characters other than the "handwritten drawing area representing the area," and select the one with the closest distance. In other words, in this embodiment, the information processing system 100 determines that the handwritten area 1003 is the corresponding area as a handwritten character surrounding the enclosed area 510 and presents it.

[0269] Subsequently, the information processing system 100 performs OCR on the handwritten area 1003 and obtains text 1013 as an OCR result. The information processing system 100 reflects the obtained OCR result text 1013 in the instruction field for the generation AI shown in Figure 18(b)1013. The reflected text 1013 can be modified by the user, as in Embodiment 6. If no corresponding handwritten area is found nearby, the information processing system 100 operates as in Embodiment 6.

[0270] Further, the information processing system 100 may present, for example, all those that are closer than the distances from other "handwritten description locations representing regions" or within a specific threshold as surrounding handwritten characters. The information processing system 100 may present, for example, all handwritten characters other than the "handwritten description locations representing regions" in the order of proximity to the enclosure line 510 regardless of whether they are surrounding or not.

[0271] Also, in this embodiment, OCR is performed after searching for the closest handwritten character, but the timing of performing OCR is not limited to this. Further, when searching for the closest handwritten character, the information processing system 100 may narrow down in advance to the handwritten locations where handwritten characters other than symbols are drawn.

[0272] As described above, according to Embodiment 9 of the present invention, the handwritten comment near the instruction location to the generation AI can be presented to the user as an instruction sentence. Thereby, the labor for the user to input and modify the instruction sentence can be reduced.

[0273] <Other Examples> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Description of Reference Numerals

[0274] 100 Information processing system 101 Information processing device 102 Generation AI server 103 Information processing server

Claims

1. In an information processing device, The process involves acquiring a document image containing multiple handwritten areas indicating regions, and instructions entered by the user. A process of identifying the parts of the multiple handwritten sections that are to be instructed to be processed by the generating AI, A conversion step is provided to convert the instruction text obtained in the acquisition step into an instruction text that allows the AI ​​to identify the instruction location. Output step: Outputting the instruction text converted in the above conversion step and the document image to the generating AI, A program characterized by causing the execution of a specific action.

2. The program according to claim 1, characterized in that the handwritten area indicating the aforementioned region includes any of the following: an area enclosed by a line, an area enclosed by parentheses, an area underlined, or an area marked with a marker.

3. The aforementioned information processing device further includes: A discrimination step for determining the type of multiple handwritten portions, The process involves receiving a selection from the user for the type of handwritten area to be designated as the specified area, and then executing the following: The program according to claim 1, characterized in that the specified step identifies the handwritten portion that corresponds to the type of handwritten portion received in the reception step as the indicated portion.

4. In the aforementioned specific step, multiple locations are identified as indicated. The program according to claim 1, characterized in that, in the conversion step, the instruction sentence obtained in the acquisition step is converted into an instruction sentence that allows the generating AI to identify each of the instruction locations.

5. The program according to claim 1, characterized in that it searches for the handwritten string closest to the indicated location and includes the OCR result for the handwritten string in the instruction.

6. The program according to claim 1, characterized in that the document image is obtained by scanning the original document.

7. The program according to claim 1, characterized in that the OCR result for the document image is included in the instruction statement.

8. The process involves acquiring a document image containing multiple handwritten areas indicating regions, and instructions entered by the user. A process of identifying the parts of the multiple handwritten sections that are to be instructed to be processed by the generating AI, A conversion step is provided to convert the instruction text obtained in the acquisition step into an instruction text that allows the AI ​​to identify the instruction location. Output step: Outputting the instruction text converted in the above conversion step and the document image to the generating AI, An information processing method characterized by having the following features.

9. The information processing method according to claim 8, characterized in that the handwritten area indicating the aforementioned region includes any of the following: an area enclosed by a line, an area enclosed by parentheses, an area underlined, or an area marked with a marker.

10. The aforementioned information processing device further includes: A discrimination step for determining the type of multiple handwritten portions, The process involves receiving a selection from the user for the type of handwritten area to be designated as the specified area, and then executing the following: The information processing method according to claim 8, characterized in that the specified step identifies the handwritten portion that corresponds to the type of handwritten portion received in the reception step as the indicated portion.

11. In the aforementioned specific step, multiple locations are identified as indicated. The information processing method according to claim 8, characterized in that, in the conversion step, the instruction sentence obtained in the acquisition step is converted into an instruction sentence that allows the generating AI to identify each of the instruction locations.

12. The information processing method according to claim 8, characterized in that it searches for the handwritten string closest to the indicated location and includes the OCR result for the handwritten string in the instruction text.

13. The information processing method according to claim 8, characterized in that the document image is obtained by scanning the original document.

14. The information processing method according to claim 8, characterized in that the OCR result for the document image is included in the instruction statement.

15. A means for acquiring a document image containing multiple handwritten areas indicating regions, and instructions entered by the user. A means for identifying the portion of the multiple handwritten portions that should be instructed to be processed by the generating AI, A conversion means that converts the instruction text acquired by the acquisition means into an instruction text that the generation AI can identify the instruction location of, Output means for outputting the instruction text converted by the conversion means and the document image to the generating AI, An information processing device characterized by having the following features.

Images