Electronic device, method, program, and storage medium storing program

Abstract

To provide an electronic device for suppressing a communication load in a configuration of outputting using a processing result received from the outside, method, program, and storage medium.SOLUTION: An OCR function of an electronic device includes: executing character recognition processing to acquire a text from character blocks 502 and 503 on an image; defining the text acquired by the character recognition processing as a processing target text and transmitting the processing target text and a processing instruction for requesting processing on the text to a generative AI; receiving a processing result based on the processing instruction; and using the text included in the received processing result and outputting a document as a result of character recognition of the image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an electronic device capable of performing an automatic character recognition function, a method for the electronic device, a program, and a storage medium storing the program. [Background technology]

[0002] An automatic character recognition system is known that automatically recognizes characters written in an image and converts them into text. However, the recognition by the automatic character recognition system can be erroneous, and Patent Document 1 describes a system for detecting typos and omissions using machine learning.

[0003] On the other hand, generative artificial intelligence (generative AI) is known. Generative AI is a type of machine learning model that uses learned data to generate new data. For example, ChatGPT (Chat Generative Pre-Trained Transformer), a chat generation AI that can converse in chat format, is widely used. ChatOCR (Chat Optical Character Recognition), an automatic character recognition system, is a plugin that provides additional functionality to ChatGPT. When using ChatOCR, image files are sent directly to the plugin. Using ChatOCR, you can extract text from PDF files and image files, as well as ask questions about the extracted text. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-145023 Summary of the Invention [Problem to be solved by the invention]

[0005] In a configuration that uses processing results output from an external source, an unexpected processing result may be output from the outside depending on the input contents used to obtain the processing result. Therefore, a method is required to prevent the use of unexpected processing results output from the outside.

[0006] An object of the present invention is to provide a mechanism that can prevent unexpected processing results output from outside from being used. [Means for solving the problem]

[0007] In order to solve the above problem, the electronic device of the present invention is characterized by having an input control means for controlling the input of a text to be processed and a processing instruction into a processing means, a determination means for comparing the text to be processed input into the processing means with a first text output from the processing means and determining whether a predetermined condition is met, and a control means for controlling the output of the first text if the predetermined condition is met by the determination means as a result of the processing instruction, or to output a second text different from the first text if the predetermined condition is not met. [Effects of the Invention]

[0008] According to the present invention, it is possible to prevent the use of unexpected processing results output from outside. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a text processing system. [Figure 2] FIG. 1 is a diagram illustrating the configuration of a PC. [Figure 3] FIG. 1 illustrates a configuration of a printing device. [Figure 4] 10 is a flowchart showing a process for outputting the execution result of the OCR function. [Figure 5] FIG. 5 is a diagram showing the flow of the process shown in FIG. 4. [Figure 6] FIG. 1 illustrates a query. [Figure 7]FIG. 10 is a diagram showing a confirmation screen. [Figure 8] 10 is a flowchart showing the process of S406. [Figure 9] FIG. 10 is a diagram showing a notification screen. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] FIG. 1 illustrates an example of the configuration of a text processing system according to this embodiment. In the text processing system 100, a PC 105 and a printing device 101 are connected to each other via a wireless LAN 102 so that they can communicate with each other. Furthermore, the PC 105 and the printing device 101 can access the Internet 104 via an access point 103. This configuration allows the PC 105 and the printing device 101 to communicate with a generative AI (Artificial Intelligence) server 106 connected to the Internet 104 and providing generative AI services. Generative AI is a machine learning model constructed using deep learning. Generative AI services can output more creative results for images, text, video, audio, and the like than conventional services. One example of a generative AI service is ChatGPT, which uses a text generation language model to realize a conversational AI service that enables human-like conversations. Note that the generative AI server 106 may be a standalone server device or a set of multiple servers working together, as long as it is an external system using generative AI.

[0012] The PC 105 is an information processing device having a communication function such as a wireless LAN or a wired LAN. A wireless LAN is sometimes called a WLAN (Wireless LAN). The PC 105 may be, for example, a smartphone, a notebook PC, a tablet terminal, or a PDA (Personal Digital Assistant). The PC 105 can communicate with the printing device 101 via a wireless LAN 102. For example, the PC 105 can instruct the PC 101 to execute a printing function or a scanning function via the wireless LAN 102. The PC 105 and the printing device 101 may be directly connected without going through the access point 103. That is, the PC 105 can communicate with the printing device 101 via a direct connection. A wired network may be used as the network between the PC 105, the printing device 101, and the access point 103, or as part of that network.

[0013] The printing device 101 is an example of a printing device having a printing function. The printing device 101 may be configured as a multifunctional printer (MFP) having a reading function (scanner), a fax function, and a telephone function. The printing device 101 also has a communication function that enables wireless communication with the PC 105. In this embodiment, the printing device 101 is described as an example, but devices of different types may be used. For example, devices having communication functions may include a facsimile machine, a scanner, a projector, a mobile terminal, a smartphone, a laptop PC, a tablet terminal, a PDA, a digital camera, a music playback device, a television, a smart speaker, and AR (Augmented Reality) glasses. The printing device 101 receives print data including image data from the PC 105 connected via the access point 103, for example, and forms an image based on the data. Alternatively, the printing device 101 transmits image data read by the scanner function to the PC 105 connected via the access point 103, for example. Other control information can also be communicated to the connected network via the access point 103 .

[0014] The access point 103 is a communication device that is provided separately (externally) from the PC 105 and the printing device 101 and operates as a WLAN base station device. The access point 103 may also be referred to as an external access point 103 or an external wireless base station. A communication device with a WLAN communication function can communicate in WLAN infrastructure mode via the access point 103. In this embodiment, the PC 105 and the printing device 101 are also examples of communication devices. The wireless infrastructure mode may also be referred to as a "wireless infrastructure mode." In other words, the wireless infrastructure mode is a mode in which, for example, the printing device 101 communicates with the PC 105 via the access point 103 to which the printing device 101 is connected. The access point 103 communicates with communication devices that have been authorized (authenticated) to connect to the access point 103 and relays wireless communications between the communication devices and other communication devices. The access point 103 is also connected to a wired LAN communication network and relays communications between communication devices connected to the network and other communication devices wirelessly connected to the access point 103. Furthermore, if the authentication method of the network formed by the access point 103 is a method that uses an authentication server (if the access point 103 supports an authentication method that uses an authentication server), the access point 103 cooperates with the authentication server (not shown) to authenticate communication devices that connect to the network, thereby performing access control. Note that the access point 103 may also support an authentication method that does not use an authentication server.

[0015] The PC 105 and the printing device 101 can use their respective WLAN communication functions to perform wireless communication in wireless infrastructure mode via an external access point 103 or in peer-to-peer mode without the external access point 103. Note that peer-to-peer mode is sometimes called "P2P mode" or "wireless direct mode" in contrast to wireless infrastructure mode. In other words, P2P mode is a mode in which the printing device 101 communicates directly with the PC 105 without the access point 103. P2P mode includes Wi-Fi Direct (registered trademark) mode and software access point (soft AP) mode. Note that Wi-Fi Direct (registered trademark) is sometimes referred to as WFD. In other words, Wireless Direct mode can be considered a communication mode compliant with the IEEE 802.11 series.

[0016] FIG. 2 is a diagram illustrating an example of the configuration of the PC 105. The PC 105 includes a main board 220 that controls the entire device, a wireless communication unit 213 that performs WLAN communication, a display unit 212, an operation unit 211, and a short-range wireless communication unit 214 that performs wireless communication different from that of the wireless communication unit 213. The main board 220 includes, for example, a CPU 201, a ROM 202, a RAM 203, an image memory 204, a data conversion unit 205, a camera unit 206, a nonvolatile memory 207, a data storage unit 208, a speaker unit 209, and a power supply unit 210. The functional units within the main board 220 are interconnected via a system bus 215. The main board 220 and the wireless communication unit 213, and the main board 220 and the short-range wireless communication unit 214 are connected, for example, via dedicated buses. The main board 220 and the display unit 212, and the main board 220 and the operation unit 211 are connected, for example, via dedicated buses.

[0017] The CPU 201 is a system control unit that controls the entire PC 105. The operation of the PC 105 described in this embodiment is realized, for example, by the CPU 201 reading a program stored in the ROM 202 into the RAM 203 and executing it. Dedicated hardware for each process may also be provided. The ROM 202 stores control programs, such as a control program executed by the CPU 201 and an embedded operating system (OS) program. The CPU 201 executes each control program stored in the ROM 202 under the management of the embedded OS stored in the ROM 202, thereby performing software control such as scheduling and task switching. The ROM 202 also stores an application (printing app) that generates information interpretable by the printing device 101. The information interpretable by the printing device 101 corresponds to functions that the printing device 101 can execute. The application can set up printing, scanning, and the like, and instruct the printing device 101 to execute each function. The RAM 203 is composed of a static RAM (SRAM) or the like. The RAM 203 stores data such as program control variables, setting values registered by the user, and management data for the PC 105. The RAM 203 may also be used as a buffer for various types of work. The image memory 204 is configured with a memory such as a DRAM (Dynamic RAM). The image memory 204 temporarily stores image data received via the wireless communication unit 213 and image data read from the data storage unit 208 for processing by the CPU 201. The non-volatile memory 207 is configured with a memory such as a flash memory, and continues to store data even when the PC 105 is powered off. Note that the memory configuration of the PC 105 is not limited to the above-described configuration. For example, the image memory 204 and the RAM 203 may be shared, or data may be backed up using the data storage unit 208. Although a DRAM is given as an example of the image memory 204, other storage media such as a hard disk or non-volatile memory may also be used.

[0018] The data conversion unit 205 analyzes data in various formats and performs data conversion such as color conversion and image conversion. The camera unit 206 has the function of electronically recording and encoding images input through a lens. Image data obtained by capturing images with the camera unit 206 is stored in the data storage unit 208. The speaker unit 209 performs control to realize the function of inputting or outputting audio. The power supply unit 210 is, for example, a portable battery, and controls the power supply to the device. The display unit 212 electronically controls the display content and performs control to display various input contents, the operating status and status of the PC 105, etc. The operation unit 211 performs control such as generating an electrical signal corresponding to the operation upon receiving a user operation and outputting the signal to the CPU 201.

[0019] The PC 105 performs wireless communication using the wireless communication unit 213, and performs data communication with other communication devices such as the printing device 101. The wireless communication unit 213 converts data into packets and transmits the packets to the other communication devices. The wireless communication unit 213 also restores packets from other external communication devices to the original data and outputs the data to the CPU 201. The wireless communication unit 213 is a unit for realizing communication in accordance with standards such as WLAN. The short-range wireless communication unit 214 communicates using a communication method different from that used by the wireless communication unit 213, such as Bluetooth (registered trademark). The configurations of the PC 105 and the main board 220 are not limited to those described above. For example, each function of the main board 220 implemented by the CPU 201 may be implemented by a processing circuit such as an ASIC (application-specific integrated circuit), and may be implemented by either hardware or software.

[0020] 3 is a block diagram showing an example of the configuration of the printing device 101. The printing device 101 has a main board 320 that controls the entire device, a USB communication unit 307, a wireless communication unit 309, a wired communication unit 310, an operation display unit 312, a power button 313, a print unit 315, and a scan unit 317.

[0021] The main board 320 is provided with a microprocessor-type CPU 301. The CPU 301 controls the printing device 101 in accordance with a control program stored in a ROM-type program memory 302 connected via an internal bus 318 and the contents stored in a RAM-type data memory 303. The operation of the printing device 101 described in this embodiment is realized, for example, by the CPU 301 reading the program stored in the program memory 302 into the data memory 303 and executing it. The CPU 301 controls the scan control unit 316 to cause the scan unit 317 to optically scan an original and store the scanned data in the image memory in the data memory 303. The scan control unit 316 is an interface for connecting the scan unit 317 to the main board 320 and performs operations such as converting the format of scanned image data. The CPU 301 controls the print control unit 314 to cause the print unit 315 to print an image of the scanned data stored in the image memory in the data memory 303 onto a recording medium (copy function). The print control unit 314 is an interface for connecting the print unit 315 and the main board 320, and performs image data conversion, etc. The data conversion unit 304 performs data conversion such as analyzing data of various formats, color conversion, and converting image data into print data. The encoding / decoding processing unit 305 performs encoding and decoding processes and scaling processes for image data (JPEG, PNG, etc.) handled by the printing device 101.

[0022] The CPU 301 controls a USB communication unit 307 via a USB communication control unit 306 to perform USB communication with an external PC 105 via a USB connection. The CPU 301 controls an operation control unit 311 to receive operation information from a power button 313 and an operation display unit 312. The CPU 301 controls the operation control unit 311 to display, for example, the status of the printing device 101 and a function selection menu on the operation display unit 312. The CPU 301 controls a wireless communication unit 309 and a wired communication unit 310 via a communication control unit 308 in accordance with the operation information received by the operation display unit 312. For example, the CPU 301 changes the settings of the communication method and sets up a connection to a network in accordance with the operation information.

[0023] The wireless communication unit 309 is a unit capable of providing WLAN communication functions. That is, the wireless communication unit 309 converts data into packets in accordance with WLAN standards and transmits the packets to other communication devices. The wireless communication unit 309 also restores packets from other external communication devices to the original data and outputs the data to the CPU 301. The wireless communication unit 309 is configured to be able to perform data (packet) communication in a WLAN system compliant with, for example, the IEEE 802.11 standard series (IEEE 802.11a / b / g / n / ac / ax, etc.). However, this configuration is not limited, and the wireless communication unit 309 may be capable of performing communication in a WLAN system compliant with other standards. The wireless communication unit 309 is also capable of performing communication in WFD mode, P2P mode, wireless infrastructure mode, etc. Note that the PC 105 and the printing device 101 are capable of wireless communication based on the WFD mode, and the wireless communication unit 309 has a soft AP function or a group owner function. That is, the wireless communication unit 309 can establish a communication network in P2P mode and determine the channel to be used for communication in P2P mode.

[0024] The wired communication unit 310 is a unit for performing wired communication. The wired communication unit 310 is capable of data (packet) communication in a wired LAN (Ethernet) system conforming to, for example, the IEEE802.3 series. Furthermore, in wired communication using the wired communication unit 310, communication in a wired communication mode is possible. The wired communication unit 310 is connected to the main board 320 via a bus cable or the like.

[0025] The printing device 101 has an OCR (Optical Character Recognition) function that recognizes character information on an image read by a scanning function and extracts text data. The OCR function may be implemented by, for example, the scan control unit 316. The OCR function creates data in a text-searchable file format. Examples of such data include PDF and XPS (XML Paper Specification). The printing device 101 can save (or send) files created by the OCR function to a specified internal or external storage destination. Instructions to set and execute such an OCR function may be given on an operation panel on the printing device 101, or may be given to the printing device 101 from an application installed on the PC 105. The printing device 101 is not limited to the configuration shown in FIG. 3 and may have a configuration appropriate for the functions that the device used as the printing device 101 can implement.

[0026] However, the results of automatic character recognition processing using an OCR function may contain errors, such as typos. In this embodiment, a configuration is assumed in which errors in the results of automatic character recognition processing using the OCR function are corrected using a generative AI service. To achieve this, for example, a configuration is considered in which ChatOCR, a plug-in for ChatGPT that provides a sentence generation language model, is used to send image files generated by the OCR function to the generative AI server 106 that provides the generative AI service. However, in such a configuration, when the OCR function is executed on a large number of image files, it is expected that the communication volume with the server will increase and communication speed will decrease. Therefore, in this embodiment, a configuration is described that enables the communication load with the generative AI server 106 to be reduced.

[0027] 4 is a flowchart showing the process of outputting the execution results of the OCR function in this embodiment. The process of FIG. 4 is realized, for example, by the CPU 301 reading a program stored in the computer-readable program memory 302 into the data memory 303 and executing it. The process of this flowchart begins when the printing device 101 starts up, a home screen is displayed on the operation display unit 312, and the user can select a desired function. For example, menu buttons corresponding to the scan function, copy function, etc. are displayed on the home screen so that the user can select them.

[0028] In S400, the CPU 301 determines the function for which selection has been accepted via the operation display unit 312. If the accepted function is a scan function, the process proceeds to S401; if the accepted function is not a scan function, the process in FIG. 4 ends and a process corresponding to the function is carried out. A description of this process will be omitted. In this embodiment, the explanation will be given assuming that selection of the scan function has been accepted via the operation display unit 312.

[0029] In S401, the CPU 301 determines whether a scan start instruction has been received with the post-scan file format specified as "PDF (OCR)." That is, in S401, it determines whether a character-recognized PDF file will be created. In other words, the determination process in S401 is a process for determining whether an image file will be created by executing a character recognition function. If a scan start instruction has been received with "PDF (OCR)" specified, the process proceeds to S402. If a scan start instruction has been received with another file format specified, the process in FIG. 4 ends, and the process corresponding to the function is performed. A description of this process will be omitted.

[0030] In S402, the CPU 301 controls the scanning unit 305 to scan the document placed on the document platen (not shown) and acquires the read data. The CPU 305 stores the read data in the data memory 303.

[0031] In S403, the CPU 301 executes a process of extracting character chunks from the read data stored in the data memory 303. Here, a character chunk is a block where characters are densely packed together.

[0032] FIG. 5 is a diagram showing the processing flow of FIG. 4. Data 501 in FIG. 5 shows an example of the execution result of the processing of S403, and character block 502 "aifu reo" and character block 503 "yakikugeko" are examples of the character blocks. Character blocks 502 and 503 correspond to areas segmented on the image represented by data 501 that are the target of automatic character recognition processing. Character block 502 "aifu reo" is an example of an incorrect result obtained by automatic character recognition, even though it is intended to represent "aiueo" from the "a" row of the Japanese syllabary. Character block 503 "yakikugeko" is an example of an incorrect result obtained by automatic character recognition, even though it is intended to represent "kakikukeko" from the "ka" row of the Japanese syllabary. Character blocks are extracted by grouping. The extracted areas correspond to the segmented areas described above. Then, CPU 301 executes the OCR function on each extracted character block and stores the execution result (processing target text) in data memory 303. Note that in S403, character blocks may be extracted after the OCR function is executed. Subsequent processing is performed by focusing on one of the character blocks extracted in S403.

[0033] In S404, the CPU 301 creates a query for input to the generative AI by concatenating the input "Please correct the typo in the following sentence" with the result of the automatic character recognition, and transmits it to the generative AI server 106 using the wireless communication unit 309. In other words, the query is an example of a processing instruction requesting processing by the generative AI server 106. In other words, transmission to the generative AI server 106 can also be said to be an input control process for the generative AI.

[0034] Figure 6 shows an example of a query created for each character block. The query created for each character block is written in the development code of a generative AI whose library is publicly available. Figure 6 shows an example written in Python (registered trademark).

[0035] The first line in Figure 6 indicates that a package related to generative AI tools will be loaded and made available for use. Although it is written as package on the diagram, the package name of the open source code, etc. will be written.

[0036] The second line in Figure 6 indicates that you need to enter the key code of the account you obtained to use the API of the generative AI tool. The key code can be any string of numbers and letters, and "YOUR_API_KEY" in Figure 6 is an example.

[0037] The third line in Figure 6 specifies the name of the variable that stores the return value of the create function in the ChatCompletion group. The name does not have to be fixed as "response" and can be anything. This allows you to call the API consecutively. For example, by naming the variables in order as "response1" and "response2," you can call the API consecutively for the extracted character blocks, as in queries 504 and 505 in Figure 5.

[0038] Lines 4 to 7 in Figure 6 correspond to the above query. Line 4 in Figure 6 specifies the identifier of the trained model. Line 5 in Figure 6 shows the content to be sent to the chat. Line 6 in Figure 6 shows that system is specified as the role. In Content, you can specify what function you want to perform; in this example, it specifies correcting typos in text. Line 7 in Figure 6 shows that user is specified as the role. Content stores the results of the OCR function.

[0039] Note that the seventh line of FIG. 6 shows an example of a sentence that is different from the example of FIG. 5. That is, FIG. 6 shows an example of an intended sentence, "The weather forecast is for sunny tomorrow. You don't need an umbrella.", but which is incorrectly output by automatic character recognition: "The weather forecast is for sunny tomorrow. You don't need an umbrella." Furthermore, in this embodiment, the seventh line of FIG. 6 does not depend on the language, and may be input in English, for example, instead of Japanese. For example, to give an example of an English sentence that corresponds to the above Japanese example, the seventh line of FIG. 6 shows an intended sentence, "A weather forecast says that it is sunny tomorrow. An umbrella is not needed.", but which is incorrectly output by automatic character recognition: "A weather forecast says that it is sunny tomorrow. An umbrella is not needed."

[0040] Lines 8 and 9 in Figure 6 are closing parentheses. Line 10 in Figure 6 indicates that only the message content (i.e., the response to the query) of the information received from the generative AI is output. This content is used to preview the correction results and to reflect them in the file that stores the results of the OCR function (i.e., a PDF with text information).

[0041] In S405, the CPU 301 receives the correction result sent back from the generative AI server 106 using the wireless communication unit 309 and stores it in the data memory 303. Correction result 507 in Figure 5 shows an example of the correction result from the generative AI server 106 corresponding to the generative AI 506. In other words, it shows that "ai-fure-o" has been corrected to "ai-ue-o."

[0042] In S406, the CPU 301 performs processing to check whether the correction result is as expected. Details of the processing in S406 will be described later with reference to FIG.

[0043] In S407, the CPU 301 determines whether there are other character blocks extracted in S403. If it is determined that there are other character blocks, it focuses on the character blocks and repeats the processing from S403. If it is determined that there are no other character blocks, it proceeds to S408.

[0044] In S408, the CPU 301 lays out the correction results received in S405 at positions corresponding to the character blocks extracted in S403.

[0045] Data 508 in Figure 5 shows an example in which correction results are laid out at positions corresponding to the character blocks (character blocks before correction) corresponding to each query input to the generative AI 506. Character block 509 shows the correction result "aiueo" received in S405 after "ai fureo" was sent to the generative AI server 106 in S404. Character block 509 is also placed corresponding to the position of "ai fureo" in character block 502 read by scanning. Character block 510 shows the correction result "ka kiku ke ko" received in S405 after "yaki ku ge ko" was sent to the generative AI server 106 in S404. Character block 510 is also placed corresponding to the position of "yaki ku ge ko" in character block 503 read by scanning.

[0046] In S409, the CPU 301 displays the results of the layout in S408 on the operation display unit 312 as a confirmation screen.

[0047] 7 is a diagram showing an example of the confirmation screen displayed in S409. A confirmation screen 701 displays a correction result preview 702. The correction result preview 702 is created based on the data 508 in FIG.

[0048] In the above, "ai-fureo" and "yaki-kugeko" were described as examples of queries, but other examples are similar. For example, if "Tomorrow's weather is forecast to be sunny. You don't need an umbrella." is input as a query in S404, "Tomorrow's weather is forecast to be sunny. You don't need an umbrella." is received as the correction result in S405. In this case, the correction result "Tomorrow's weather is forecast to be sunny. You don't need an umbrella." is laid out in correspondence with the position on the manuscript of the character block "Tomorrow's weather is forecast to be sunny. You don't need an umbrella." The above is an example in Japanese, but the same applies in English. To give an English example corresponding to the above example, the correction result "A weather forecast says that it is sunny tomorrow. An umbrella is not needed." is laid out in correspondence with the position on the manuscript of "A weather forecast says that it is sunny tomorrow. An umbrella is not needed."

[0049] The confirmation screen 701 is a screen for allowing the user to confirm the results of corrections made by the generative AI server 106, and displays a button 703 for accepting user confirmation that the corrections have been made, and a button 704 for accepting instructions to make further corrections.

[0050] In S410, CPU 301 accepts a user operation on confirmation screen 701 via operation display unit 312. In S411, CPU 301 determines whether or not to make further corrections based on the accepted user operation. Specifically, for example, if pressing of button 704 is accepted, CPU 301 determines to make further corrections, and if pressing of button 703 is accepted, CPU 301 determines not to make further corrections. If it is determined to make further corrections, CPU 301 proceeds to S412, and if it is determined not to make further corrections, CPU 301 proceeds to S413.

[0051] In S412, after inputting "Please correct the typo in the following sentence," the CPU 301 concatenates the correction results received from the generative AI server 106 in S405 to create a query to input to the generative AI, and transmits this to the generative AI server 106 using the wireless communication unit 309. Then, the processing from S405 is repeated.

[0052] In S413, the CPU 301 executes processing to output, as a document, the layout result displayed as the confirmation screen 701 in S409. For example, the CPU 301 controls the print unit 315 to execute printing processing. Alternatively, the CPU 301 executes processing to transmit to the PC 105 using the wireless communication unit 309. Thereafter, the processing in FIG. 4 ends.

[0053] Next, the correction result confirmation process in S406 will be described with reference to FIG.

[0054] In S801, the CPU 301 initializes the number of times that the generative AI server 106 performs the process of determining the correction result. Specifically, for example, the number of times is cleared to zero. Note that the variable representing the number of times is secured in advance in the data memory 303.

[0055] In S802, the CPU 301 compares the number of characters in the character block before and after correction by the generative AI server 106. For example, the number of characters in character block 502 and the number of characters in character block 509 in FIG. 5 are compared. In S803, the CPU 301 determines whether the difference in the number of characters in the character block before and after correction by the generative AI server 106 is within an expected error range. Specifically, for example, the CPU 301 references the data memory 303 and determines whether the number of characters in the character block extracted in S403 is within an error of 10% of the number of characters in the character block that is the correction result received from the generative AI server 106 in S405, that is, whether the error is within 90% to 110%. If it is determined that the difference is within the expected error range, the process proceeds to S804; if it is determined that the difference is not within the expected error range, the process proceeds to S806.

[0056] In S804, the CPU 301 compares the character strings of the character chunk before and after correction by the generative AI server 106 to obtain the similarity. Specifically, for example, the CPU 301 refers to the data memory 303 and calculates the similarity between the character string of the character chunk extracted in S403 and the character string of the character chunk received from the generative AI server 106 in S405 by Gestalt pattern matching.

[0057] In S805, the CPU 301 determines whether the similarity obtained in S804 is equal to or greater than a threshold. Specifically, for example, the CPU 301 determines whether the similarity calculated by Gestalt pattern matching is equal to or greater than 0.8. If it is determined that the similarity is equal to or greater than the threshold, the process in FIG. 8 ends. If it is determined that the similarity is not equal to or greater than the threshold, i.e., is less than the threshold, the process proceeds to S806.

[0058] As described above, in this embodiment, after the correction result is returned from the generative AI server 106 in S405, the generative AI server 106 performs a process of determining the correction result in S803 and S805. In other words, this is a process of determining whether the processing result in the generative AI server 106 was performed as expected. In other words, this is a process of determining whether the generative AI server 106 corrected the typo as expected. In S803, the difference in the number of characters is used as the determination condition, and in S805, the similarity is used as the determination condition. The conditions for determining whether the typo correction was performed as expected are not limited to these, and other conditions may also be used. For example, the number of words, the number of subjects, etc. may be used as conditions.

[0059] In S806, the CPU 301 increments the number of times the correction result determination process is performed. In S807, the CPU 301 determines whether the number of times the correction result determination process is performed has reached a predetermined number (whether the process has been performed a predetermined number of times). Specifically, for example, the CPU 301 determines whether the number of times the correction result determination process is performed has reached 10. If it is determined that the predetermined number of times has been reached, the process proceeds to S808, and if it is determined that the predetermined number of times has not been reached, i.e., the predetermined number of times is less than the predetermined number, the process proceeds to S809.

[0060] In S808, the CPU 301 displays an error notification screen on the operation display unit 312 to notify that the correction by the generative AI server 106 was not performed correctly, i.e., that the generative AI server 106 failed to correct the typo, and then terminates the processing of Figures 8 and 4.

[0061] 9 is a diagram showing an example of the error notification screen displayed in S808. The error notification screen 900 displays a message indicating that the generative AI server 106 did not perform the correction correctly. An area 901 displays the character block extracted in S403 as the text before correction.

[0062] In S809, the CPU 301 creates an input query to the generative AI saying, "There is a defect in the current correction result. Please correct it correctly." and transmits it to the generative AI server 106 using the wireless communication unit 309. The CPU 301 then receives the correction result returned from the generative AI server 106 using the wireless communication unit 309, stores it in the data memory 303, and repeats the processing from S802. In this embodiment, the input query to the generative AI in S809 is different from the input queries to the generative AI in S404 and S412. That is, in S809, a processing instruction is resent to the generative AI server 106 using a query (correction processing instruction) obtained by correcting the query used in S404. This makes it possible to expect a result different from the correction result received in S405. Then, a re-determination process is performed in S803 and S805 using the result. Note that the above re-transmission process is, in other words, a re-input process to the generative AI server 106.

[0063] The effect obtained by executing the process in Figure 8 will be explained below. For example, assume that the sentence before the typo correction is an imperative sentence containing a typo, such as "Describe Halloween in 100 words." In this case, the generative AI receives the sentence before the typo correction as an instruction sentence, and outputs to the user the following sentence as the result of the conversion process other than the typo correction: "Halloween is a traditional festival celebrated every year on October 31st. This festival is popular mainly in the United States and Canada, but has spread all over the world."

[0064] For the above example, when the process shown in Figure 8 is executed, the conversion result of "Halloween is a traditional festival celebrated annually on October 31st. This festival is popular mainly in the United States and Canada, but is widespread worldwide." is determined to have a difference of 47 characters compared to the sentence before the typo correction. This is determined to be outside the margin of error, and a "No" result is given in S803. If the result in S803 were "Yes," a similarity comparison would be performed using Gestalt pattern matching in S805. Since the similarity between the conversion result and the sentence before conversion is 0.18, which is below the threshold, a "No" result is given in S805. As a result, the instruction sentence is changed without being output to the user, and the generative AI is instructed to perform conversion again in S809. This discards any conversion results that do not meet expectations without being output to the user, increasing the likelihood of outputting a more expected conversion result to the user.

[0065] In the present embodiment, it has been described that in S403, processing is performed to extract character blocks from the read data stored in the data memory 303. At this time, it may be determined whether the data volume of the read data is equal to or greater than a threshold value. Then, the processing of FIG. 4 may be controlled based on whether the data volume is equal to or greater than the threshold value. For example, if it is determined that the data volume is equal to or greater than the threshold value, the processing of FIG. 4 is executed as described in the present embodiment. On the other hand, if it is determined that the data volume is not equal to or greater than the threshold value, i.e., is less than the threshold value, in S404 and S412, transmission control may be performed to transmit an image file corresponding to the read data, rather than a character string such as that on line 7 of FIG. 6.

[0066] In this embodiment, the process of checking the typo correction results of the generative AI has been described. However, the operation of this embodiment can be applied not only to typo correction but also to summarization, anonymization, and translation processes.

[0067] Summarization processing is performed to extract key points from pre-conversion character strings. When the processing of FIG. 8 is applied to summarization processing, in S404, a query input to the generative AI is, for example, an instruction sentence such as "Please summarize the following sentence in 100 characters or less," followed by a pre-conversion character string of 500 characters. In this case, since the instruction sentence specifies a character limit of 100 characters, in S803, for example, it is determined whether the number of characters in the output result of the generative AI is within the range of 90 to 100 characters. Then, in S804, the CPU 301 extracts words from the output result of the generative AI, and in S805, it is determined whether the extracted words are included in the pre-conversion character string. If it is determined that the extracted words are included, the processing of FIG. 8 ends; if it is determined that the extracted words are not included, the processing proceeds to S806. The processing of the other steps is the same as the processing described in this embodiment. As a result, summary processing results that do not meet expectations are discarded without being output to the user, thereby increasing the possibility of outputting a more expected summary processing result.

[0068] The anonymization process is performed to prevent the output of personal information such as names and addresses. When the process of FIG. 8 is applied to the anonymization process, in S404, a query input to the generative AI is, for example, a prompt such as "Please anonymize the following sentence." followed by a pre-conversion string containing personal information. In this case, for example, the name included in the pre-conversion sentence is abstracted and replaced with a character string such as "Mr. A" that does not identify the individual, and the address is replaced with a notation up to the prefecture name. Since the anonymization process is expected to produce an output result that is similar to the input result, the number of characters is compared in S803 and the similarity is compared in S805 to determine whether the desired conversion process result has been obtained, similar to the typo correction in this embodiment. If it is determined that the desired conversion process result has been obtained, the process of FIG. 8 ends; if it is determined that the desired conversion process result has not been obtained, the process proceeds to S806. This discards any anonymization process results that do not meet expectations without being output to the user, thereby increasing the likelihood of producing a more desired anonymization process result.

[0069] The translation process is performed to replace a pre-conversion string with a language different from the pre-conversion string. When the process of FIG. 8 is applied to the translation process, in S404, a query input to the generative AI is, for example, an instruction sentence such as "Please translate the following sentence into English." followed by a pre-conversion string written in Japanese. In the translation process, while the number of characters in the post-conversion string is expected to differ from the number of characters in the pre-conversion string, the number of words is expected to remain the same. Therefore, the process of S803 is not performed. Instead, in S804, the CPU 301 extracts the number of words, such as nouns, verbs, and subjects, from the post-conversion string. Then, in S805, the number of words in the pre-conversion string and the number of words in the post-conversion string are compared to determine whether the difference is within a predetermined range, thereby determining whether the desired conversion process result has been obtained. If it is determined that the desired conversion process result has been obtained, the process of FIG. 8 ends. If it is determined that the desired conversion process result has not been obtained, the process proceeds to S806. This discards any translation process results that do not meet expectations without being output to the user, thereby increasing the likelihood of outputting a more expected translation process result.

[0070] The operation of this embodiment has been described as being performed by CPU 301, but the various controls described above may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0071] Furthermore, although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0072] Furthermore, in the above-described embodiment, the present invention has been described as being applied to the printing device 101, but the present invention is not limited to this example and can be applied to any electronic device that can execute character recognition functions. In other words, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, printers with displays, digital photo frames, electronic book readers, OCR cameras, and the like.

[0073] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0074] The disclosure of the present embodiment includes the following electronic device, method, program, and storage medium. (Item 1) an input control means for controlling input of the text to be processed and processing instructions to the processing means; a determination means for comparing the processing target text input to the processing means with a first text output from the processing means and determining whether a predetermined condition is satisfied; a control means for controlling the processing so that, when the predetermined condition is satisfied by the determination means as a result of the processing instruction, the first text is output, and when the predetermined condition is not satisfied, a second text different from the first text is output; An electronic device comprising: (Item 2) The electronic device described in item 1, characterized in that the processing means is provided in an external system, and the input control means controls the text to be processed and the processing instructions to be sent to the external system so that they are input to the processing means. (Item 3) 3. The electronic device according to item 1 or 2, wherein the processing means is a generative AI (Artificial Intelligence) system. (Item 4) The electronic device described in any one of items 1 to 3, characterized in that the specified condition is a condition for determining whether the first text is the result of processing based on the processing instruction being performed as expected. (Item 5) When it is determined that the predetermined condition is not satisfied, the input control means controls the processing means to input the text to be processed and a second processing instruction obtained by changing the processing instruction, 5. The electronic device according to any one of items 1 to 4, wherein the second text is text output from the processing means as a result of the second processing instruction. (Item 6) When it is determined that the predetermined condition is not satisfied, the input control means performs a re-input process to control input of the processing object text and a modified processing instruction obtained by changing the processing instruction to the processing means; the determining means performs a re-determining process of comparing the processing target text with the text output from the processing means as a result of the correction processing instruction and determining whether the predetermined condition is satisfied; The electronic device described in any one of items 1 to 4, characterized in that if it is determined that the specified conditions are not met even after performing the re-input process and the re-determination process a specified number of times, the control means controls to issue an error notification indicating a failure of processing based on the processing instruction. (Item 7) The scanner further includes a scan control means for controlling the scanner to scan the document; 7. The electronic device according to any one of items 1 to 6, wherein the processing target text is text detected from a document scanned under the control of the scan control means. (Item 8) 8. The electronic device according to any one of items 1 to 7, wherein the processing instruction is a query. (Item 9) 9. The electronic device of claim 8, wherein the query includes a sentence requesting that a typographical error in text be corrected. (Item 10) 10. The electronic device according to item 8 or 9, wherein the query includes a sentence requesting a summary of a text. (Item 11) 11. The electronic device according to any one of items 8 to 10, wherein the query includes a sentence requesting abstraction of a specific character string in a text. (Item 12) 12. The electronic device of any one of items 8 to 11, wherein the query includes a sentence requesting translation of text. (Item 13) 13. The electronic device according to any one of items 1 to 12, further comprising an acquisition means for dividing an image to be subjected to character recognition processing into a plurality of regions and acquiring text from each of the divided regions as the processing target text. (Item 14) Item 14. The electronic device described in item 13, characterized in that the input control means controls the processing means to input, for each of the plurality of areas, the text to be processed and processing instructions corresponding to the text obtained from each of the plurality of areas. (Item 15) 15. The electronic device according to any one of items 1 to 14, further comprising an arrangement means for arranging the first text or the second text output from the processing means in accordance with the layout of the text to be processed. (Item 16) The electronic device described in any one of items 1 to 15, characterized in that the input control means controls whether to input text obtained by character recognition processing to the processing means as the text to be processed, or whether to input an image file to the processing means instead of the text to be processed, based on the data amount of the image file. (Item 17) 1. A method performed in an electronic device, comprising: an input control step of controlling input of the text to be processed and processing instructions to the processing means; a determining step of comparing the processing target text input to the processing means with a first text output from the processing means to determine whether a predetermined condition is satisfied; a control step of controlling the output of the first text when the predetermined condition is satisfied in the determination step as a result of the processing instruction, and outputting a second text different from the first text when the predetermined condition is not satisfied; A method comprising: (Item 18) A program for causing a computer to function as each means of the electronic device described in any one of items 1 to 16. (Item 19) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device described in any one of items 1 to 16.

[0075] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0076] 101 Printer: 105 PC: 106 Generative AI Server: 201, 301 CPU

Claims

1. an input control means for controlling input of the text to be processed and processing instructions to the processing means; a determining means for comparing the processing target text input to the processing means with a first text output from the processing means and determining whether a predetermined condition is satisfied; a control means for controlling the processing so that, when the predetermined condition is satisfied by the determination means as a result of the processing instruction, the first text is output, and when the predetermined condition is not satisfied, a second text different from the first text is output; An electronic device comprising:

2. 2. The electronic device according to claim 1, wherein the processing means is provided in an external system, and the input control means controls to send the text to be processed and the processing instruction to the external system so that they are input to the processing means.

3. 2. The electronic device according to claim 1, wherein the processing means is a generative AI (Artificial Intelligence) system.

4. 2. The electronic device according to claim 1, wherein the predetermined condition is a condition for determining whether the first text is a result of processing according to the processing instruction being performed as expected.

5. When it is determined that the predetermined condition is not satisfied, the input control means controls the processing means to input the text to be processed and a second processing instruction obtained by changing the processing instruction, 2. The electronic device according to claim 1, wherein the second text is a text output from the processing means as a result of the second processing instruction.

6. When it is determined that the predetermined condition is not satisfied, the input control means performs a re-input process to control input of the processing object text and a modified processing instruction obtained by changing the processing instruction to the processing means; the determining means performs a re-determining process of comparing the processing target text with the text output from the processing means as a result of the correction processing instruction and determining whether the predetermined condition is satisfied; The electronic device according to claim 1, characterized in that if it is determined that the predetermined condition is not met even after performing the re-input process and the re-determination process a predetermined number of times, the control means controls to issue an error notification indicating a failure of the processing based on the processing instruction.

7. The scanner further includes a scan control means for controlling the scanner to scan the document; 2. The electronic device according to claim 1, wherein the text to be processed is text detected from a document scanned under the control of the scan control means.

8. The electronic device according to claim 1 , wherein the processing instruction is a query.

9. The electronic device of claim 8 , wherein the query includes a sentence requesting that a typographical error in text be corrected.

10. The electronic device of claim 8 , wherein the query includes a sentence requesting a summary of a text.

11. The electronic device of claim 8 , wherein the query includes a sentence requesting abstraction of a specific string of characters in a text.

12. The electronic device of claim 8 , wherein the query includes a sentence requesting translation of text.

13. 2. The electronic device according to claim 1, further comprising an acquisition unit that divides an image to be subjected to character recognition processing into a plurality of regions and acquires text from each of the divided regions as the processing target text.

14. 14. The electronic device according to claim 13, wherein the input control means controls the processing means to input, for each of the plurality of regions, the target text and a processing instruction corresponding to the text acquired from each of the plurality of regions.

15. 2. The electronic device according to claim 1, further comprising: an arrangement unit that arranges the first text or the second text output from the processing unit in accordance with a layout of the text to be processed.

16. 2. The electronic device according to claim 1, wherein the input control means controls whether to input text obtained by character recognition processing to the processing means as the text to be processed, or whether to input an image file to the processing means instead of the text to be processed, based on the amount of data in the image file.

17. 1. A method performed in an electronic device, comprising: an input control step of controlling input of the text to be processed and processing instructions to the processing means; a determining step of comparing the processing target text input to the processing means with a first text output from the processing means to determine whether a predetermined condition is satisfied; a control step of controlling the output of the first text when the predetermined condition is satisfied in the determination step as a result of the processing instruction, and outputting a second text different from the first text when the predetermined condition is not satisfied; A method comprising:

18. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 16.

19. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 16.

Images