Electronic apparatus, control method, program, and storage medium for storing program

Abstract

This electronic apparatus includes: an acquisition means that acquires text in an image by performing character recognition processing; a transmission means for transmitting, to a processing means, text to be processed which is the text acquired by the character recognition processing, and a processing instruction for requesting processing of the text; a reception means that receives a processing result based on the processing instruction from the processing means; and an output means that outputs a document serving as the result of character recognition on the image using the text included in the processing result received by the reception means.

Description

Electronic device, control method, program, and storage medium for storing the program

[0001] The present invention relates to an electronic device capable of performing an automatic character recognition function, a method for the electronic device, a program, and a storage medium storing the program.

[0002] An automatic character recognition system is known that automatically recognizes characters written in an image and converts them into text. However, the recognition by the automatic character recognition system can be erroneous, and Patent Document 1 describes a system for detecting typos and omissions using machine learning.

[0003] On the other hand, generative artificial intelligence (generative AI) is known. Generative AI is a machine learning model that generates new data by utilizing learned data. For example, ChatGPT (Chat Generative Pre-Trained Transformer), a chat generation AI that enables conversations in chat format, is widely used. ChatOCR (Chat Optical Character Recognition), an automatic character recognition system, is a plug-in that provides additional functionality to ChatGPT. When using ChatOCR, image files are sent directly to the plug-in. Using ChatOCR, you can extract text from PDF files and image files, as well as ask questions about the extracted text.

[0004] Japanese Patent Application Laid-Open No. 2019-145023

[0005] In a configuration in which processing results received from an external device are used for output, if image files are sent directly, it is expected that communication volume will increase and communication speed will decrease. Therefore, a method for reducing communication load is required in a configuration in which processing results received from an external device are used for output.

[0006] The present invention provides a mechanism for reducing communication load in a configuration in which processing results received from an external device are used for output.

[0007] The electronic device of the present invention is characterized by having an acquisition means for performing character recognition processing to acquire text on an image, a transmission means for transmitting the text acquired by the character recognition processing as the processing target text and a processing instruction requesting processing of the text to the processing means, a receiving means for receiving a processing result based on the processing instruction from the processing means, and an output means for outputting a document as a result of character recognition of the image using the text included in the processing result received by the receiving means.

[0008] According to the present invention, it is possible to reduce the communication load in a configuration in which processing results received from an external device are used for output.

[0009] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0010] The accompanying drawings are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention, and together with the description are used to explain the principles of the present invention. Figure 1 shows the configuration of a text processing system. Figure 2 shows the configuration of a PC. Figure 3 shows the configuration of a printing device. Figure 4 shows a flowchart of processing to output the results of execution of an OCR function. Figure 5 shows the processing flow of Figure 4. Figure 6 shows a query. Figure 7 shows a confirmation screen. Figure 8 shows a flowchart of processing in S406. Figure 9 shows a notification screen.

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] FIG. 1 illustrates an example of the configuration of a text processing system according to this embodiment. In the text processing system 100, a PC 105 and a printing device 101 are connected to each other via a wireless LAN 102 so that they can communicate with each other. Furthermore, the PC 105 and the printing device 101 can access the Internet 104 via an access point 103. This configuration allows the PC 105 and the printing device 101 to communicate with a generative AI server 106 that provides generative AI (Artificial Intelligence) services connected to the Internet 104. Generative AI is a machine learning model built using deep learning. Generative AI services can output more creative results for images, text, video, audio, and the like than conventional services. One example of a generative AI service is ChatGPT, which uses a text generation language model and provides a conversational AI service that enables human-like conversations. In addition, the generative AI server 106 may be a single server device or may be a configuration in which multiple server devices work together, as long as it is an external system that uses generative AI.

[0013] The PC 105 is an information processing device having communication functions such as a wireless LAN and a wired LAN. A wireless LAN is sometimes called a WLAN (Wireless LAN). Examples of the PC 105 include a smartphone, a notebook PC, a tablet terminal, and a PDA (Personal Digital Assistant). The PC 105 can communicate with the printing device 101 via a wireless LAN 102. For example, the PC 105 can instruct the PC 101 to execute a printing function or a scanning function via the wireless LAN 102. The PC 105 and the printing device 101 may be directly connected without going through the access point 103. That is, the PC 105 can communicate with the printing device 101 via a direct connection. A wired network may be used as the network between the PC 105, the printing device 101, and the access point 103, or as part of that network.

[0014] The printing device 101 is an example of a printing device having a printing function. The printing device 101 may be configured as a multifunctional printer (MFP) having a reading function (scanner), a fax function, and a telephone function. The printing device 101 also has a communication function that enables wireless communication with the PC 105. In this embodiment, the printing device 101 is described as an example, but devices of different types may be used. For example, devices having a communication function, such as a facsimile machine, a scanner, a projector, a mobile terminal, a smartphone, a laptop PC, a tablet terminal, a PDA, a digital camera, a music playback device, a television, a smart speaker, or AR (Augmented Reality) glasses, may be used. The printing device 101 receives print data including image data from the PC 105 connected via the access point 103, for example, and forms an image based on the data. Alternatively, the printing device 101 transmits image data read by, for example, a scanner function to the PC 105 connected via the access point 103. Other control information can also be communicated with the network connected via the access point 103.

[0015] The access point 103 is a communication device that is provided separately (externally) from the PC 105 and the printing device 101 and operates as a WLAN base station device. The access point 103 may also be referred to as an external access point 103 or an external wireless base station. Communication devices with WLAN communication capabilities can communicate in WLAN infrastructure mode via the access point 103. In this embodiment, the PC 105 and the printing device 101 are also examples of communication devices. The wireless infrastructure mode may also be referred to as a "wireless infrastructure mode." In other words, the wireless infrastructure mode is a mode in which, for example, the printing device 101 communicates with the PC 105 via the access point 103 to which the printing device 101 is connected. The access point 103 communicates with communication devices that have authorized (authenticated) connection to the access point 103 and relays wireless communications between the communication devices and other communication devices. The access point 103 is also connected to a wired LAN communication network and relays communications between communication devices connected to the network and other communication devices wirelessly connected to the access point 103. Furthermore, if the authentication method of the network formed by the access point 103 is a method that uses an authentication server (if the access point 103 supports an authentication method that uses an authentication server), the access point 103 cooperates with the authentication server (not shown) to authenticate communication devices that connect to the network, thereby performing access control. Note that the access point 103 may also support an authentication method that does not use an authentication server.

[0016] The PC 105 and the printing device 101 can use their respective WLAN communication functions to perform wireless communication in a wireless infrastructure mode via an external access point 103 or in a peer-to-peer mode that does not involve the external access point 103. Note that peer-to-peer mode is sometimes referred to as "P2P mode" or, in contrast to wireless infrastructure mode, as "wireless direct mode." In other words, P2P mode is a mode in which the printing device 101 communicates directly with the PC 105 without going through the access point 103. Examples of P2P mode include Wi-Fi Direct (registered trademark) mode and software access point (soft AP) mode. Note that Wi-Fi Direct (registered trademark) is sometimes referred to as WFD. In other words, wireless direct mode can be considered a communication mode that complies with the IEEE 802.11 series.

[0017] FIG. 2 is a diagram showing an example of the configuration of the PC 105. The PC 105 includes a main board 220 that controls the entire device, a wireless communication unit 213 that performs WLAN communication, a display unit 212, an operation unit 211, and a short-range wireless communication unit 214 that performs wireless communication different from that of the wireless communication unit 213. The main board 220 includes, for example, a CPU 201, a ROM 202, a RAM 203, an image memory 204, a data conversion unit 205, a camera unit 206, a non-volatile memory 207, a data storage unit 208, a speaker unit 209, and a power supply unit 210. The functional units within the main board 220 are interconnected via a system bus 215. The main board 220 and the wireless communication unit 213 and the short-range wireless communication unit 214 are connected, for example, via dedicated buses. The main board 220 and the display unit 212 and the operation unit 211 are connected, for example, via dedicated buses.

[0018] The CPU 201 is a system control unit that controls the entire PC 105. The operation of the PC 105 described in this embodiment is realized, for example, by the CPU 201 reading a program stored in the ROM 202 into the RAM 203 and executing it. Dedicated hardware for each process may also be provided. The ROM 202 stores control programs, such as a control program executed by the CPU 201 and an embedded operating system (OS) program. The CPU 201 executes each control program stored in the ROM 202 under the management of the embedded OS stored in the ROM 202, thereby performing software control such as scheduling and task switching. The ROM 202 also stores applications (printing apps) that generate information interpretable by the printing device 101. Information interpretable by the printing device 101 corresponds to functions that the printing device 101 can execute. The applications can configure the printing device 101 for operations such as printing and scanning, and instruct the printing device 101 to execute each function. The RAM 203 is composed of a static RAM (SRAM) or the like. The RAM 203 stores data such as program control variables, setting values ​​registered by the user, and management data for the PC 105. The RAM 203 may also be used as a buffer for various types of work. The image memory 204 is configured with memory such as a dynamic random access memory (DRAM). The image memory 204 temporarily stores image data received via the wireless communication unit 213 and image data read from the data storage unit 208 for processing by the CPU 201. The non-volatile memory 207 is configured with memory such as a flash memory, and continues to store data even when the PC 105 is powered off. Note that the memory configuration of the PC 105 is not limited to the above configuration. For example, the image memory 204 and the RAM 203 may be shared, or data may be backed up using the data storage unit 208. Although a DRAM is given as an example of the image memory 204, other storage media such as a hard disk or non-volatile memory may also be used.

[0019] The data conversion unit 205 analyzes data in various formats and performs data conversion such as color conversion and image conversion. The camera unit 206 has the function of electronically recording and encoding images input through a lens. Image data obtained by capturing images using the camera unit 206 is stored in the data storage unit 208. The speaker unit 209 performs control to realize the function of inputting or outputting audio. The power supply unit 210 is, for example, a portable battery, and controls the supply of power to the device. The display unit 212 electronically controls the display content and performs control to display various input contents, the operating status and status of the PC 105, etc. The operation unit 211 performs control such as generating an electrical signal corresponding to the operation upon receiving a user operation and outputting the signal to the CPU 201.

[0020] The PC 105 performs wireless communication using the wireless communication unit 213 to perform data communication with other communication devices such as the printing device 101. The wireless communication unit 213 converts data into packets and transmits the packets to the other communication devices. The wireless communication unit 213 also restores packets from other external communication devices to the original data and outputs the data to the CPU 201. The wireless communication unit 213 is a unit for realizing communication compliant with standards such as WLAN. The short-range wireless communication unit 214 communicates using a communication method different from that used by the wireless communication unit 213, such as Bluetooth (registered trademark). The configurations of the PC 105 and the main board 220 are not limited to those described above. For example, the individual functions of the main board 220 implemented by the CPU 201 may be implemented by a processing circuit such as an ASIC (application-specific integrated circuit), and may be implemented by either hardware or software.

[0021] 3 is a block diagram showing an example of the configuration of the printing device 101. The printing device 101 has a main board 320 that controls the entire device, a USB communication unit 307, a wireless communication unit 309, a wired communication unit 310, an operation display unit 312, a power button 313, a print unit 315, and a scan unit 317.

[0022] The main board 320 is provided with a microprocessor-type CPU 301. The CPU 301 controls the printing device 101 in accordance with a control program stored in a ROM-type program memory 302 connected via an internal bus 318 and the contents stored in a RAM-type data memory 303. The operation of the printing device 101 described in this embodiment is realized, for example, by the CPU 301 reading the program stored in the program memory 302 into the data memory 303 and executing it. The CPU 301 controls the scan control unit 316 to cause the scan unit 317 to optically scan an original and store the scanned data in the image memory in the data memory 303. The scan control unit 316 is an interface for connecting the scan unit 317 to the main board 320 and performs functions such as converting the format of scanned image data. The CPU 301 controls the print control unit 314 to cause the print unit 315 to print an image of the scanned data stored in the image memory in the data memory 303 onto a recording medium (copy function). The print control unit 314 is an interface for connecting the print unit 315 and the main board 320, and performs image data conversion, etc. The data conversion unit 304 performs data conversion such as analyzing data of various formats, color conversion, and converting image data into print data. The encoding / decoding processing unit 305 performs encoding and decoding processes and scaling processes for image data (JPEG, PNG, etc.) handled by the printing device 101.

[0023] The CPU 301 controls the USB communication unit 307 via a USB communication control unit 306 to perform USB communication with the external PC 105 via a USB connection. The CPU 301 controls an operation control unit 311 to receive operation information from a power button 313 and an operation display unit 312. The CPU 301 controls the operation control unit 311 to display, for example, the status of the printing apparatus 101 and a function selection menu on the operation display unit 312. The CPU 301 controls the wireless communication unit 309 and the wired communication unit 310 via a communication control unit 308 in accordance with the operation information received by the operation display unit 312. For example, the CPU 301 changes the communication method settings and sets up a connection to a network in accordance with the operation information.

[0024] The wireless communication unit 309 is a unit capable of providing WLAN communication functions. That is, the wireless communication unit 309 converts data into packets in accordance with WLAN standards and transmits the packets to other communication devices. The wireless communication unit 309 also restores packets from other external communication devices to the original data and outputs the data to the CPU 301. The wireless communication unit 309 is configured to be able to perform data (packet) communication in a WLAN system compliant with, for example, the IEEE 802.11 standard series (IEEE 802.11a / b / g / n / ac / ax, etc.). However, this configuration is not limited, and the wireless communication unit 309 may also be capable of performing communication in a WLAN system compliant with other standards. The wireless communication unit 309 is also capable of performing communication in WFD mode, P2P mode, wireless infrastructure mode, etc. Note that the PC 105 and the printing device 101 are capable of wireless communication based on WFD mode, and the wireless communication unit 309 has a soft AP function or a group owner function. That is, the wireless communication unit 309 can build a communication network in P2P mode and determine the channel to be used for communication in P2P mode.

[0025] The wired communication unit 310 is a unit for performing wired communication. The wired communication unit 310 is capable of data (packet) communication in a wired LAN (Ethernet) system conforming to, for example, the IEEE 802.3 series. Furthermore, wired communication using the wired communication unit 310 is capable of communication in a wired communication mode. The wired communication unit 310 is connected to the main board 320 via a bus cable or the like.

[0026] The printing device 101 has an OCR (Optical Character Recognition) function that recognizes character information on an image scanned by a scanning function and extracts text data. The OCR function may be implemented, for example, by the scan control unit 316. The OCR function creates data in a text-searchable file format. Examples of such data include PDF and XPS (XML Paper Specification). The printing device 101 can save (or send) files created by the OCR function to a specified internal or external storage location. Instructions to set and execute the OCR function may be given via an operation panel on the printing device 101 or from an application installed on the PC 105 to the printing device 101. The printing device 101 is not limited to the configuration shown in FIG. 3 and may have a configuration appropriate for the functions that the device used as the printing device 101 can implement.

[0027] However, the results of automatic character recognition processing using the OCR function may contain errors such as typos. In this embodiment, a configuration is assumed in which errors in the results of automatic character recognition processing using the OCR function are corrected using a generative AI service. To achieve this, for example, a configuration is considered in which ChatOCR, a plug-in for ChatGPT that provides a sentence generation language model, is used to send image files generated by the OCR function to a generative AI server 106 that provides generative AI services. However, in such a configuration, when the OCR function is executed on a large number of image files, it is expected that the communication volume with the server will increase or the communication speed will decrease. Therefore, in this embodiment, a configuration is described that enables the communication load with the generative AI server 106 to be reduced.

[0028] 4 is a flowchart showing a process for outputting the execution results of the OCR function in this embodiment. The process in FIG. 4 is realized, for example, by the CPU 301 reading a program stored in the computer-readable program memory 302 into the data memory 303 and executing it. The process of this flowchart begins when the printing apparatus 101 starts up, a home screen is displayed on the operation display unit 312, and a user can select a desired function. The home screen displays menu buttons corresponding to functions such as scanning and copying so that the user can select them.

[0029] In S400, the CPU 301 determines the function whose selection has been accepted via the operation display unit 312. If the accepted function is a scan function, the process proceeds to S401; if the accepted function is not a scan function, the process in FIG. 4 ends, and the process corresponding to the function is carried out. A description of this process will be omitted. In this embodiment, the explanation will be given assuming that the selection of the scan function has been accepted via the operation display unit 312.

[0030] In S401, the CPU 301 determines whether a scan start instruction has been received with the post-scan file format designated as "PDF (OCR)." That is, in S401, it is determined whether a character-recognized PDF file is to be created. In other words, the determination process in S401 is a process for determining whether an image file is to be created by executing the character recognition function. If a scan start instruction has been received with "PDF (OCR)" designated, the process proceeds to S402. If a scan start instruction has been received with another file format designated, the process in FIG. 4 ends, and the process corresponding to the function is performed. A description of this process will be omitted.

[0031] In step S402, the CPU 301 controls the scanning unit 305 to scan the document placed on the document platen (not shown) and acquires read data. The CPU 305 stores the read data in the data memory 303.

[0032] In S403, the CPU 301 executes a process of extracting character chunks from the read data stored in the data memory 303. Here, a character chunk is a block where characters are densely packed together.

[0033] FIG. 5 is a diagram showing the processing flow of FIG. 4 . Data 501 in FIG. 5 shows an example of the execution result of the processing of S403, with character block 502 "apple" and character block 503 "6anana" being examples of the character blocks. Character blocks 502 and 503 correspond to areas segmented on the image represented by data 501 that are the target of automatic character recognition processing. Character block 502 "apple" is an example of an incorrect result obtained by automatic character recognition, even though "apple" was intended. Character block 503 "6anana" is also an example of an incorrect result obtained by automatic character recognition, even though "banana" was intended. Character blocks are extracted by grouping. The extracted areas correspond to the segmented areas. The CPU 301 then executes an OCR function on each extracted character block and stores the execution result (processing target text) in data memory 303. In S403, character blocks may be extracted after the OCR function is executed. The subsequent processing is performed by focusing on one of the character blocks extracted in S403.

[0034] In S404, the CPU 301 creates a query for input to the generative AI by concatenating the input "Please correct the typo in the following sentence" with the result of the automatic character recognition, and transmits this to the generative AI server 106 using the wireless communication unit 309. In other words, the query is an example of a processing instruction requesting processing by the generative AI server 106. In other words, transmission to the generative AI server 106 can also be said to be an input control process for the generative AI.

[0035] 6 shows an example of a query created for each character block. The query created for each character block is written in the development code of a generative AI whose library is publicly available. FIG. 6 shows an example written in Python (registered trademark).

[0036] The first line in Figure 6 indicates that a package related to generative AI tools is to be loaded and made available for use. Although it is written as package in the figure, the package name of the open source code, etc., is written.

[0037] The second line in Figure 6 indicates that the key code of the account acquired to use the API of the generative AI tool should be entered. The key code can be a string of numbers and letters, and "YOUR_API_KEY" in Figure 6 is an example.

[0038] The third line in Figure 6 specifies the name of the variable that stores the return value of the create function in the ChatCompletion group. The name does not have to be "response" and can be anything. This allows for consecutive API calls. For example, by naming the variables in order as "response1" and "response2," it is possible to call the API consecutively for the extracted character blocks, as in queries 504 and 505 in Figure 5.

[0039] Lines 4 to 7 of Figure 6 correspond to the above query. Line 4 of Figure 6 specifies the identifier of the trained model. Line 5 of Figure 6 shows the content to be sent to the chat. Line 6 of Figure 6 shows that system is specified as the role. In Content, the desired function can be specified; in this example, correction of typos in text is specified. Line 7 of Figure 6 shows that user is specified as the role. The execution result of the OCR function is stored in Content.

[0040] Note that the seventh line in Figure 6 shows a sentence example that is different from the example in Figure 5. That is, Figure 6 shows an example of a sentence that was intended to be "A weather forecast says that it is sunny tomorrow. An umbrella is not needed," but was incorrectly output by automatic character recognition as "A weather forecast says that it is sunny tomorrow. An umbrella is not needed."

[0041] Lines 8 and 9 in Figure 6 are closing parentheses. Line 10 in Figure 6 indicates that only the message content (i.e., the response to the query) of the information received from the generative AI is output. This content is used to preview the correction results and to reflect them in a file (i.e., a PDF with text information) that stores the results of the OCR function.

[0042] In S405, the CPU 301 receives the correction result returned from the generative AI server 106 using the wireless communication unit 309 and stores it in the data memory 303. The correction result 507 in Figure 5 shows an example of the correction result from the generative AI server 106 corresponding to the generative AI 506. In other words, it shows that "opple" has been corrected to "apple".

[0043] In S406, the CPU 301 performs processing to check whether the correction result is as expected. Details of the processing in S406 will be described later with reference to FIG.

[0044] In S407, the CPU 301 determines whether there are other character blocks extracted in S403. If it is determined that there are other character blocks, it focuses on the character block and repeats the processing from S403. If it is determined that there are no other character blocks, it proceeds to S408.

[0045] In S408, the CPU 301 lays out the correction results received in S405 at positions corresponding to the character blocks extracted in S403.

[0046] Data 508 in Figure 5 shows an example in which correction results are laid out at positions corresponding to character blocks (character blocks before correction) corresponding to each query input to the generative AI 506. Character block 509 shows the correction result "apple" received in S405 after "apple" was sent to the generative AI server 106 in S404. Character block 509 is also placed corresponding to the position of "apple" in character block 502 read by scanning. Character block 510 shows the correction result "banana" received in S405 after "6anana" was sent to the generative AI server 106 in S404. Character block 510 is also placed corresponding to the position of "6anana" in character block 503 read by scanning.

[0047] In S409, the CPU 301 displays the results of the layout in S408 on the operation display unit 312 as a confirmation screen.

[0048] 7 is a diagram showing an example of the confirmation screen displayed in S409. A confirmation screen 701 displays a correction result preview 702. The correction result preview 702 is created based on the data 508 in FIG.

[0049] In the above, "apple" and "6anana" have been described as examples of queries, but the same applies to other examples. For example, if "A weather forecast says that it is summery tomorrow. An umbrella is not needed." is input as a query in S404, "A weather forecast says that it is sunny tomorrow. An umbrella is not needed." is received as the correction result in S405. In this case, the correction result "A weather forecast says that it is sunny tomorrow. An umbrella is not needed." is laid out in accordance with the position on the document of the character block "A weather forecast says that it is summery tomorrow. An umbrella is not needed." read by scanning.

[0050] The confirmation screen 701 is a screen for allowing the user to confirm the correction results made by the generative AI server 106, and displays a button 703 for accepting user confirmation that the corrections have been made, and a button 704 for accepting instructions to make further corrections.

[0051] In S410, the CPU 301 accepts a user operation on the confirmation screen 701 via the operation display unit 312. In S411, the CPU 301 determines whether or not to make further corrections based on the accepted user operation. Specifically, for example, if the CPU 301 accepts the pressing of button 704, it determines that further corrections will be made, and if the CPU 301 accepts the pressing of button 703, it determines that further corrections will not be made. If it is determined that further corrections will be made, the process proceeds to S412, and if it is determined that further corrections will not be made, the process proceeds to S413.

[0052] In S412, after inputting "Please correct the typo in the following sentence," the CPU 301 concatenates the correction results received from the generative AI server 106 in S405 to create a query to input to the generative AI, and transmits it to the generative AI server 106 using the wireless communication unit 309. Then, the processing from S405 is repeated.

[0053] In S413, the CPU 301 executes processing to output, as a document, the layout result displayed as the confirmation screen 701 in S409. For example, the CPU 301 controls the print unit 315 to execute printing processing. Alternatively, the CPU 301 executes processing to transmit the layout result to the PC 105 using the wireless communication unit 309. Thereafter, the processing in FIG. 4 ends.

[0054] Next, the correction result confirmation process in S406 will be described with reference to FIG.

[0055] In S801, the CPU 301 initializes the number of times that the generative AI server 106 performs the process of determining the correction result. Specifically, for example, the number of times is cleared to zero. Note that the variable representing the number of times is secured in advance in the data memory 303.

[0056] In S802, the CPU 301 compares the number of characters in the character chunk before and after the correction by the generative AI server 106. For example, the number of characters in character chunk 502 and the number of characters in character chunk 509 in FIG. 5 are compared. In S803, the CPU 301 determines whether the difference in the number of characters in the character chunk before and after the correction by the generative AI server 106 is within an expected error range. Specifically, for example, the CPU 301 references the data memory 303 and determines whether the number of characters in the character chunk extracted in S403 is within an error of 10% of the number of characters in the character chunk that is the correction result received from the generative AI server 106 in S405, that is, whether the number is within a range of 90% to 110%. If it is determined that the difference is within the expected error range, the process proceeds to S804; if it is determined that the difference is not within the expected error range, the process proceeds to S806.

[0057] In S804, the CPU 301 obtains the similarity by comparing the character strings of the character chunk before and after the correction by the generative AI server 106. Specifically, for example, the CPU 301 refers to the data memory 303 and calculates the similarity between the character string of the character chunk extracted in S403 and the character string of the character chunk received from the generative AI server 106 in S405 by Gestalt pattern matching.

[0058] In S805, the CPU 301 determines whether the similarity obtained in S804 is equal to or greater than a threshold. Specifically, for example, the CPU 301 determines whether the similarity calculated by Gestalt pattern matching is equal to or greater than 0.8. If it is determined that the similarity is equal to or greater than the threshold, the processing in FIG. 8 ends. If it is determined that the similarity is not equal to or greater than the threshold, i.e., is less than the threshold, the processing proceeds to S806.

[0059] As described above, in this embodiment, after the correction result is returned from the generative AI server 106 in S405, the generative AI server 106 performs a process of determining the correction result in S803 and S805. In other words, this is a process of determining whether the processing result in the generative AI server 106 was performed as expected. In other words, this is a process of determining whether the generative AI server 106 performed the typo correction as expected. In S803, the difference in the number of characters is used as the determination condition, and in S805, the similarity is used as the determination condition. The conditions for determining whether the typo correction was performed as expected are not limited to these, and other conditions may also be used. For example, the number of words, the number of subjects, etc. may be used as conditions.

[0060] In S806, the CPU 301 increments the number of times the correction result determination process is performed. In S807, the CPU 301 determines whether the number of times the correction result determination process has been performed has reached a predetermined number (whether the process has been performed a predetermined number of times). Specifically, for example, the CPU 301 determines whether the number of times the correction result determination process has been performed has reached 10. If it is determined that the predetermined number of times has been reached, the process proceeds to S808, and if it is determined that the predetermined number of times has not been reached, i.e., the number of times is less than the predetermined number, the process proceeds to S809.

[0061] In S808, the CPU 301 displays an error notification screen on the operation display unit 312 to notify that the correction by the generative AI server 106 was not performed correctly, i.e., that the generative AI server 106 failed to correct the typo, and then terminates the processing of Figures 8 and 4.

[0062] 9 is a diagram showing an example of the error notification screen displayed in S808. The error notification screen 900 displays a message indicating that the correction by the generative AI server 106 was not performed correctly. An area 901 displays the character block extracted in S403 as the text before correction.

[0063] In S809, the CPU 301 creates an input query to the generative AI, saying, "There is a defect in the current correction result. Please correct it correctly." and transmits it to the generative AI server 106 using the wireless communication unit 309. The CPU 301 then receives the correction result returned from the generative AI server 106 using the wireless communication unit 309, stores it in the data memory 303, and repeats the process from S802. In this embodiment, the input query to the generative AI in S809 is different from the input query to the generative AI in S404 and S412. In other words, in S809, a processing instruction is retransmitted to the generative AI server 106 using a query (correction processing instruction) that is a modified version of the query used in S404. This allows for a result different from the correction result received in S405. Then, a re-determination process is performed in S803 and S805 using the result. The above retransmission process is, in other words, a re-input process to the generative AI server 106.

[0064] The effect obtained by executing the process of Figure 8 will be described. For example, assume that the sentence before the typo correction is an imperative sentence containing a typo, such as "Describe Halloween in 100 wins." In this case, the generative AI receives the sentence before the typo correction as an instruction sentence, and outputs to the user the following sentence as a result of the conversion process other than the typo correction: "Halloween is a traditional festival celebrated every year on October 31st. This festival is popular mainly in the United States and Canada, but has spread throughout the world."

[0065] For the above example, when the process of FIG. 8 is executed, the conversion result of "Halloween is a traditional festival celebrated every October 31st. This festival is mainly popular in the United States and Canada, but is widespread worldwide." is determined to have a difference of 47 characters compared to the sentence before the typo correction. This is determined to be beyond the margin of error, and a "No" determination is made in S803. If a "Yes" determination is made in S803, a similarity comparison is made using Gestalt pattern matching in S805. Since the similarity between the conversion result and the sentence before conversion is 0.18, which is below the threshold, a "No" determination is made in S805. As a result, the instruction sentence is changed without being output to the user, and the generative AI is instructed to perform conversion again in S809. This allows unanticipated conversion results to be discarded without being output to the user, increasing the likelihood of outputting a more expected conversion result to the user.

[0066] In the present embodiment, in S403, the process of extracting character blocks from the scanned data stored in the data memory 303 is performed. At this time, it may be determined whether the amount of data in the scanned data is equal to or greater than a threshold. Then, the process of FIG. 4 may be controlled based on whether the amount of data is equal to or greater than the threshold. For example, if it is determined that the amount of data is equal to or greater than the threshold, the process of FIG. 4 is performed as described in this embodiment. On the other hand, if it is determined that the amount of data is not equal to or greater than the threshold, i.e., is less than the threshold, in S404 and S412, transmission control may be performed to transmit an image file corresponding to the scanned data, rather than a character string such as that shown on line 7 in FIG. 6 .

[0067] In this embodiment, the process of checking the typographical error correction results of the generative AI has been described. However, the operation of this embodiment can be applied not only to typographical error correction but also to summarization, anonymization, and translation processes.

[0068] The summarization process is performed to extract key points from the pre-conversion string. When the process of FIG. 8 is applied to the summarization process, in S404, a query input to the generative AI is, for example, an instruction sentence such as "Please summarize the following sentence in 100 characters or less," followed by a 500-character pre-conversion string. In this case, since the instruction sentence specifies a character limit of 100 characters, in S803, for example, it is determined whether the number of characters in the output result of the generative AI is within the range of 90 to 100 characters. Then, in S804, the CPU 301 extracts words from the output result of the generative AI, and in S805, it determines whether the extracted words are included in the pre-conversion string. If it is determined that the extracted words are included, the process of FIG. 8 ends; if it is determined that the extracted words are not included, the process proceeds to S806. The processing of the other steps is the same as the processing described in this embodiment. As a result, unsatisfactory summarization results are discarded without being output to the user, thereby increasing the possibility of outputting more expected summarization results.

[0069] The anonymization process is performed to prevent the output of personal information such as names and addresses. When the process of FIG. 8 is applied to the anonymization process, in S404, a query input to the generative AI is, for example, an instruction sentence such as "Please anonymize the following sentence." followed by a pre-conversion string containing personal information. In this case, for example, the name included in the pre-conversion sentence is abstracted and replaced with characters such as "Mr. A," which does not identify the individual, and the address is replaced with a notation up to the prefecture name. In the anonymization process, since it is expected that the output result will not differ significantly from the input result, the number of characters is compared in S803 and the similarity is compared in S805, similar to the typo correction in this embodiment, to determine whether the desired conversion process result has been obtained. If it is determined that the desired conversion process result has been obtained, the process of FIG. 8 ends; if it is determined that the desired conversion process result has not been obtained, the process proceeds to S806. As a result, an anonymization process result that does not meet expectations is discarded without being output to the user, thereby increasing the possibility of outputting a more expected anonymization process result.

[0070] The translation process is performed to replace a pre-conversion string with a language different from the pre-conversion string. When the process of FIG. 8 is applied to the translation process, in S404, a query input to the generative AI is, for example, an instruction sentence such as "Please translate the following sentence into English" followed by a pre-conversion string written in Japanese. In the translation process, while it is expected that the number of characters in the post-conversion string will differ from the number of characters in the pre-conversion string, it is expected that the number of words will not change significantly. Therefore, the process of S803 is not performed, and in S804, the CPU 301 extracts the number of words, such as nouns, verbs, and subjects, from the post-conversion string. Then, in S805, the number of words in the pre-conversion string and the number of words in the post-conversion string are compared to determine whether the difference is within a predetermined range, thereby determining whether the desired conversion process result has been obtained. If it is determined that the desired conversion process result has been obtained, the process of FIG. 8 ends. If it is determined that the desired conversion process result has not been obtained, the process proceeds to S806. As a result, a translation process result that does not meet expectations is discarded without being output to the user, thereby increasing the possibility of outputting a more expected translation process result.

[0071] The operation of this embodiment has been described as being performed by CPU 301, but the various controls described above may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0072] Furthermore, although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0073] Furthermore, in the above-described embodiment, the present invention has been described as being applied to the printing device 101, but the present invention is not limited to this example and can be applied to any electronic device that can execute character recognition functions. That is, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, printers with displays, digital photo frames, electronic book readers, OCR cameras, and the like.

[0074] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0075] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0076] 101 Printing device: 105 PC: 106 Generative AI server: 201, 301 CPU This application claims priority based on Japanese Patent Application No. 2024-008827, filed on January 24, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An acquisition means for performing character recognition processing to obtain text on an image; a transmission means for transmitting, as a processing target text, the text obtained by the character recognition processing, the processing target text, and a processing instruction for requesting processing on the text, to a processing means; a reception means for receiving a processing result based on the processing instruction from the processing means; and an output means for outputting a document as a result of character recognition of the image using the text included in the processing result received by the reception means. An electronic device characterized by comprising the above.

2. The acquisition means according to claim 1, wherein the acquisition means divides an image on which the character recognition processing is to be performed into a plurality of regions, and acquires text as a processing target text from each of the plurality of divided regions.

3. The transmission means according to claim 2, wherein the transmission means transmits, for each processing target text corresponding to each of the plurality of regions, the processing target text and a processing instruction for requesting processing on the text acquired from each of the plurality of regions, to the processing means.

4. The electronic device according to any one of claims 1 to 3, further comprising an arrangement means for arranging the processing result received by the reception means according to the layout of the text acquired by the acquisition means on the image.

5. The electronic device according to claim 4, wherein the output means outputs the document based on the layout arranged by the arrangement means.

6. The electronic device according to any one of claims 1 to 5, further comprising a determination means for comparing the processing target text transmitted to the processing means with a first text included in the processing result received by the reception means, and determining whether a predetermined condition is satisfied.

7. The electronic device according to claim 6, wherein the predetermined condition is a condition for determining whether the first text is a result of the processing according to the processing instruction being performed as expected.

8. The electronic device according to claim 6 or 7, wherein when it is determined by the determination means that the predetermined condition is satisfied, the output means outputs the first text as the document.

9. The electronic device according to any one of claims 6 to 8, further comprising control means for controlling to receive, from the processing means, a processing result including a second text different from the first text when it is determined by the determination means that the predetermined condition is not satisfied.

10. When it is determined by the determination means that the predetermined condition is not satisfied, the control means controls the transmission means to retransmit the text to be processed and a second processing instruction obtained by changing the processing instruction to the processing means, and the second text is a text included in the processing result received from the processing means as a result of the second processing instruction. The electronic device according to claim 9.

11. The electronic device further comprises re-determination means for comparing the text to be processed with the second text included in the processing result received from the processing means as a result of the second processing instruction to re-determine whether the predetermined condition is satisfied, and even if the re-transmission and the re-determination are performed a predetermined number of times and it is determined that the predetermined condition is not satisfied, the control means controls to issue an error notification indicating the failure of the processing according to the processing instruction. The electronic device according to claim 10.

12. The processing means is provided in an external system, and the transmission means controls to transmit the text to be processed and the processing instruction to the external system so as to be input to the processing means. The electronic device according to any one of claims 1 to 11.

13. The electronic device according to any one of claims 1 to 12, wherein the processing means is a generative AI (Artificial Intelligence) system.

14. The electronic device according to any one of claims 1 to 13, further comprising scan control means for controlling to scan a manuscript, and the image is an image obtained by scanning the manuscript under the control of the scan control means.

15. The electronic device according to any one of claims 1 to 14, further comprising transmission control means for controlling whether to transmit the text obtained by the character recognition processing as the text to be processed to the processing means or to transmit an image file corresponding to the image to the processing means based on the data amount of the image.

16. The electronic device according to any one of claims 1 to 15, wherein the processing instruction is a query.

17. The electronic device according to claim 16, wherein the query includes a sentence requesting correction of text typos.

18. The electronic device according to claim 16 or 17, wherein the query includes a sentence requesting a summary of the text.

19. The electronic device according to any one of claims 16 to 18, wherein the query includes a sentence requesting abstraction of a specific character string in the text.

20. The electronic device according to any one of claims 16 to 19, wherein the query includes a sentence requesting translation of the text.

21. A control method for an electronic device executed in the electronic device, comprising: an acquisition step of performing character recognition processing to acquire text on an image; a transmission step of transmitting, as a processing target text, the text acquired by the character recognition processing, the processing target text, and a processing instruction requesting processing on the text, to a processing means; a reception step of receiving a processing result based on the processing instruction from the processing means; and an output step of outputting a document as a result of character recognition of the image, using the text included in the processing result received in the reception step.

22. A program for causing a computer to function as: an acquisition means for performing character recognition processing to acquire text on an image; a transmission means for transmitting, as a processing target text, the text acquired by the character recognition processing, the processing target text, and a processing instruction requesting processing on the text, to a processing means; a reception means for receiving a processing result based on the processing instruction from the processing means; and an output means for outputting a document as a result of character recognition of the image, using the text included in the processing result received by the reception means.

23. A storage medium storing a program for causing a computer to function as an acquisition means for performing character recognition processing to acquire text on an image, a transmission means for transmitting, as a processing target text, the text acquired by the character recognition processing and a processing instruction for requesting processing on the text to a processing means, a reception means for receiving a processing result based on the processing instruction from the processing means, and an output means for outputting a document as a result of character recognition of the image using the text included in the processing result received by the reception means.

Images