Electronic device, control method, program, memory medium that stores a program

Abstract

An electronic device comprises: an acquisition means for acquiring text on an image by performing character recognition processing; a transmission means for transmitting the text acquired by the character recognition processing as a text to be processed and a processing instruction requesting processing of the text to a processing means; a receiving means for receiving a processing result based on the processing instruction from the processing means; and an output means for outputting a document as a result of character identification of the image using the text included in the processing result received by the receiving means.

Description

Technology Field

[0001] The present invention relates to an electronic device capable of executing an automatic character recognition function, a method, a program, and a storage medium storing the program in the electronic device. Background Technology

[0002] Automatic character recognition systems are known to automatically recognize characters written on an image and convert them into text. However, there may be errors in the recognition of automatic character recognition systems, and Patent Document 1 describes a system for detecting typographical errors and omissions through machine learning.

[0003] Meanwhile, generative artificial intelligence (Generative AI) is well known. Generative AI is a type of machine learning model that generates new data by utilizing data that has already been trained. For example, ChatGPT (Chat Generative Pre-Trained Transformer), a chat generative AI capable of conversing in a chat format, is widely used. As a plugin that provides additional features to ChatGPT, there is an automatic character recognition system called ChatOCR (Chat Optical Character Recognition). When using ChatOCR, image files are sent directly to the plugin. By using ChatOCR, in addition to extracting text from PDF files or image files, one can also ask questions about the extracted text. Prior art literature

[0004] Japanese Patent Publication No. 2019-145023 The problem to be solved

[0005] In a configuration that outputs using processing results received from an external source, it is assumed that directly transmitting image files will result in increased communication capacity or a decrease in communication speed. Therefore, research is required to suppress the communication load in a configuration that outputs using processing results received from an external source.

[0006] The present invention provides a structure that suppresses communication load in a configuration that outputs using processing results received from an external source. means of solving the problem

[0007] An electronic device according to the present invention is characterized by having: an acquisition means for acquiring text on an image by performing character recognition processing; a transmission means for transmitting the text acquired by the character recognition processing as a text to be processed and a processing instruction requesting processing of the text to a processing means; a receiving means for receiving a processing result based on the processing instruction from the processing means; and an output means for outputting a document as a result of character identification of the image using the text included in the processing result received by the receiving means. Effects of the invention

[0008] According to the present invention, communication load can be suppressed in a configuration that outputs using processing results received from the outside.

[0009] Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. Additionally, in the accompanying drawings, identical or similar components are given the same reference numerals. Brief explanation of the drawing

[0010] The attached drawings are included in the specification, constitute a part thereof, illustrate embodiments of the present invention, and are used to explain the principles of the present invention together with the description. Figure 1 is a diagram illustrating the configuration of a text processing system. Figure 2 is a diagram illustrating the configuration of a PC. Figure 3 is a diagram illustrating the configuration of a printing device. Figure 4 is a flowchart illustrating the process of outputting the execution result of the OCR function. Figure 5 is a diagram illustrating the flow of the processing of Figure 4. Figure 6 is a diagram illustrating a query. Figure 7 is a drawing illustrating a confirmation screen. Figure 8 is a flowchart showing the processing of S406. Figure 9 is a drawing illustrating a notification screen. Specific details for implementing the invention

[0011] Embodiments are described in detail below with reference to the attached drawings. Furthermore, the following embodiments do not limit the invention covered by the patent claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined at will. Additionally, in the attached drawings, the same reference number is assigned to identical or similar components, and redundant descriptions are omitted.

[0012] FIG. 1 is a diagram illustrating an example of the configuration of a text processing system in the present embodiment. In the text processing system (100), a PC (105) and a printing device (101) are connected to communicate with each other via a wireless LAN (102). Additionally, the PC (105) and the printing device (101) can access the Internet (104) via an access point (103). With such a configuration, the PC (105) and the printing device (101) can communicate with a generative AI server (106) that provides a generative AI (Artificial Intelligence) service connected to the Internet (104). Generative AI is a machine learning model built using deep learning, and the generative AI service can output creative results regarding images, text, video, voice, etc., compared to conventional services. As an example of a generative AI service, ChatGPT, which uses a sentence generation language model, is known and realizes a conversational AI service that realizes human-like conversation. Additionally, the generative AI server (106) may be a server device of a single entity, or a form in which multiple server devices are linked, provided that it is an external system using generative AI.

[0013] A PC (105) is an information processing device having communication functions such as a wireless LAN or a wired LAN. Additionally, a wireless LAN is sometimes referred to as a WLAN (Wireless LAN). For example, a smartphone, a laptop PC, a tablet terminal, or a PDA (Personal Digital Assistant) is used as the PC (105). The PC (105) can communicate with the printing device (101) via the wireless LAN (102). For example, the PC (105) can instruct the PC (101) to perform a printing function or a scanning function via the wireless LAN (102). Additionally, the PC (105) and the printing device (101) may be directly connected without passing through an access point (103). That is, the PC (105) can communicate with the printing device (101) via a direct connection. Furthermore, a wired network may be used as a network between the PC (105), the printing device (101), and the access point (103), or as part of that network.

[0014] The printing device (101) is an example of a printing device having a printing function. The printing device (101) may be configured as a multifunctional printer (MFP) having a reading function (scanner), a FAX function, or a telephone function. Additionally, the printing device (101) has a communication function capable of wireless communication with a PC (105). In this embodiment, the printing device (101) is described as an example, but a device of a different type from the printing device (101) may be used. For example, a fax machine, a scanner, a projector, a mobile terminal, a smartphone, a laptop PC, a tablet terminal, a PDA, a digital camera, a music player, a television, a smart speaker, AR (Augmented Reality) glasses, etc., having a communication function may be used. The printing device (101) receives printing data including image data from a PC (105) connected via, for example, an access point (103), and forms an image based on the data. Alternatively, the printing device (101) transmits image data read by, for example, a scanner function to a PC (105) connected via an access point (103). Other control information, etc., can also be communicated with a network connected via the access point (103).

[0015] The access point (103) is a communication device that is provided separately (outside) from the PC (105) and the printing device (101) and operates as a base station device for the WLAN. Additionally, the access point (103) may be referred to as an external access point (103) or an external wireless base station. A communication device having WLAN communication capabilities can perform communication in the WLAN infrastructure mode through the access point (103). In this embodiment, the PC (105) and the printing device (101) are also examples of communication devices. Additionally, the wireless infrastructure mode may be referred to as "wireless infrastructure mode." In other words, wireless infrastructure mode is a mode for, for example, the printing device (101) to communicate with the PC (105) through the access point (103) to which the printing device (101) is connected. The access point (103) communicates with a communication device that has authorized (authenticated) access to the device and relays wireless communication between that communication device and another communication device. Additionally, the access point (103) is connected to a wired LAN communication network and relays communication between a communication device connected to that network and another communication device wirelessly connected to the access point (103). Furthermore, if the authentication method of the network configured by the access point (103) is a method using an authentication server (if the access point (103) corresponds to an authentication method using an authentication server), it performs access control by performing authentication of the communication device connected to the network in conjunction with an authentication server (not shown). Additionally, the access point (103) may correspond to an authentication method that does not use an authentication server.

[0016] The PC (105) and the printing device (101) can perform wireless communication by using their respective WLAN communication functions, either through a wireless infrastructure mode via an external access point (103) or through a peer-to-peer mode that does not go through an external access point (103). Additionally, the peer-to-peer mode may be referred to as "P2P mode" or, in contrast to the wireless infrastructure mode, as "wireless direct mode." In other words, the P2P mode is a mode for the printing device (101) to communicate directly with the PC (105) without going through the access point (103). The P2P mode includes Wi-Fi Direct (registered trademark) mode and software access point (soft AP) mode. Additionally, Wi-Fi Direct (registered trademark) may be referred to as WFD. That is, the wireless direct mode can also be described as a communication mode based on the IEEE 802.11 series.

[0017] FIG. 2 is a diagram illustrating an example of the configuration of a PC (105). The PC (105) has a main board (220) that controls the entire device, a wireless communication unit (213) that performs WLAN communication, a display unit (212), an operation unit (211), and a short-range wireless communication unit (214) that performs wireless communication other than that of the wireless communication unit (213). The main board (220) includes, for example, a CPU (201), ROM (202), RAM (203), image memory (204), data conversion unit (205), camera unit (206), non-volatile memory (207), data storage unit (208), speaker unit (209), and power supply unit (210). Each functional unit within the main board (220) is connected to each other via a system bus (215). Additionally, the main board (220) and the wireless communication unit (213), and the main board (220) and the short-range wireless communication unit (214) are connected, for example, via a dedicated bus. Additionally, the main board (220) and the display unit (212), and the main board (220) and the control unit (211) are each connected, for example, via a dedicated bus.

[0018] The CPU (201) is a system control unit and controls the entire PC (105). The operation of the PC (105) described in this embodiment is realized, for example, by the CPU (201) reading a program stored in the ROM (202) into the RAM (203) and executing it. Additionally, dedicated hardware for each process may be provided. The ROM (202) stores control programs and built-in operating system (OS) programs that the CPU (201) executes. Software control, such as scheduling and task switching, is performed by the CPU (201) executing each control program stored in the ROM (202) under the management of the built-in OS stored in the ROM (202). Additionally, the ROM (202) stores applications (printing applications) that generate information that can be interpreted by the printing device (101). Information that can be interpreted by the printing device (101) is information corresponding to the functions that the printing device (101) can execute, and the application can perform settings such as printing or scanning on the printing device (101) or instruct the execution of each function. The RAM (203) is composed of SRAM (Static RAM), etc. The RAM (203) stores data such as program control variables, setting values ​​registered by the user, and management data of the PC (105). In addition, the RAM (203) may be used as a buffer for various work. The image memory (204) is composed of memory such as DRAM (Dynamic RAM). The image memory (204) temporarily stores image data received through the wireless communication unit (213) or image data read from the data accumulation unit (208) for processing by the CPU (201). The non-volatile memory (207) is composed of memory such as flash memory, for example, and continues to store data even when the power of the PC (105) is turned off. In addition, the memory configuration of the PC (105) is not limited to the configuration described above.For example, the image memory (204) and RAM (203) may be shared, and data backup, etc. may be performed using the data storage unit (208). In addition, although DRAM is given as an example of the image memory (204), other storage media such as a hard disk or non-volatile memory may be used.

[0019] The data conversion unit (205) performs data conversion such as interpreting data of various formats, color conversion, and image conversion. The camera unit (206) has the function of electronically recording and encoding an image input through a lens. Image data obtained by capturing by the camera unit (206) is stored in the data accumulation unit (208). The speaker unit (209) performs control to realize the function of inputting or outputting voice. The power supply unit (210) is, for example, a portable battery and performs control to supply power into the device. The display unit (212) electronically controls the display content and performs control to display various input content, the operating status of the PC (105), the status status, etc. The operation unit (211) receives user operation and performs control such as generating an electrical signal corresponding to the operation and outputting it to the CPU (201).

[0020] The PC (105) performs wireless communication using a wireless communication unit (213) to perform data communication with other communication devices, such as a printing device (101). The wireless communication unit (213) converts data into packets and transmits the packets to other communication devices. Additionally, the wireless communication unit (213) restores packets from other external communication devices into original data and outputs them to the CPU (201). The wireless communication unit (213) is a unit for realizing communication in accordance with standards such as WLAN. The short-range wireless communication unit (214) performs communication using a different communication method from the wireless communication unit (213), for example, Bluetooth (registered trademark). The configuration of the PC (105) and the main board (220) is not limited to the above. For example, individual functions of the main board (220) realized by the CPU (201) may be realized by a processing circuit such as an ASIC (Application-Specific Integrated Circuit), or by either hardware or software.

[0021] FIG. 3 is a block diagram illustrating an example of the configuration of a printing device (101). The printing device (101) has a main board (320) that controls the entire device, a USB communication unit (307), a wireless communication unit (309), a wired communication unit (310), an operation display unit (312), a power button (313), a print unit (315), and a scan unit (317).

[0022] A CPU (301) in the form of a microprocessor is provided on the main board (320). The CPU (301) controls the printing device (101) according to a control program stored in a program memory (302) in the form of ROM connected via an internal bus (318) and contents stored in a data memory (303) in the form of RAM. The operation of the printing device (101) described in this embodiment is realized, for example, by the CPU (301) reading the program stored in the program memory (302) into the data memory (303) and executing it. The CPU (301) controls the scan control unit (316) to optically read the original document from the scan unit (317) and stores the read data in the image memory within the data memory (303). The scan control unit (316) is an interface for connecting the scan unit (317) and the main board (320) and performs conversion of the scanned image data format, etc. The CPU (301), for example, controls the print control unit (314) to print the image of the read data stored in the image memory of the data memory (303) in the print unit (315) onto the recording medium (copy function). The print control unit (314) is an interface for connecting the print unit (315) and the main board (320) and performs image data conversion, etc. The data conversion unit (304) performs data conversion such as interpreting data of various formats, color conversion, and conversion from image data to print data. The code decoding processing unit (305) performs encoding processing and decoding processing, as well as enlargement and reduction processing, of image data (JPEG, PNG, etc.) handled by the printing device (101).

[0023] The CPU (301) controls the USB communication unit (307) through the USB communication control unit (306) to perform USB communication with an external PC (105) via a USB connection. The CPU (301) controls the operation control unit (311) to receive operation information from the power button (313) or the operation display unit (312). The CPU (301) controls the operation control unit (311) to display, for example, the status of the printing device (101) or a function selection menu on the operation display unit (312). The CPU (301) controls the wireless communication unit (309) and the wired communication unit (310) through the communication control unit (308) according to the operation information received from the operation display unit (312). For example, the CPU (301) changes the communication method settings or sets up a connection to a network according to the operation information.

[0024] The wireless communication unit (309) is a unit capable of providing WLAN communication functions. That is, the wireless communication unit (309) converts data into packets according to WLAN standards and transmits the packets to other communication devices. Additionally, the wireless communication unit (309) restores packets from other external communication devices into original data and outputs them to the CPU (301). The wireless communication unit (309) is configured to enable data (packet) communication in a WLAN system compliant with, for example, the IEEE 802.11 standard series (IEEE 802.11a / b / g / n / ac / ax, etc.). However, it is not limited to this configuration, and the wireless communication unit (309) may be capable of enabling communication in a WLAN system compliant with other standards. Additionally, the wireless communication unit (309) can perform communication in WFD mode, P2P mode, wireless infrastructure mode, etc. Additionally, the PC (105) and the printing device (101) are capable of wireless communication based on WFD mode, and the wireless communication unit (309) has a soft AP function or a group owner function. That is, the wireless communication unit (309) can establish a communication network in P2P mode or determine a channel used for communication in P2P mode.

[0025] The wired communication unit (310) is a unit for performing wired communication. The wired communication unit (310) is capable of data (packet) communication in a wired LAN (Ethernet) system, for example, in accordance with the IEEE 802.3 series. In addition, in wired communication using the wired communication unit (310), communication in a wired communication mode is possible. The wired communication unit (310) is connected to the main board (320) via a bus cable, etc.

[0026] The printing device (101) has an OCR (Optical Character Recognition) function that recognizes character information on an image read by a scanning function and extracts text data. The OCR function may be implemented, for example, by a scan control unit (316). By the OCR function, data in a text-searchable file format is created. Such data includes, for example, PDF, XPS (XML Paper Specification), etc. The printing device (101) can save (or transmit) the file generated by the OCR function to a designated storage location, either internal or external. Instructions for setting and executing such OCR functions may be given on an operation panel on the printing device (101) or given to the printing device (101) from an application installed on the PC (105). The printing device (101) is not limited to the configuration shown in FIG. 3, but appropriately has a configuration according to the functions that can be performed by the device applied as the printing device (101).

[0027] However, there may be errors, such as typos, in the results of automatic character recognition processing by the OCR function. In this embodiment, a configuration is assumed in which errors in the results of automatic character recognition processing by the OCR function are corrected by a generative AI service. For this purpose, for example, a configuration is conceived in which an image file generated by the OCR function is transmitted to a generative AI server (106) that provides a generative AI service by using ChatOCR, a plugin of ChatGPT that provides a sentence generation language model. However, in such a configuration, it is assumed that the communication capacity between the server and the OCR function increases or the communication speed decreases when the OCR function is executed on a large number of image files. Therefore, in this embodiment, a configuration is described that enables suppressing the communication load between the generative AI server (106).

[0028] FIG. 4 is a flowchart illustrating the process of outputting the execution result of an OCR function in the present embodiment. The process of FIG. 4 is realized, for example, by a CPU (301) reading a program stored in a computer-readable program memory (302) into a data memory (303) and executing it. The process of the flowchart is initiated from a state in which the printing device (101) is started, a home screen is displayed on the operation display unit (312), and a user can select a desired function. On the home screen, menu buttons corresponding to, for example, a scan function, a copy function, etc. are displayed so that they can be selected by the user.

[0029] In S400, the CPU (301) determines the function selected through the operation display unit (312). If the received function is a scan function, proceed to S401; if it is not a scan function, terminate the processing of FIG. 4 and perform the processing corresponding to the function. The description of the processing is omitted. In this embodiment, it is described as having received the selection of a scan function through the operation display unit (312).

[0030] In S401, the CPU (301) determines whether it has received a scan start instruction while the file format after scanning is specified as "PDF (OCR)". That is, in S401, it determines whether to create a character-identified PDF file. In other words, the determination process of S401 is a process of determining whether to create an image file by executing a character recognition function. If a scan start instruction is received while the format is specified as "PDF (OCR)", the process proceeds to S402. If a scan start instruction is received while a different file format is specified, the process of FIG. 4 is terminated, and the processing corresponding to the function is performed. The description of that processing is omitted.

[0031] In S402, the CPU (301) controls the scanning unit (305) to perform a scan of a document installed on a document holder not shown, and acquires read data. The CPU (305) stores the read data in the data memory (303).

[0032] In S403, the CPU (301) performs a process of cutting out character blocks from the read data stored in the data memory (303). Here, a character block is a cluster (block) of densely packed characters.

[0033] FIG. 5 is a diagram illustrating the flow of the processing of FIG. 4. Data (501) in FIG. 5 represents an example of the execution result of the processing of S403, and "opple" of character block (502) and "6anana" of character block (503) are examples of the above character blocks. Character blocks (502, 503) correspond to areas that are subject to automatic character recognition processing, which are separated on the image represented by data (501). "opple" of character block (502) represents an example where "apple" is intended but resulted in an incorrect result due to automatic character recognition. Also, "6anana" of character block (503) represents an example where "banana" is intended but resulted in an incorrect result due to automatic character recognition. Character blocks are cut out by grouping. The cut-out area corresponds to the above-mentioned separation area. Then, the CPU (301) performs an OCR function for each cut character block and stores the execution result (text to be processed) in the data memory (303). Additionally, in S403, the character blocks may be cut after the OCR function is performed. Subsequent processing is performed by focusing on one of the character blocks cut in S403.

[0034] In S404, the CPU (301) creates a query for input to the generative AI by concatenating the above automatic character recognition results after inputting "Please correct the typos in the following sentence," and transmits it to the generative AI server (106) using a wireless communication unit (309). In other words, a query is an example of a processing instruction requesting processing by the generative AI server (106). Furthermore, transmission to the generative AI server (106) can also be described as input control processing to the generative AI.

[0035] FIG. 6 is a diagram illustrating an example of a query written for each character block. The query written for each character block is written in the development code of a generative AI for which a library is publicly available. FIG. 6 shows an example written in Python (registered trademark).

[0036] The first row of Fig. 6 indicates that a package related to a generative AI tool can be loaded and used. Although labeled as "package" in the drawing, package names such as open source code are listed.

[0037] The second row of Fig. 6 shows the input of an account key code obtained to use the API of a generative AI tool. The key code can be a string of numbers and letters, and "YOUR_API_KEY" in Fig. 6 is an example thereof.

[0038] In the third row of FIG. 6, a variable name is specified to store the return value of the create function in the ChatCompletion group. The name is not fixed to 'response' and can be anything in particular. By doing so, APIs can be called sequentially. For example, by naming them sequentially as response1 and response2, APIs can be called sequentially on the cut character blocks, such as the query (504, 505) in FIG. 5.

[0039] Lines 4 through 7 of Fig. 6 correspond to the above query. In line 4 of Fig. 6, the identifier of the completed model is specified. In line 5 of Fig. 6, the content to be sent to the chat is indicated. In line 6 of Fig. 6, it is indicated that 'system' is specified as the role. Content can specify what function to be executed, and in this example, the correction of typos in the sentence is specified. Line 7 of Fig. 6 indicates that 'user' is specified as the role. The execution result of the OCR function is stored in Content.

[0040] In addition, in the 7th line of Fig. 6, a sentence example different from the example in Fig. 5 is shown. That is, Fig. 6 shows an example where the intended sentence is "A weather forecast says that it is sunny tomorrow. An umbrella is not needed." but the result of automatic character recognition is incorrect: "A weather forecast says that it is sunny tomorrow. An umbrella is not needed."

[0041] The 8th and 9th lines of Fig. 6 are closing parentheses. The 10th line of Fig. 6 indicates that only the content of the message (i.e., the response to the query) among the information received from the generative AI is output. This content is used for displaying a preview of the modification result or for reflecting it in a file that saves the execution result of the OCR function (i.e., a PDF containing character information).

[0042] In S405, the CPU (301) receives a correction result returned from the generative AI server (106) using a wireless communication unit (309) and stores it in the data memory (303). The correction result (507) of FIG. 5 shows an example of a correction result from the generative AI server (106) corresponding to the generative AI (506). That is, it shows that "opple" has been corrected to "apple".

[0043] In S406, the CPU (301) performs a process to check whether the modified result is as expected. The details of the process in S406 will be described later using FIG. 8.

[0044] In S407, the CPU (301) determines whether there is another character block cut out in S403. If it is determined that there is another character block, it focuses on that character block and repeats the processing from S403. If it is determined that there is no other character block, it proceeds to S408.

[0045] In S408, the CPU (301) lays out the modification result received in S405 at a position corresponding to the character block cut out in S403.

[0046] The data (508) of FIG. 5 illustrates an example in which a modified result is laid out at a position corresponding to a character block (character block before modification) corresponding to each query input into the generative AI (506). The character block (509) represents the modified result "apple" received at S405 after "opple" was transmitted to the generative AI server (106) at S404. Additionally, the character block (509) is positioned corresponding to the position of "opple" in the character block (502) read by scanning. The character block (510) represents the modified result "banana" received at S405 after "6anana" was transmitted to the generative AI server (106) at S404. Additionally, the character block (510) is positioned corresponding to the position of "6anana" in the character block (503) read by scanning.

[0047] In S409, the CPU (301) displays the result laid out in S408 on the operation display unit (312) as a confirmation screen.

[0048] FIG. 7 is a drawing illustrating an example of a confirmation screen displayed in S409. A modification result preview (702) is displayed on the confirmation screen (701). The modification result preview (702) is created based on the data (508) of FIG. 5.

[0049] Although "opple" and "6anana" were described above as examples of queries, the same applies to other examples. For instance, if "A weather forecast says that it is sunny tomorrow. An umbrella is not needed." is entered as a query in S404, S405 receives "A weather forecast says that it is sunny tomorrow. An umbrella is not needed." as a correction result. In that case, the correction result "A weather forecast says that it is sunny tomorrow. An umbrella is not needed." is laid out corresponding to the position on the manuscript of the scanned character block "A weather forecast says that it is sunny tomorrow. An umbrella is not needed."

[0050] The confirmation screen (701) is a screen for confirming the modification results from the generation system AI server (106) to the user, and displays a button (703) for accepting user confirmation of the modified content and a button (704) for accepting instructions to perform further modifications.

[0051] In S410, the CPU (301) receives a user operation on the confirmation screen (701) through the operation display unit (312). In S411, the CPU (301) determines whether to perform a second modification based on the received user operation. Specifically, for example, if a button (704) is pressed, it is determined to perform a second modification, and if a button (703) is pressed, it is determined not to perform a second modification. If it is determined to perform a second modification, proceed to S412, and if it is determined not to perform a second modification, proceed to S413.

[0052] In S412, after inputting "Please correct the typos in the following sentence," the CPU (301) creates a query for input to the generative AI by connecting the correction result received from the generative AI server (106) in S405, and transmits it to the generative AI server (106) using a wireless communication unit (309). After that, the processing from S405 is repeated.

[0053] In S413, the CPU (301) executes a process of outputting the layout result displayed as a confirmation screen (701) in S409 as a document. For example, the CPU (301) executes a print process by controlling the print unit (315). Alternatively, the CPU (301) executes a process of transmitting to the PC (105) using the wireless communication unit (309). After that, the process of FIG. 4 is terminated.

[0054] Next, the verification process for the modification result of S406 will be explained with reference to Fig. 8.

[0055] In S801, the CPU (301) initializes the number of times the judgment processing of the modification result by the generation AI server (106) is performed. Specifically, for example, the number is cleared to zero. In addition, a variable representing the number of times is pre-allocated in the data memory (303).

[0056] In S802, the CPU (301) compares the number of characters in a character block before and after modification by the generation AI server (106). For example, the number of characters in the character block (502) of FIG. 5 and the number of characters in the character block (509) are compared. In S803, the CPU (301) determines whether the difference in the number of characters in a character block before and after modification by the generation AI server (106) is within the range of a presumed error. Specifically, for example, the CPU (301) refers to the data memory (303) to determine whether the number of characters in the character block cut out in S403 falls within 10% of the error with respect to the number of characters in the character block that is the modification result received from the generation AI server (106) in S405, that is, whether it falls within 90% to 110%. If it is determined to be within the presumed range of error, proceed to S804, and if it is determined not to be within the presumed range of error, proceed to S806.

[0057] In S804, the CPU (301) obtains a similarity by comparing the strings of character blocks before and after modification by the generative AI server (106). Specifically, for example, the CPU (301) refers to the data memory (303) and calculates the similarity between the string of the character block cut out in S403 and the string of the character block received from the generative AI server (106) in S405 by Gestalt pattern matching.

[0058] In S805, the CPU (301) determines whether the similarity obtained in S804 is above a threshold. Specifically, for example, the CPU (301) determines whether the similarity calculated by Gestalt pattern matching is 0.8 or higher. If it is determined to be above the threshold, the processing of FIG. 8 is terminated. If it is determined not to be above the threshold, that is, below the threshold, the process proceeds to S806.

[0059] As such, in this embodiment, after the correction result is returned from the generative AI server (106) in S405, the determination of the correction result by the generative AI server (106) is performed in S803 and S805. In other words, this can be described as a process for determining whether the result of the processing by the generative AI server (106) was performed as expected. That is, it can be described as a process for determining whether the typo correction was performed as expected by the generative AI server (106). In S803, the difference in the number of characters is the determination condition, and in S805, the similarity is the determination condition. The determination conditions for whether the typo correction was performed as expected are not limited to these, and other conditions may be used. For example, the number of words or the number of subjects may be used as conditions.

[0060] In S806, the CPU (301) increments the number of times the judgment processing of the modification result is performed. In S807, the CPU (301) determines whether the number of times the judgment processing of the modification result is performed has reached a predetermined number (has been performed a predetermined number of times). Specifically, for example, the CPU (301) determines whether the number of times the judgment processing of the modification result is performed has reached 10. If it is determined that the predetermined number has been reached, proceed to S808, and if it is determined that the predetermined number has not been reached, that is, that the number is less than the predetermined number, proceed to S809.

[0061] In S808, the CPU (301) displays an error notification screen on the operation display unit (312) to notify that the correction by the generation AI server (106) was not performed correctly, that is, that the correction of errors by the generation AI server (106) failed, and then terminates the processing of FIGS. 8 and FIGS. 4.

[0062] FIG. 9 is a diagram illustrating an example of an error notification screen displayed in S808. On the error notification screen (900), a message indicating that the correction by the generation system AI server (106) was not performed correctly is displayed. In the area (901), a character block cut out in S403 is displayed as the text before correction.

[0063] In S809, the CPU (301) writes a query for input to the generative AI, stating, "There is a defect in the current correction result. Please correct it correctly," and transmits it to the generative AI server (106) using a wireless communication unit (309). Then, the CPU (301) receives the correction result returned from the generative AI server (106) using the wireless communication unit (309), stores it in the data memory (303), and repeats the processing from S802. In this embodiment, the input query to the generative AI in S809 is different from the input query to the generative AI in S404 and S412. That is, in S809, the retransmission of the processing instruction to the generative AI server (106) is performed by a query (correction processing instruction) that is a modified version of the query used in S404. By doing so, a result different from the correction result received in S405 can be expected. Then, re-evaluation processing is performed by S803 and S805 using the result. In addition, the above re-transmission processing is, in other words, re-input processing to the generation system AI server (106).

[0064] The effect obtained by executing the processing of Fig. 8 is explained. For example, let us assume a case where the sentence before typo correction is a command containing typos such as "Descri6e Halloween in 100 wonds." In that case, the generative AI receives the sentence before typo correction as a command sentence, and as a result of conversion processing other than typo correction, the sentence "Halloween is a traditional festival celebrated every year on October 31. This festival is mainly popular in the United States and Canada, but it is spreading throughout the world." is output to the user.

[0065] For the above example, when the processing of Fig. 8 is executed, the conversion processing result, "Halloween is a traditional festival celebrated on October 31st every year. This festival is mainly popular in the United States and Canada, but it is spreading throughout the world." is determined to have a difference of 47 characters in the number of characters compared to the sentence before typo correction. This is determined to exceed the error range, and is judged as "No" in S803. Furthermore, if it were judged as "Yes" in S803, a comparison of similarity by Gestalt pattern matching is performed in S805. Since the similarity between the conversion processing result and the sentence before conversion is 0.18, which is below the threshold, it is judged as "No" in S805. As a result, it is not output to the user, and the instruction sentence is changed, and the generation AI is instructed to convert again in S809. In this way, the conversion processing result that does not meet expectations is discarded without being output to the user, thereby increasing the possibility of outputting a more expected conversion processing result to the user.

[0066] In this embodiment, it was explained that in S403, a process of cutting out character blocks is executed for the read data stored in the data memory (303). At that time, it may be determined whether the amount of data in the read data is greater than or equal to a threshold. Also, the processing of FIG. 4 may be controlled based on whether the amount of data is greater than or equal to a threshold. For example, if it is determined that the amount of data is greater than or equal to a threshold, the processing of FIG. 4 is executed as described in this embodiment. On the other hand, if it is determined that the amount of data is not greater than or equal to a threshold, that is, is less than a threshold, in S404 and S412, a transmission control may be executed to transmit an image file corresponding to the read data, rather than a string such as the 7th line of FIG. 6.

[0067] In this embodiment, a process for verifying the result of typo correction of the generative AI has been described. However, the operation of this embodiment can be applied not only to typo correction but also to each of the processes of summarization, anonymization, and translation.

[0068] Summarization processing is performed to extract key points from the string before conversion. When applying the processing of FIG. 8 to the summarization processing, in S404, a query is made to input the string before conversion, which is 500 characters long, to the generative AI, for example, by concatenating the string before conversion with a command, such as “Please summarize the following sentence within 100 characters.” In this case, since the command specifies a character limit of 100 characters, in S803, for example, it is determined whether the number of characters in the output result of the generative AI falls within the range of 90 to 100 characters. Then, in S804, the CPU (301) extracts a word from the output result of the generative AI, and in S805, it is determined whether the extracted word is included in the string before conversion. If it is determined that it is included, the processing of FIG. 8 is terminated, and if it is determined that it is not included, the process proceeds to S806. The processing of other steps is the same as the processing described in this embodiment. As a result, summary processing results that do not meet expectations are discarded without being displayed to the user, thereby increasing the likelihood of outputting more expected summary processing results.

[0069] Anonymization processing is performed to prevent the output of personal information, such as names or addresses. When applying the processing of Fig. 8 to the anonymization processing, in S404, a query is input into the generation system AI by concatenating a string before conversion containing personal information after a command sentence, for example, "Please anonymize the following sentence." In this case, for example, a name included in the sentence before conversion is replaced by abstracting it into characters such as "Mr. A" that cannot identify an individual, and an address is replaced with a notation up to the name. Since it is assumed that the output result in the anonymization processing is not significantly different from the input result, just like the typographical correction in this embodiment, a comparison of the number of characters is performed in S803 and a similarity comparison is performed in S805 to determine whether the desired conversion processing result has been obtained. If it is determined that the desired conversion processing result has been obtained, the processing of Fig. 8 is terminated, and if it is determined that the desired conversion processing result has not been obtained, the process proceeds to S806. As a result, anonymization results that do not meet expectations are not output to the user and are discarded, thereby increasing the possibility of outputting more expected anonymization results.

[0070] Translation processing is performed to replace the string before conversion with a different language from the string before conversion. When applying the processing of FIG. 8 to translation processing, in S404, a query is made to input to the generative AI by concatenating the string before conversion written in Japanese after a command sentence, such as "Please translate the following sentence into English." In translation processing, it is assumed that the number of characters in the string after conversion is different from the number of characters in the string before conversion, while the number of words is assumed not to be significantly different. Therefore, processing of S803 is not performed, and in S804, the CPU (301) extracts the number of words, such as nouns, verbs, and subjects, from the string after conversion. Then, in S805, a comparison is made between the number of words in the string before conversion and the number of words in the string after conversion, and by determining whether the difference is within a predetermined range, it is determined whether the desired conversion processing result has been obtained. If it is determined that the desired conversion processing result has been obtained, the processing of FIG. 8 is terminated, and if it is determined that the desired conversion processing result has not been obtained, the process proceeds to S806. As a result, translation processing results that do not meet expectations are not output to the user and are discarded, thereby increasing the possibility of outputting more expected translation processing results.

[0071] Although the operation of this embodiment has been described as being performed by the CPU (301), the various controls described above may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0072] Furthermore, although the present invention has been described in detail based on suitable embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope that do not deviate from the gist of the present invention are also included in the present invention. In addition, each of the embodiments described above merely represents one embodiment of the present invention, and it is also possible to appropriately combine each embodiment.

[0073] In addition, in the above-described embodiment, the present invention was described using the case where it is applied to a printing device (101) as an example, but this is not limited to this example and can be applied to any electronic device capable of performing a character recognition function. That is, the present invention can be applied to a personal computer, PDA, mobile phone terminal, portable image viewer, printer device equipped with a display, digital photo frame, electronic book reader, OCR camera, etc.

[0074] (Other embodiments)

[0075] The present invention can also be realized in a process in which a program realizing one or more functions of the above-described embodiment is supplied to a system or device through a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. In addition, it can also be realized by a circuit (e.g., an ASIC) realizing one or more functions.

[0076] The invention is not limited to the above embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. Explanation of the symbols

[0077] 101: Printing device 105: PC 106: Generative AI Server 201, 301: CPU This application claims priority based on Japanese patent application No. 2024-008827 filed on January 24, 2024, and incorporates all of its contents herein by reference.

Claims

Claim 1 An electronic device characterized by comprising: an acquisition means for acquiring text on an image by performing character recognition processing; a transmission means for transmitting the text acquired by the character recognition processing as a text to be processed and a processing instruction requesting processing of the text to a processing means; a receiving means for receiving a processing result based on the processing instruction from the processing means; and an output means for outputting a document as a result of character identification of the image using the text included in the processing result received by the receiving means. Claim 2 An electronic device according to claim 1, wherein the acquisition means divides an image of a target subject to character recognition processing into a plurality of regions and acquires text as a text to be processed from each of the divided plurality of regions. Claim 3 An electronic device according to paragraph 2, wherein the transmitting means transmits to the processing means a processing instruction requesting processing of the text to be processed and the text obtained from each of the plurality of regions for each text to be processed corresponding to each of the plurality of regions. Claim 4 An electronic device characterized by further comprising, in any one of claims 1 to 3, a placement means for arranging the processing result received by the receiving means according to the layout of the image of the text acquired by the acquiring means. Claim 5 An electronic device according to claim 4, wherein the output means outputs the document based on a layout arranged by the arrangement means. Claim 6 An electronic device characterized by further comprising, in any one of claims 1 to 5, a determination means for comparing the processing target text transmitted to the processing means and the first text included in the processing result received by the receiving means to determine whether a predetermined condition is satisfied. Claim 7 An electronic device according to claim 6, wherein the above-mentioned predetermined condition is a condition for determining whether the above-mentioned first text is the result of processing by the above-mentioned processing instruction being performed as intended. Claim 8 An electronic device according to claim 6 or 7, wherein when the determination means determines that the predetermined condition is satisfied, the output means outputs the first text as the document. Claim 9 An electronic device characterized by further having a control means for controlling to receive a processing result from the processing means, which includes a second text different from the first text, when the determination means determines that the predetermined condition is not satisfied in any one of claims 6 to 8. Claim 10 An electronic device according to claim 9, wherein if the determination means determines that the predetermined condition is not satisfied, the control means controls the transmission means to retransmit the processing target text and a second processing instruction that modifies the processing instruction to the processing means, and the second text is a text included in the processing result received from the processing means as a result of the second processing instruction. Claim 11 An electronic device according to claim 10, further comprising a re-evaluation means for re-evaluating whether the processing target text and the second text included in the processing result received from the processing means as a result of the second processing instruction satisfy the predetermined condition, wherein if it is determined that the predetermined condition is not satisfied even after the retransmission and the re-evaluation are performed a predetermined number of times, the control means controls the execution of an error notification indicating the failure of processing by the processing instruction. Claim 12 An electronic device according to any one of claims 1 to 11, wherein the processing means is provided in an external system, and the transmitting means controls the transmission of the processing target text and the processing instruction to the external system so as to be input into the processing means. Claim 13 An electronic device characterized in that, in any one of claims 1 to 12, the processing means is a generative AI (Artificial Intelligence) system. Claim 14 An electronic device according to any one of claims 1 to 13, further comprising a scan control means for controlling to scan a manuscript, wherein the image is an image obtained by scanning the manuscript by control by the scan control means. Claim 15 An electronic device characterized by further comprising, in any one of claims 1 to 14, a transmission control means for controlling whether to transmit text obtained by the character recognition processing to the processing means as a text to be processed, or to transmit an image file corresponding to the image to the processing means, based on the amount of data of the image. Claim 16 An electronic device characterized in that, in any one of claims 1 to 15, the processing instruction is a query. Claim 17 An electronic device according to claim 16, wherein the above query includes a sentence requesting the correction of typographical errors in the text. Claim 18 An electronic device according to claim 16 or 17, wherein the query includes a sentence requesting a summary of text. Claim 19 An electronic device characterized in that, in any one of claims 16 to 18, the query includes a sentence that requires abstracting a specific string of characters in text. Claim 20 An electronic device characterized in that, in any one of claims 16 to 19, the query includes a sentence requesting a translation of text. Claim 21 A control method for an electronic device executed in an electronic device, comprising: an acquisition process for acquiring text on an image by performing character recognition processing; a transmission process for transmitting the text acquired by the character recognition processing as a text to be processed and a processing instruction requesting processing of the text to a processing means; a reception process for receiving a processing result based on the processing instruction from the processing means; and an output process for outputting a document as a result of character identification of the image using the text included in the processing result received in the reception process. Claim 22 A program for making a computer function as an acquisition means for acquiring text on an image by performing character recognition processing; a transmission means for transmitting the text acquired by the character recognition processing as a text to be processed and a processing instruction requesting processing of the text to a processing means; a receiving means for receiving a processing result based on the processing instruction from the processing means; and an output means for outputting a document as a result of character identification of the image using the text included in the processing result received by the receiving means. Claim 23 A storage medium storing a program for a computer to function as an acquisition means for acquiring text on an image by performing character recognition processing, a transmission means for transmitting the text acquired by the character recognition processing as a text to be processed and a processing instruction requesting processing of said text to a processing means, a receiving means for receiving a processing result based on said processing instruction from said processing means, and an output means for outputting a document as a result of character identification of said image using the text included in said processing result received by said receiving means.

Images