scanner

Abstract

To provide a technique for increasing the possibility of outputting an image subjected to suitable image processing even if a user does not select the type of the image.SOLUTION: MFP1 can access the generation AI server 200 that uses the learned model 201. The learned model 201 has been learned so that, based on input scan data, an image indicated by the scan data can be separated into regions based on the type of the image, and image processing suitable for the type of the image can be performed. In a case where there is a copy instruction with automatic AI via the user MFP1, IF13 uses the reading engine 16 to scan a document and generate scan AI, and transmits the generated scan output to the generation and data generation server 200. After that, in a case in which the trained model 201 receives processed AI that has undergone image processing from the generation / output server 200, the MFP1 performs output based on the received processed output.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technical field disclosed in this specification relates to a scanner that reads an image of a document and outputs image data. [Background technology]

[0002] Conventionally, there is known a scanner that outputs image data of a scanned image according to a scanning setting corresponding to the type of image included in a document. For example, Patent Document 1 discloses a scanner that accepts an operation to select the type of image from a user and changes the correction amount for show-through according to the selected type of image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-92375 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in Patent Document 1, the user must select the image type. Therefore, the selection result is influenced by the user's sense, and the appropriate type is not necessarily selected. Also, there is technology that automatically selects the image type using a scanner, but when the original image contains a mixture of text and photographs, automatic selection of the image type does not necessarily result in an appropriate output result. [Means for solving the problem]

[0005] A scanner made to solve this problem is a scanner comprising a reading engine, a user interface, and a communication interface, wherein the scanner can access a server that uses a trained model via the communication interface, and the trained model has been trained to be able to separate an image shown in input scan data into regions based on the type of image, and the trained model has been further trained to be able to perform image processing appropriate for the type of image, and when a first scan instruction is received via the user interface, the scanner uses the reading engine to scan a document and outputs an image of the document based on the scan results. and a transmission process of transmitting the generated first scan data to the server via the communication interface. After performing the transmission process, when the server receives, via the communication interface, first processed scan data output by the trained model based on the first scan data, the first processed scan data being data obtained by the trained model performing image processing appropriate for the type of image on each area of ​​an image separated according to the type of image for the image shown in the first scan data, the server is configured to perform a first output process of performing an output based on the received first processed scan data.

[0006] The scanner disclosed in this specification transmits first scan data generated by scanning a document to a server that uses a trained model. The trained model has been trained to perform both region separation based on the image type and image processing appropriate for the image type based on the input scan data. After transmitting the first scan data, when the scanner receives first processed scan data from the server, it outputs based on the received first processed scan data. The first processed scan data is data in which the trained model has performed image processing appropriate for the image type on each region separated according to the image type for the image shown in the first scan data. This makes it highly likely that an image that has been subjected to appropriate image processing will be output, even if the user does not select the image type.

[0007] A control method for realizing the functions of the scanner, a computer program, and a computer-readable storage medium storing the computer program are also novel and useful. [Effects of the Invention]

[0008] The technology disclosed in this specification realizes a technology that increases the likelihood that an image that has been subjected to suitable image processing can be output, even without the user having to select the type of image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram showing a schematic configuration of an MFP according to an embodiment of the present invention. [Figure 2] FIG. 10 is a sequence diagram illustrating an example of a copy procedure. [Figure 3] FIG. 10 is an explanatory diagram showing an example of an operation procedure for executing a copy; [Figure 4] FIG. 2 is an explanatory diagram showing an example of an image of a document; [Figure 5] FIG. 1 is a sequence diagram showing an example of an AI automated procedure. [Figure 6] FIG. 10 is an explanatory diagram showing an example of region separation of scan data using a trained model. [Figure 7]FIG. 10 is an explanatory diagram showing an example of a preview screen. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of a scanner will be described in detail with reference to the accompanying drawings. This specification discloses a multifunction peripheral (hereinafter referred to as "MFP") having various functions including an image reading function and a communication function.

[0011] As shown in Fig. 1, the MFP 1 of this embodiment includes a controller 10 including a CPU 11 and a memory 12. The MFP 1 also includes a user interface (hereinafter referred to as "user IF") 13, a communication interface (hereinafter referred to as "communication IF") 14, a print engine 15, and a reading engine 16, which are electrically connected to the controller 10. The MFP 1 is an example of a scanner. Note that the controller 10 in Fig. 1 is a general term for hardware and software used to control the MFP 1, and does not necessarily represent a single piece of hardware actually present in the MFP 1.

[0012] The CPU 11 of the MFP 1 executes various processes in accordance with programs read from the memory 12 and based on user operations. The memory 12 of the MFP 1 stores various programs and data, including an operating system (hereinafter referred to as "OS") 21 and an image correction program 22. The memory 12 is also used as a work area when various processes are executed. The buffer provided in the CPU 11 is also an example of the memory 12. Details of the programs and data will be described later.

[0013] Note that an example of memory 12 is not limited to a ROM, RAM, HDD, etc. built into MFP 1, but may also be a storage medium that is readable and writable by CPU 11. For example, an external memory such as a USB memory or HDD connected to MFP 1, or a memory or HDD provided in a device connected to MFP 1 via communication IF 14 are also examples of memory.

[0014] Computer-readable storage media are non-transitory media. In addition to the above examples, non-transitory media also include recording media such as CD-ROMs and DVD-ROMs. Non-transitory media are also tangible media. On the other hand, electrical signals carrying programs downloaded from servers on the Internet are computer-readable signal media, which is a type of computer-readable medium, but are not included in non-transitory computer-readable storage media.

[0015] The user IF 13 includes hardware for displaying a screen for notifying the user of information and hardware for accepting operations by the user. The user IF 13 may include a touch panel having a screen display function and an operation acceptance function, or may include a combination of a display and hardware buttons, etc.

[0016] The communication IF 14 includes hardware for communicating with external devices. The communication IF 14 includes functions compatible with communication standards such as Wi-Fi (registered trademark), Ethernet (registered trademark), and USB. The MFP 1 may be equipped with multiple communication IFs 14 compatible with multiple communication standards.

[0017] Print engine 15 includes a configuration for printing an image on a print medium such as a sheet. The image formation method of print engine 15 is, for example, an electrophotographic method or an inkjet method. MFP1 of this embodiment is equipped with print engine 15 that can perform both color printing using multiple colored colorants and monochrome printing using a single colored colorant.

[0018] The reading engine 16 includes a configuration for scanning a document placed on the flatbed or a document placed on a document feeder and transported to a reading position, and generating scan data as the scan result. The MFP 1 of this embodiment is equipped with the reading engine 16 that can perform color reading, which reads a document as a color image.

[0019] The MFP1 can connect to the Internet 100 via the communication IF 14, and can access a generation AI server 200 via the Internet 100, for example, as shown in FIG. 1. The generation AI server 200 includes a trained model 201 that has been trained in advance using various data so as to output response data based on input data. The generation AI server 200 is an example of a server that uses a trained model. The trained model 201 may be a server of an AI provider that is publicly available on the Internet 100. An example of an AI provider is OpenAI.

[0020] The trained model 201 has been trained so that, upon receiving input of image data, it can recognize the image represented by the image data by dividing it into multiple regions and recognizing the type of image each region represents. The trained model 201 can recognize the image data by dividing it into regions where the image type is text and regions where the image type is other than text, for example.

[0021] For convenience, dividing an image into multiple regions and recognizing them, and recognizing which type of image each region is, will be simply referred to as "region separation," "separating image data," or "separating." For convenience, a certain type of image will also be described in the format of "image type + 'image'." For example, an image whose image type is text will also be simply described as a "text image." For convenience, "image type + 'region'" will be described to indicate a region that is that type of image. For convenience, "image type + 'image'" may also be described to indicate a region that is that type of image. For convenience, "image type" will also be described as "image type."

[0022] The trained model 201 may be able to recognize image types such as photos, receipts, figures, tables, illustrations, etc. in addition to text. The trained model 201 may, for example, include information indicating multiple predetermined types of image types. The trained model 201 may also be able to recognize image types without including information indicating predetermined image types.

[0023] Furthermore, the trained model 201 has been trained to perform image processing appropriate for each type of image on image data separated into regions for each type of image. For example, the trained model 201 can perform edge enhancement processing, which is image processing appropriate for text images, on text regions. Also, for example, the trained model 201 can perform error diffusion processing, which is image processing appropriate for photographic images, on photographic regions.

[0024] For example, the trained models on the servers provided by AI providers are trained using big data, which is expected to enable them to analyze image data, separate it into regions of different image types with high accuracy, and then perform appropriate image processing for each separated region.

[0025] Furthermore, the trained model 201 may be trained to perform smoothing processing to smooth the boundaries of regions when performing image processing appropriate for the type of image for each region. Furthermore, the trained model 201 may be trained to perform each process without reducing the number of pixels when performing image processing and smoothing processing appropriate for the type of image.

[0026] The generation AI server 200 may be provided with an API for instructing the trained model 201 to separate the image data. The generation AI server 200 may be provided with an API for instructing the trained model 201 to output image data that has been subjected to image processing for each region after separation. For example, the MFP 1 may be able to instruct the trained model 201 to output the processed image data by using an API provided in the generation AI server 200 for instructing the trained model 201 to output the image data that has been subjected to image processing.

[0027] Alternatively, the generation AI server 200 may accept instructions in the form of a prompt, i.e., a string of characters. For example, the MFP 1 may input image data and a prompt instructing region separation of the image data and image processing for each region to the generation AI server 200, thereby instructing the trained model 201 to perform region separation and image processing.

[0028] Next, the procedure for scanning by the MFP1 will be described. The following processes basically refer to CPU processing in accordance with instructions written in a program. In other words, processes such as "judging," "extracting," "selecting," "calculating," "deciding," "identifying," "acquiring," "receiving," and "control" in the following description represent CPU processing. CPU processing also includes hardware control using an OS API. In this specification, the operation of each program will be described without mentioning the OS. In other words, in the following description, a statement to the effect that "Program B controls Hardware C" may also mean that "Program B controls Hardware C using the OS API." Furthermore, CPU processing in accordance with instructions written in a program may be described in abbreviated terms. For example, it may be described as "performed by the CPU." Furthermore, CPU processing in accordance with instructions written in a program may be described in abbreviated terms, such as "performed by Program A."

[0029] In this specification, the terms "notification," "alert," "notification," "reply," "response," and "answer" are used not only to mean the transmission of information to a person, but also to mean communication or exchange of information between devices or between components within a device. Note that the components within a device include software.

[0030] Note that "obtaining" is used as a concept that does not require a request. In other words, the process of receiving data without the CPU requesting it is also included in the concept of "the CPU obtaining data." Furthermore, "data" in this specification is represented as a bit string that can be read by a computer. Data with the same substantial meaning but different formats will be treated as the same data. The same applies to "information" in this specification. Furthermore, "requesting" and "instructing" are concepts that indicate outputting information indicating a request or an instruction to the other party. Furthermore, information indicating a request or an instruction will also be simply referred to as a "request" or "instruction."

[0031] Furthermore, the process by a CPU to determine whether information A indicates event B is sometimes conceptually described as "determining whether event B is true from information A." The process by a CPU to determine whether information A indicates event B or event C is sometimes conceptually described as "determining whether event B or event C is true from information A."

[0032] In addition, in this specification, a setting item may be simply referred to as a "setting." A setting value may be simply referred to as a "setting." A setting value may also be referred to as a "parameter." Furthermore, storing a setting value in a memory or the like may be simply referred to as a "setting." An operation for setting or an input for setting may be simply referred to as a "setting."

[0033] A copy procedure, which is a procedure for executing printing based on scan data generated by scanning, will be described as an example of a procedure related to scanning by the MFP 1. The copy procedure by the MFP 1 will be described with reference to the sequence diagram in FIG.

[0034] When the MFP 1 is in a standby state, it can display a standby screen on the user IF 13 (A01). For example, as shown in FIG. 3(A), the MFP 1 displays a standby screen 50 including a copy icon 51. In addition to the copy icon 51, the standby screen 50 also displays a plurality of icons that can accept instructions for various functions that can be executed by the MFP 1. By accepting an operation on each icon, the MFP 1 accepts selection of a function corresponding to the operated icon. The user selects the copy function by operating the copy icon 51 on the user IF 13 (A02).

[0035] In this specification, the term "icon" or "button" refers to a representation of a processing target or processing content corresponding to an input operation, using an image, symbol, text, or the like of a certain size on a screen displayed on the user IF 13. In this specification, the terms "icon" and "button" are not limited to general icons and buttons, but also include so-called operators for receiving input operations, such as menus for receiving selection instructions.

[0036] When an operation on the copy icon 51 is accepted, the MFP 1 displays a parameter selection screen 60 for accepting parameter selection on the user IF 13 (A03), as shown in Fig. 3(B), for example. The parameter selection screen 60 includes an image type button 61 for accepting the selection of an image type, a black and white copy execution button 62, and a color copy execution button 63.

[0037] When an operation on the image type button 61 is accepted (A04), the MFP 1 displays an image type selection screen 70 on the user IF 13 (A05), as shown in FIG. 3C, for example. The image type selection screen 70 includes options: an AI automatic (with preview) button 71, an AI automatic (without preview) button 72, an automatic main unit button 73, a text button 74, a photo button 75, and a receipt button 76. As will be described in more detail below, the user can select a processing procedure by operating any of the AI ​​automatic (with preview) button 71, the AI ​​automatic (without preview) button 72, and the automatic main unit button 73. The user can also select an image type by operating any of the text button 74, the photo button 75, and the receipt button 76 (A06).

[0038] The AI ​​Auto (with preview) button 71 and the AI ​​Auto (without preview) button 72 are options for receiving instructions for copy procedures, including procedures using the trained model 201. The procedures using the trained model 201 include a procedure for scanning a document to generate scanned data, and a procedure for transmitting the generated scanned data to the generation AI server 200 equipped with the trained model 201. The AI ​​Auto (with preview) button 71 and the AI ​​Auto (without preview) button 72 are examples of a first icon. As described above, the trained model 201 is capable of performing region separation of image data and image processing for each region.

[0039] The AI ​​Auto (with preview) button 71 is an option for a procedure that includes a preview procedure for previewing the separation results after the separation procedure using the trained model 201 and before the correction procedure. The AI ​​Auto (without preview) button 72 is an option for a procedure that does not include a preview procedure. The AI ​​Auto (with preview) button 71 is an example of an icon with preview, and the AI ​​Auto (without preview) button 72 is an example of an icon without preview.

[0040] The Main Unit Auto button 73, Text button 74, Photo button 75, and Receipt button 76 are options for accepting instructions for copy procedures that do not use the trained model 201. When any of the Main Unit Auto button 73, Text button 74, Photo button 75, and Receipt button 76 is selected, the MFP 1 scans the document and generates scanned data, but does not send the generated scanned data to the generation AI server 200. The Main Unit Auto button 73, Text button 74, Photo button 75, and Receipt button 76 are all examples of second icons.

[0041] The automatic main body button 73 is an option for an automatic determination procedure in which the MFP 1 analyzes the generated scan data and determines the type of image shown in the scan data. By displaying the automatic main body button 73, the user can select automatic determination, which is a procedure in which the MFP 1 determines the type of image, thereby reducing the burden on the user of considering the type of image.

[0042] On the other hand, the text button 74, photo button 75, and receipt button 76 can also be considered as options for accepting a user's designation of the image type before generating scan data. For example, when an operation on the text button 74 is accepted, the MFP 1 determines that the image type is text without analyzing the scan data and operates accordingly. A user who is aware of the image type contained in the document can select one of the text button 74, photo button 75, and receipt button 76 to omit processing by the MFP 1 and trained model 201 and shorten the time until the copy result is output. Note that the image types that can be designated are not limited to the three types of text, photo, and receipt, but may be two types, text and non-text, or four or more types including other types.

[0043] On the image type selection screen 70, the MFP 1 prioritizes display of the AI ​​Auto (with preview) button 71 and the AI ​​Auto (without preview) button 72 over any of the other options, namely, the Main Unit Auto button 73, the Text button 74, the Photo button 75, and the Receipt button 76. The AI ​​Auto (with preview) button 71 and the AI ​​Auto (without preview) button 72 may be displayed preferentially by being positioned higher than the other options, as shown in FIG. 3(C), for example. Alternatively, they may be displayed in a larger size than the other options, in a more noticeable color, or highlighted by flashing or a border. This makes it easier to select the AI ​​Auto (with preview) button 71 or the AI ​​Auto (without preview) button 72, and by selecting a procedure that uses the trained model 201, higher-quality copy results are likely to be obtained.

[0044] Furthermore, as shown in FIG. 3(C), the MFP 1 displays the AI ​​Auto (with preview) button 71 with priority over the AI ​​Auto (without preview) button 72. The MFP 1 displays the AI ​​Auto (with preview) button 71 larger and more prominently, for example, in a higher position than the AI ​​Auto (without preview) button 72. In other words, the MFP 1 displays the AI ​​Auto (with preview) button 71 so that it is more likely to be selected than the AI ​​Auto (without preview) button 72. The region separation results obtained by the trained model 201 may differ from the user's intention. Obtaining user confirmation through a preview makes it easier to avoid outputting scan data based on separation results that the user did not intend.

[0045] 3B, the image type button 61 of the parameter selection screen 60 displayed in A03 has "AI Auto (with preview)" selected as the default. In other words, if the parameter selection screen 60 displayed in A03 accepts an operation on the black and white copy execution button 62 or the color copy execution button 63 without accepting an operation on the image type button 61 in A04, the MFP 1 performs the same operation as when the MFP 1 accepts an operation on the black and white copy execution button 62 or the color copy execution button 63 (A11) on the parameter selection screen 60 (A07) after the AI ​​Auto (with preview) button 71 is selected on the image type selection screen 70 displayed in A05. Because "AI Auto (with preview)" is the default, the user is likely to obtain high-quality copy results simply by operating the black and white copy execution button 62 or the color copy execution button 63.

[0046] Note that a procedure that uses the trained model 201 requires processing time by the trained model 201, and therefore the overall processing time is likely to be longer than a procedure that does not use the trained model 201. Therefore, as shown in FIG. 3(C), for example, the MFP 1 associates the AI ​​Auto (with preview) button 71 and the AI ​​Auto (without preview) button 72 with each other and displays a message 77 indicating that the processing time required to complete may be longer than if other options are selected. By displaying information indicating that the processing time will be longer if the AI ​​Auto option is selected before accepting the selection, it is expected that the user's stress will be reduced if the processing time by the trained model 201 is longer. Furthermore, it is possible to suggest to the user that it would be better to select other options if the user wants to obtain copy results in a short time.

[0047] When the MFP 1 receives the user's selection on the image type selection screen 70, it returns to displaying the parameter selection screen 60 (A07). The image type button 61 on the parameter selection screen 60 displayed in A07 displays the processing procedure or image type selected by the user on the image type selection screen 70.

[0048] The user places a document on the flatbed or document feeder and issues a copy execution instruction by operating the black and white copy execution button 62 or the color copy execution button 63 on the parameter selection screen 60 (A11). Operating the black and white copy execution button 62 or the color copy execution button 63 when either the AI ​​automatic (with preview) button 71 or the AI ​​automatic (without preview) button 72 is selected is an example of a first scan instruction. Operating the black and white copy execution button 62 or the color copy execution button 63 when the AI ​​automatic (with preview) button 71 is selected is an example of a first scan instruction with a preview. Operating the black and white copy execution button 62 or the color copy execution button 63 when the AI ​​automatic (without preview) button 72 is selected is an example of a first scan instruction without a preview. Operating the black and white copy execution button 62 or the color copy execution button 63 when either the main body automatic button 73, the text button 74, the photo button 75, or the receipt button 76 is selected is an example of a second scan instruction.

[0049] Note that the MFP 1 may display an icon for accepting an instruction to perform AI automatic copying on the standby screen 50, separate from the copy icon 51. For example, when the MFP 1 accepts an operation on that icon, it may accept a selection of whether to perform with or without a preview, and operate as if the accepted processing procedure has been selected. Alternatively, when the MFP 1 accepts an operation on that icon, it may operate as if it has accepted an instruction for AI automatic copying with a preview. The icon for accepting an instruction to perform AI automatic copying, which is displayed on the standby screen, is also an example of a first icon. An operation on the icon for accepting an instruction to perform AI automatic copying, which is displayed on the standby screen, is also an example of a first scan instruction.

[0050] Furthermore, the MFP 1 may be capable of accepting setting instructions for various scanning parameters and various printing parameters in addition to processing procedures and image types. For example, when the MFP 1 accepts an operation on a button included in the parameter selection screen 60 for accepting the setting of various parameters, the MFP 1 causes the user IF 13 to display parameter options for the setting item corresponding to the button. Furthermore, when the MFP 1 accepts an operation to select one of the displayed options, the MFP 1 changes the parameters used in scanning or printing for the setting item corresponding to the operated button to the parameters corresponding to the selected option. Note that the parameters set as defaults may be values ​​stored at the time of shipment of the MFP 1, or may be values ​​that can be changed by an administrator or user operation after shipment.

[0051] Upon receiving the instruction at A11, the MFP1 drives the reading engine 16 to read the image of the document and generate scan data (A12). The MFP1 performs, for example, high-resolution color reading and generates scan data of the color image. A12 is an example of a first scan process or a second scan process, and the scan data generated at A12 is an example of the first scan data or the second scan data.

[0052] The document is prepared by the user, and a single document image may contain multiple types of partial images. For example, as shown in Figure 4, a document containing a mixture of text, photos, and receipts may be set. Even when a document containing multiple types of partial images is set, the MFP1 reads the entire document and generates one scan data item that represents a single document image containing multiple types of partial images.

[0053] If the execution instruction in A11 is AI automatic (with preview) or AI automatic (without preview) (alt: [AI automatic]), the MFP1 executes the AI ​​automatic procedure (A21). The AI ​​automatic procedure will be described with reference to FIG. 5.

[0054] The MFP1 transmits the scan data generated in A12 to the generation AI server 200 (B01). B01 is an example of a transmission process. Note that in B01, if the execution instruction of A11 is AI automatic (with preview), the MFP1 requests preview data from the generation AI server 200, and if the execution instruction of A11 is AI automatic (without preview), the MFP1 requests processed data from the generation AI server 200. The following first describes the case where AI automatic (with preview) is selected.

[0055] If the user's selection is AI automatic (with preview) (opt: [with preview]), the MFP 1 sends the scan data to the generation AI server 200 in B01 along with a predetermined instruction indicating that a preview is available. The predetermined instruction indicating that a preview is available is, for example, an instruction requesting that the trained model 201 separate images for each type of image included in the scan data and return preview data indicating the separation results. As the preview data, the MFP 1 may request, for example, image data indicating an image with a border added to surround the image area for each type of separated image.

[0056] The MFP1 may issue instructions by sending scan data to the generation AI server 200 via a dedicated API, or by a prompt. The scan data that the MFP1 sends to the generation AI server 200 may be the raw data obtained as a result of reading by the reading engine 16, i.e., RAW data, or may be data that has undergone processing such as format conversion by the MFP1.

[0057] Furthermore, the generation AI server 200 may perform various processes on the scan data received from the MFP1 without significantly altering its content before inputting it to the trained model 201. For example, the generation AI server 200 may perform filtering, such as a process that emphasizes the features of the scan data or a process that removes noise from the scan data. In this specification, the case where the scan data sent by the MFP1 is input to the trained model 201 after undergoing various processes is also considered to be included in the category of the MFP1 inputting scan data to the trained model 201.

[0058] At B01, the MFP 1 transmits the scan data to the generation AI server 200 along with a prompt indicating, for example, "Separate the image data into a photo area, a text area, and a receipt area, and frame the photo area with a blue frame, the text area with a red frame, and the receipt area with a yellow frame. Please send the separated image data in which the original image and each color frame are superimposed. The transmitted image data will be used for preview display." For example, when the trained model 201 receives the scan data obtained by reading the document shown in FIG. 4, it can generate separated scan data 9 separated into photo areas 9a and 9b, text areas 9c and 9d, and a receipt area 9e, as shown in FIG. 6. Furthermore, in response to an instruction from the MFP 1, the trained model 201 can generate preview data showing an image in which each separated area is framed, and transmit the preview data to the MFP 1.

[0059] After sending B01, when the MFP1 receives preview data from the generation AI server 200 (B11), it displays a preview screen on the user IF based on the preview data received from the generation AI server 200 (B12). The preview data is an example of separated scan data.

[0060] 7, the MFP 1 displays a preview screen 80 including a preview image 81 based on preview data received from the generation AI server 200, a redo button 82 for receiving an instruction to redo region separation using the trained model 201, a cancel button 83 for receiving an instruction to cancel copying, and an OK button 84 for receiving an instruction to execute printing. The MFP 1 then becomes capable of receiving user operations on the buttons 82, 83, and 84 on the currently displayed preview screen 80. B12 is a process for displaying the preview screen 80 and becoming capable of receiving user operations, and is an example of a preview process.

[0061] The preview image 81 is an image obtained by scanning a document and superimposing colored borders on the boundaries of each region separated by the trained model 201. The preview image 81 shown in FIG. 7 displays photo region frames 85a and 85b, text region frames 85c and 85d, and a receipt region frame 85e. Each of the frames 85a, 85b, 85c, 85d, and 85e is an example of a marker indicating the region boundary. The preview image 81 is an example of an image with markers.

[0062] By adding the above-mentioned prompt and sending the scan data to the generation AI server 200, the MFP1 is likely to receive preview data in which areas are separated by image type and borders of different colors are added to the boundaries of each area depending on the image type. However, if the MFP1 determines in B11 that the data received from the generation AI server 200 is not appropriate preview data, or if it has not received preview data even after a predetermined time has passed, it may repeat the instruction in B01.

[0063] The user can check the preview screen 80, change the position or shape of the border, and operate one of the redo button 82, cancel button 83, and OK button 84. Each border is displayed as an object separate from the image, and is displayed so as to be able to accept user operations.

[0064] When an operation to change the border line, that is, an operation to modify the boundary of the area, is received (opt:[Modify], B15), the MFP 1 changes the border line based on the user's operation and displays the preview screen 80 with the preview image 81 updated (B16). For example, when the MFP 1 receives an operation to select one of the border lines currently being displayed, the MFP 1 may be able to receive an operation to change that border line. The MFP 1 is then able to receive further user operations on the updated preview screen 80.

[0065] Furthermore, when the MFP1 accepts an operation on the OK button 84 (opt: [OK], B17), it sends a request to the generation AI server 200 to send processed data for which image processing has been performed for each region (B18). The operation on the OK button 84 is an example of a selection to output scan data, and B18 is an example of post-selection transmission processing. In B18, the MFP1 sends a prompt indicating instructions such as, "Perform optimal image processing for each separated region, and send image data in which each processed region is combined. Combine the regions so that the boundaries between them are smooth. Also, perform processing that does not degrade image quality."

[0066] If the border line has been changed by the operation of B15, the MFP1 sends a transmission request to the generation AI server 200 in B18 together with information indicating the position of the changed border line or the preview image 81 included in the updated preview screen 80, and instructs the generation AI server 200 to perform processing based on the changed area.

[0067] Upon receiving processed data that has undergone image processing by the trained model 201 from the generation AI server 200 based on the request of B18 (B21), the MFP1 executes printing based on the received processed data (B22). The processed data received in B21 is an example of first processed scan data, and B22 is an example of first output processing.

[0068] As with input data, the generation AI server 200 may perform various processes on the data output by the trained model 201 without significantly altering its content, and then transmit it to the MFP 1. In this specification, the case where the MFP 1 receives data that has been subjected to various processes after being output by the trained model 201 is also included in the category of the MFP 1 receiving data output by the trained model.

[0069] If the processed data sent from generation AI server 200 is data in which, for example, trained model 201 separates the scanned data into regions for each image type based on instructions from MFP 1 and performs image processing appropriate for the image type on each separated region, it is highly likely that a copy result appropriate for the image of the original will be obtained. Image processing appropriate for the image type may be, for example, strong noise removal and edge emphasis processing if the image type is text, weak noise removal and smoothing processing if the image type is a photograph, or strong noise removal, edge emphasis, and density adjustment if the image type is a receipt.

[0070] Furthermore, the processing corresponding to the instruction to combine the images so as to smooth the boundaries is processing that suppresses sudden changes in gray value on both sides of the boundaries, and may be, for example, smoothing processing, processing that does not remove background color or overall noise. If the trained model 201 has been trained to perform smoothing processing, it is likely that processed data that has undergone smoothing processing will be transmitted based on the instruction of B18. By having the trained model 201 smooth the boundaries of the separated regions, the boundaries of the regions are prevented from becoming noticeable during image processing.

[0071] Furthermore, processing that corresponds to an instruction not to reduce image quality is, for example, processing that does not reduce the number of pixels and processing that maintains the overall gradation of the image. For example, if the number of pixels is reduced as a result of image processing, the appearance of the copy may become rough. By issuing the instruction B18, it is highly likely that processed data that has been smoothed without reducing image quality will be sent, and therefore it is highly likely that a high-quality copy will be obtained.

[0072] When AI Automatic (with preview) is selected, the MFP 1 displays a preview screen 80 including a preview image 81, and becomes capable of accepting a user operation on an OK button 84. In other words, the MFP 1 displays the separation result obtained by the trained model 201 and obtains the user's confirmation, thereby preventing the output of a copy result based on a separation result unintended by the user.

[0073] Furthermore, the MFP 1 can accept an instruction to change the border line indicating the separation result by the trained model 201 on the preview screen 80, and when the border line is changed, it updates and displays the preview image 81. Furthermore, when the MFP 1 accepts an operation on the OK button 84 on the updated preview screen 80, it sends information on the updated preview image 81 to the generation AI server 200 and instructs it to perform image processing based on the boundary of the area according to the corrected border line, so that it is possible to expect the output of a copy result based on the separation result intended by the user.

[0074] If the MFP 1 receives an operation on the Retry button 82 on the preview screen 80, it returns to B01, resends the scan data to the generation AI server 200, and receives the preview data. Even if the same scan data and the same prompt are sent, the processing results by the trained model 201 may be different.

[0075] Furthermore, when an operation on the cancel button 83 on the preview screen 80 is accepted, the MFP 1 cancels the copy processing and returns to displaying the standby screen 50.

[0076] On the other hand, if AI Auto (without preview) is selected instead of AI Auto (with preview), the MFP 1 requests processed data, not preview data, from the generation AI server 200 in B01. The processed data is data in which the trained model 201 separates the images contained in the scan data into each type of image, and appropriate image processing is performed on each separated area.

[0077] In this case, at B01, the MFP1 sends the scanned data to the generation AI server 200 along with a prompt indicating instructions such as, "Separate the image data into a photo area, a text area, and a receipt area, perform optimal image processing for each area, and then send the image data obtained by combining the processed areas. Combine the areas so that the boundaries between them are smooth. Also, perform the processing in a way that does not degrade the image quality."

[0078] After sending this instruction and receiving processed data from the generation AI server 200 (B21), the MFP1 executes printing based on the received processed data (B22). In this case, B22 is also an example of the first output process. Because the above prompt has been sent, the MFP1 is likely to receive processed data from the generation AI server 200, in which appropriate image processing has been performed for each region by the trained model 201.

[0079] Returning to the explanation of Figure 2, if the execution instruction in A11 is neither AI automatic (with preview) nor AI automatic (without preview) (alt: [other than AI automatic]), the MFP1 does not execute processing using the generation AI server 200. In this case, the MFP1 corrects the scan data generated in A12 based on the user's instruction (A31). A31 is an example of correction processing. The data processed in A31 is an example of second-processed scan data. Furthermore, the MFP1 executes printing based on the scan data after correction processing (A32). A32 is an example of second output processing.

[0080] For example, if the MFP 1 accepts an instruction to execute A11 by pressing the text button 74 or receipt button 76 on the image type selection screen 70 shown in FIG. 3(C), the MFP 1 performs image processing corresponding to text or receipt on the entire image shown in the scan data in A31. For example, the MFP 1 performs strong noise reduction or edge enhancement. If the original image is entirely text, this processing is likely to produce an appropriate copy result. However, if the original image contains a mixture of various images, as shown in FIG. 4, the copy may end up with strangely emphasized details such as fine wrinkles on a person or thin shadows in a landscape.

[0081] Furthermore, for example, if the user operates the Photo button 75 on the image type selection screen 70 shown in FIG. 3C and then accepts the execution instruction in A11, the MFP 1 performs image processing appropriate for photographs on the entire image shown in the scan data in A31. The MFP 1 performs, for example, weak noise reduction and smoothing. If the document contains only a photograph, this processing is likely to produce an appropriate copy result. However, if the document contains a mixture of various images, as shown in FIG. 4, there is a possibility that the copy will contain blurred small numbers, for example, on a receipt.

[0082] 3C, when the MFP1 receives an instruction to execute A11 by operating the automatic main unit button 73 on the image type selection screen 70, the MFP1 analyzes the scanned data in A31, determines the type of the entire image as text, photo, or receipt, and performs image processing corresponding to the determined type. As shown in FIG. 4, in the case of a document containing a mixture of various images, regardless of the determined type, there is a possibility that inappropriate correction will occur, as described above.

[0083] In other words, in the case of a document in which the entire image is of one image type, or in the case of a user who is knowledgeable about image types, by selecting either the main unit automatic button 73, text button 74, photo button 75, or receipt button 76, the process does not require processing time by the trained model 201, so the processing time to output is shorter than with the AI ​​automatic procedure, and there is a high possibility that a copy output will be obtained quickly.

[0084] In contrast, when an operation is received on the AI ​​Auto (with preview) button 71 or the AI ​​Auto (without preview) button 72, the trained model 201 is used to separate regions by image type, and printing is performed based on processed data in which image processing appropriate for each region has been performed. This means that there is a high possibility that appropriate correction will be performed for each region. By using the trained model 201, there is a high possibility that appropriate image processing will be performed for each region, and high-quality data can be expected to be output. Furthermore, because processing is performed automatically by the generation AI server 200, the user does not need to specify the image type, and even a user who is not familiar with image types is likely to obtain appropriate copy output.

[0085] In addition to the AI ​​automatic procedure that uses the trained model 201, the MFP1 allows the user to specify the image type of the entire image themselves, so the user can select the scanning procedure according to the type of document and their own knowledge.

[0086] 3A, the MFP 1 can also accept operations on icons other than the copy icon 51. For example, when an operation on the scan icon that accepts a scan instruction is accepted, the MFP 1 can also use the trained model 201 of the generation AI server 200.

[0087] When an operation on the scan icon is accepted, the MFP 1 can accept, for example, a designation of a storage location to which the scanned data will be output, or a designation of a file format for the data to be saved. The storage location may be the memory 12 of the MFP 1, a USB memory attached to the MFP 1, or an external device or external storage with which the MFP 1 can communicate.

[0088] In response to the scan instruction, the MFP 1 may then scan the document, generate scanned data, and transmit it to the generation AI server 200, separate the data into regions according to the image type, and then instruct the generation AI server 200 to perform image processing appropriate for the image type of each region. The MFP 1 may also instruct the generation AI server 200 to output the processed data in a specified file format. Upon receiving the processed data converted into the specified file format from the generation AI server 200, the MFP 1 can output and save the processed data in a specified storage location. In particular, if the processed data is received in the specified file format, it can be saved as is, making it easier for the user to use the saved data.

[0089] As described above in detail, the MFP 1 of the embodiment transmits scan data generated by scanning a document to the generation AI server 200 that uses the trained model 201. The trained model 201 has been trained to perform both region separation based on image type and image processing appropriate for each image type of the separated region based on the input scan data. After transmitting the scan data, when the MFP 1 receives processed data from the generation AI server 200, it outputs based on the processed data. The processed data is likely to be data in which the trained model 201 has separated regions into regions by image type for the image shown in the scan data and applied image processing appropriate for the image type to each region. This increases the likelihood that an image that has been subjected to appropriate image processing will be output, even if the user does not select the image type.

[0090] The embodiments are merely examples and do not limit the present invention in any way. Therefore, the technology disclosed in this specification can naturally be improved and modified in various ways without departing from the spirit of the present invention. For example, the scanner is not limited to the MFP 1, and may be a copy machine or a fax machine as long as it has an image reading function and a communication function.

[0091] Furthermore, the display format in the embodiment is not limited to the illustrated example. For example, the types and number of icons displayed on the standby screen 50 and the shapes of the icons are not limited to the illustrated example. Furthermore, the types and shapes of buttons displayed on the parameter selection screen 60 and the image type selection screen 70 are not limited to the illustrated example.

[0092] Furthermore, in the embodiment, the MFP 1 is capable of accepting the selection of either AI automatic (with preview) or AI automatic (without preview) on the image type selection screen 70 (see FIG. 3(C)), but it may be configured to accept only one of them. Furthermore, the MFP 1 is configured to accept an operation to correct the border on the preview screen 80 (see FIG. 7), but it does not have to accept this. For example, if the user thinks that the separated area is inappropriate, the user may operate the redo button 82 to have the trained model 201 separate the area again.

[0093] Furthermore, in the embodiment, a procedure has been described in which the trained model 201 is not instructed on the type of image when separating regions or on details of image processing for each type of image. However, the MFP 1 may specifically specify the type of image and image processing to perform the processing. For example, the MFP 1 may instruct, "Please classify into photos, text, and receipts." Or, for example, the MFP 1 may instruct, "Please perform error diffusion processing on photo regions." On the other hand, if the type of image and image processing are not specified, the trained model 201 can use image types and image processing that the MFP 1 does not have.

[0094] Furthermore, in the embodiment, when an AI automatic selection is accepted, the trained model 201 is made to perform region separation and image processing for each region, but it may also be possible to select a procedure in which the trained model 201 is made to perform only region separation or only image processing for each region.

[0095] Furthermore, in the embodiment, the MFP 1 generates scan data using its own reading engine 16 and transmits the generated scan data to the generation AI server 200 via its own communication IF 14. However, the generation of scan data and the transmission of scan data may be performed by different devices. For example, scan data generated by a scanner may be received by a terminal device separate from the scanner, and transmitted to the generation AI server 200 via the terminal device's communication IF 14.

[0096] Furthermore, the generation AI server 200 is not limited to one that has the trained model 201, but may be able to use a trained model 201 owned by another server. In this case, the generation AI server 200 passes the received scan data, etc. to the other server that has the trained model 201, and transmits response data to the MFP1 based on the response from the trained model 201. In other words, the generation AI server 200 is not limited to a server provided by an AI provider such as OpenAI, but may be, for example, a dedicated server provided by the manufacturer of the MFP1.

[0097] In addition, in the embodiment, a configuration is illustrated in which the trained model 201 of the generation AI server 200 is used, but instead of the trained model 201, a configuration may also be used in which a program created based on the coding of a programmer is used.

[0098] Furthermore, in any flowchart or sequence diagram disclosed in the embodiments, the execution order of multiple processes in any multiple steps can be changed or executed in parallel as desired, as long as no contradictions occur in the processing content.

[0099] The processes disclosed in the embodiments may be executed by hardware such as a single CPU, multiple CPUs, or ASIC, or a combination thereof. The processes disclosed in the embodiments may be realized in various ways, such as a recording medium on which a program for executing the processes is recorded, or a method. [Explanation of symbols]

[0100] 1 MFP 13 User Interface 14 Communication Interface 16 reading engines 200 Generation AI Server 201 trained models

Claims

1. A reading engine and A user interface; a communication interface; A scanner comprising: The scanner A server that uses a trained model is accessible via the communication interface, and the trained model has been trained to separate an image shown in the scan data into regions based on the type of image based on the input scan data, and the trained model has been further trained to perform image processing appropriate for the type of image, The scanner further comprises: When a first scan instruction is given via the user interface, a first scanning process for scanning an original document using the reading engine and generating first scan data representing an image of the original document based on the scanning result; a transmission process of transmitting the generated first scan data to the server via the communication interface; Run After executing the transmission process, when the first processed scan data is received from the server via the communication interface, the first processed scan data is output by the trained model based on the first scan data, and the trained model has performed the image processing appropriate for the type of image on each area of ​​the image shown in the first scan data separated according to the type of image, executing a first output process for outputting based on the received first processed scan data; A scanner configured to:

2. 2. The scanner of claim 1, In the transmission process, The generated first scan data and an instruction to smooth the boundary of the region are transmitted to the server via the communication interface, and the trained model has been trained to perform a smoothing process to smooth the boundary of the region based on the type of image; The scanner executes the first output process when the trained model receives, from the server via the communication interface, the first processed scan data, which is data obtained by performing the image processing appropriate for the type of image for each area separated according to the type of image and the smoothing process for smoothing the boundaries of the separated areas for an image shown in the first scan data. A scanner configured to:

3. 3. A scanner according to claim 2, In the transmission process, the generated first scan data, the instruction to smooth the boundary of the region, and an instruction not to reduce the number of pixels from the first scan data are transmitted to the server via the communication interface, and the trained model has been trained to be able to perform the smoothing process; The scanner When the trained model receives the first processed scan data from the server via the communication interface, the first processed scan data is data obtained by performing, without reducing the number of pixels, the image processing appropriate for the type of image and the smoothing processing that smooths the boundaries of the separated regions for each region separated according to the type of image for an image shown in the first scan data. A scanner configured to:

4. 2. The scanner of claim 1, The scanner the first scan instruction and the second scan instruction can be received via the user interface, and for the second scan instruction, the type of image for the entire image of the document can be specified by an operation via the user interface, and for the first scan instruction, the type of image cannot be specified; When the second scan instruction specifying the type of image is given via the user interface, a second scan process for scanning an original document using the reading engine and generating second scan data representing an image of the original document based on the scan result; a correction process for performing image processing corresponding to a designated image type on the entire image shown in the generated second scan data; a second output process for performing output based on second-processed scan data, which is data that has been subjected to image processing by the correction process; Run executes the first scan process and the transmission process when the first scan instruction is received without receiving an operation to select an image type via the user interface; and after executing the transmission process, when the first processed scan data is received from the server via the communication interface, executing the first output process. A scanner configured to:

5. 5. A scanner according to claim 4, The scanner The image types that can be specified in the second scan instruction include at least text, non-text, and automatic determination, The scanner further comprises: When the second scan instruction is given via the user interface and specifies text as the type of image, the correction process applies image processing suitable for text to the entire image shown in the generated second scan data, and the second output process performs output based on the second processed scan data, which is data that has been subjected to the image processing suitable for text; When the second scan instruction is given via the user interface and a type of image other than text is specified, the correction process applies image processing suitable for the type of image other than text to the entire image shown in the generated second scan data, and the second output process performs output based on the second processed scan data, which is data that has been subjected to the image processing suitable for the type of image other than text. When the second scan instruction is given via the user interface, specifying automatic determination as the type of image, the type of image is determined based on the image shown in the generated second scan data, and the correction process performs image processing suitable for the determined type of image on the entire image shown in the generated second scan data, and the second output process performs output based on the second processed scan data, which is data that has been subjected to the image processing suitable for the determined type of image. A scanner configured to:

6. 5. A scanner according to claim 4, The scanner displaying a screen including a first icon for receiving the first scan instruction and a second icon for receiving the second scan instruction using the user interface, and displaying the first icon with priority over the second icon; when an operation on the first icon is accepted without accepting an operation on the second icon, the first scan process and the transmission process are executed, and thereafter, when the first processed scan data is received from the server via the communication interface, the first output process is executed. A scanner configured to:

7. 7. A scanner according to claim 6, The scanner displaying the first icon with priority over the second icon, and displaying, in association with the first icon, information indicating that a processing time may be longer than when the second icon is selected. A scanner configured to:

8. 8. A scanner according to claim 7, The first scan instruction includes a first scan instruction with a preview and a first scan instruction without a preview, The scanner The first scan instruction with preview, the first scan instruction without preview, and the second scan instruction can be received via the user interface; When an instruction for the first scan with preview is given via the user interface, the first scan process and the transmission process are executed; After executing the transmission process, when the separated scan data is received from the server via the communication interface, the separated scan data being separated scan data output by the trained model based on the first scan data, the separated scan data being data in which the trained model indicates with markers boundaries of areas separated for each image type for an image shown in the first scan data, the preview process displays, using the user interface, a marked image in which the markers are added to an image of an original document based on the received separated scan data, and accepts a selection as to whether or not to output the scan data; when the selection to output the scan data is accepted in the preview process, a post-selection transmission process is executed to transmit a transmission request for the first processed scan data to the server via the communication interface; After the post-selection transmission process is executed, when the first processed scan data is received from the server, the first output process is executed; When an instruction for the first scan without preview is given via the user interface, the first scan process and the transmission process are executed; and after executing the transmission process, when the first processed scan data is received from the server via the communication interface without receiving the separated scan data, executes the first output process. A scanner configured to:

9. 9. A scanner according to claim 8, The scanner displaying, using the user interface, the screen including, as the first icons, a preview icon for receiving the instruction for the first scan with a preview and a no preview icon for receiving the instruction for the first scan without a preview, and the preview icon is displayed with priority over the no preview icon; A scanner configured to:

10. 9. A scanner according to claim 8, In the preview process, displaying the image with the markers using the user interface based on the received separated scan data, accepting a change to the boundary indicated by the markers and the selection of whether or not to output the scan data, and updating the image with the markers when the change to the boundary is accepted; The scanner When the selection to output the scan data is accepted after the change of the boundary is accepted, in the post-selection transmission process, the transmission request of the first processed scan data is associated with the updated marked image and instructed to the server; After executing the post-selection transmission process, when the trained model receives from the server the first processed scan data, the first processed scan data being data obtained by performing the image processing appropriate for the type of image for each area separated according to the type of image based on the image shown in the first scan data and the updated image with markers, the first output process is executed. A scanner configured to:

11. 2. The scanner of claim 1, The scanner When the first scan instruction is received via the user interface, the first scan process and the transmission process are executed; After executing the transmission process, when the separated scan data is received from the server via the communication interface, the separated scan data being separated scan data output by the trained model based on the first scan data, the separated scan data being data in which the trained model indicates with markers boundaries of areas separated for each image type for an image shown in the first scan data, the preview process displays, using the user interface, a marked image in which the markers are added to an image of an original document based on the received separated scan data, and accepts a selection as to whether or not to output the scan data; when the selection to output the scan data is accepted in the preview process, a post-selection transmission process is executed to transmit a transmission request for the first processed scan data to the server via the communication interface; and after executing the post-selection transmission process, when the first processed scan data is received from the server, executing the first output process. A scanner configured to:

12. 12. A scanner according to claim 11, In the preview process, displaying the image with the markers using the user interface based on the received separated scan data, accepting a change to the boundary indicated by the markers and the selection of whether or not to output the scan data, and updating the image with the markers when the change to the boundary is accepted; The scanner When the selection to output the scan data is accepted after the change of the boundary is accepted, in the post-selection transmission process, the transmission request of the first processed scan data is associated with the updated marked image and instructed to the server; After executing the post-selection transmission process, when the trained model receives from the server the first processed scan data, the first processed scan data being data obtained by performing the image processing appropriate for the type of image for each area separated according to the type of image based on the image shown in the first scan data and the updated image with markers, the first output process is executed. A scanner configured to:

13. 2. The scanner of claim 1, The scanner A designation of a storage location to be an output destination can be accepted via the user interface; In the first output process, outputting and storing the received first processed scan data in the designated storage location; A scanner configured to:

14. 14. A scanner according to claim 13, The scanner A file format for saving the first processed scan data can be specified via the user interface; In the first output process, converting the received first processed scan data into a specified file format, and outputting and saving the converted first processed scan data in the specified saving location; A scanner configured to:

15. A scanner according to any one of claims 1 to 14, Equipped with a print engine, In the first output process, performing printing based on the received first processed scan data using the print engine; A scanner configured to:

Images