Image forming apparatus, program, and control method

Abstract

To solve such a problem that image abnormalities include small point-like or line-like abnormalities, and it is difficult for a user to accurately specify a region of an image abnormality on a screen.SOLUTION: An image forming apparatus according to the present invention receives information on a position specified by a user via a screen on which an image obtained by scanning a printed matter is displayed, specifies at least one candidate region among the candidate regions of image abnormalities as a region of the image abnormality intended to be specified by the user on the basis of one or more candidate regions of image abnormalities and respective confidence levels obtained by analysis of the image, and the information on the position specified by the user, and displays information corresponding to the at least one candidate region that has been specified.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an apparatus, a program, and a method for easily identifying an abnormal area in an image, which is specified by a user. [Background technology]

[0002] Failure or contamination of parts of an image forming device may cause print abnormalities in the printed material produced by the image forming device. In such cases, the scanned image obtained by scanning the printed material also contains defects (hereinafter referred to as image abnormalities). Therefore, image diagnostic services have been developed that analyze scanned images containing image abnormalities to identify the faulty part causing the image abnormality.

[0003] Patent Document 1 discloses that when an abnormality occurs in a printed document, a test print is made using an MFP, the test print is scanned using the MFP, and the scanned image is sent to a server. The server then compares the received scanned image with the normal character pattern used in the test print to identify the type of defect and how to repair it. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-043867 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-119269 Summary of the Invention [Problem to be solved by the invention]

[0005] When performing image diagnosis by performing a test print, as in Patent Document 1, the normal pattern used in the test print can be used as a correct image, so the type of abnormality can be determined by comparing the scanned image of the printed material with the correct image. However, Patent Document 1 requires the time and effort of printing a test pattern. Therefore, the present applicant is considering a system that allows a user of an image forming device to perform image diagnosis using the printed material as is, without using a test pattern, when they discover an abnormality in a printed material obtained by printing based on print data prepared by the user. However, when a user discovers an abnormality in a printed material printed based on the user's print data, the print data is often already deleted from the image forming device, making it impossible to use the print data as correct image data for comparison. Therefore, the present applicant is considering a system that allows a user to specify an area in a scanned image obtained by scanning a printed material where the user found an abnormality, and then performs image diagnosis on the specified image. Patent Document 2 discloses a method in which a user specifies an area in image data, and the cause of the abnormal image contained in the specified area is estimated based on image feature information contained in the specified area.

[0006] On the other hand, image anomalies can be small, such as dots or lines, making it difficult for users to accurately specify the area of ​​the image anomaly on the screen. [Means for solving the problem]

[0007] In order to solve the above problem, the image forming apparatus of the present invention is characterized by comprising: a receiving means for receiving information on a position specified by a user via a screen on which an image obtained by scanning a printed material is displayed; an identifying means for identifying at least one of the candidate areas for image abnormalities as the area of ​​image abnormality that the user intended to specify based on one or more candidate areas for image abnormalities and their certainty levels obtained by analyzing the image and the information on the position specified by the user; and a display means for controlling the display of information corresponding to at least one candidate area identified by the identifying means. [Effects of the Invention]

[0008] According to the present invention, a region in an image displayed on a screen where an image abnormality occurs, which the user intends to specify, can be identified by a simple input operation by the user. [Brief explanation of the drawings]

[0009] [Figure 1] System configuration diagram of the first embodiment [Figure 2] Hardware configuration diagram of image forming device [Figure 3] Flowchart of the image forming apparatus in the first embodiment [Figure 4] User image data before and after diagnostic imaging [Figure 5] Image anomaly detection results [Figure 6] An example of an image abnormality location input screen [Figure 7] Flowchart for identifying image anomaly candidates in the first embodiment [Figure 8] Illustration of image anomaly candidate identification [Figure 9] Confidence distance calculation result [Figure 10] An example of an image abnormality candidate confirmation screen [Figure 11] An example of a screen for checking detected image abnormalities [Figure 12] An example of an image diagnosis result screen [Figure 13]An example of a user printout in which abnormal lines of the same cause appeared side by side [Figure 14] Flowchart for identifying image anomaly candidates in the second embodiment [Figure 15] Illustration of image anomaly candidate identification DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following, printed matter printed by an image forming device based on a user's print data (print data based on documents, images, etc. created by the user) will be referred to as user printed matter, and the image obtained by reading the user printed matter with a scanner will be referred to as user image data. [Example]

[0012] FIG. 1 is a system configuration diagram of a first embodiment, which includes an image forming apparatus 102, an image diagnostic apparatus 103, a sales company's management server 104, and a network 101. The image forming apparatus 102 is, for example, a digital multifunction peripheral (MFP) having a print function and a scan function. The image diagnostic apparatus 103 performs image diagnosis on image data transmitted from the image forming apparatus 102 via the network 101 and transmits an image abnormality detection result to the image forming apparatus 102. The image diagnostic apparatus 103 may be a server device connected to the network, a virtual server on a cloud system, or may be included in the image forming apparatus 102. The sales company's management server 104 manages customer information to which the sales company provides services, and information such as the model number and fault location of the image forming apparatus 102 transmitted from the image forming apparatus 102 via the network 101.

[0013] The hardware configuration of an image forming apparatus 102 according to an embodiment of the present invention will be described with reference to Fig. 2. The image forming apparatus 102 includes a CPU 201, a ROM 203, a RAM 204, a network interface card 205, an external memory 206, an operation panel 207, a storage device 208, a device interface 209, a printer 210, and a scanner 202. These components are connected via a system bus 210.

[0014] A CPU (processor) 201 comprehensively controls access to various devices connected to a system bus 210. The CPU 201 functions as a processing unit for executing various processes of the image forming apparatus 102, which will be described later, by reading out control programs and resource data (resource information) stored in a ROM 203 or an external memory 206 into a RAM 204 and executing them.

[0015] The ROM 203 stores programs such as a basic I / O program, a program for receiving instructions from a user related to image diagnosis, various data, etc. The RAM 204 functions as the main memory, work area, etc. of the CPU 201, and is configured so that the memory capacity can be expanded by an optional RAM connected to an expansion port (not shown).

[0016] The network interface card 205 is an interface with external devices, and the image forming apparatus 102 exchanges data with external devices such as the image diagnostic apparatus 103 and the management server 104 via the network interface card 205. The operation panel 207 is configured with a liquid crystal touch panel for displaying an operation screen and accepting operation instructions from a user via the operation screen. Note that the operation panel 207 may not only be an LCD touch panel, but may also be provided with separate physical buttons for setting the operation mode of the image forming apparatus, setting the number of copies or prints, issuing start instructions, etc.

[0017] The storage device 208 is an external storage means that functions as a large-capacity memory. The device interface 209 is a connection interface with an external device that can be connected via USB or the like. The printer 210 prints print data converted into a PDL (Page Description Language) language or image data converted from a PDF (Portable Document Format) file or the like onto paper. The printer 210 uses a known printing function and may be any type, such as an electrophotographic system (laser beam system), an inkjet system, or a dye sublimation (thermal transfer) system.

[0018] The scanner 202 utilizes a known image reading function, for example, optically scanning a paper document (printed material) placed on a transparent top plate and converting it into image data. It may also have a function to continuously read multiple paper documents placed on an automatic document feeder (ADF) and convert them into image data. The image data (scanned image) is temporarily stored in a storage area such as the storage device 208, RAM 1004, or cache. When performing image diagnosis, the CPU 201 of the image forming device 102 transmits the scanned image to the image diagnosis device 103 via the network 101.

[0019] The diagnostic imaging device 103 performs an analysis process on the scanned image to detect candidate image abnormalities in the scanned image, and detects the type and location of the image abnormality contained in the scanned image. Furthermore, the diagnostic imaging device 103 can also perform an image diagnosis process to identify the faulty part that caused the image abnormality based on the location and type of the image abnormality identified as the image abnormality. Note that in this embodiment, candidate image abnormalities are detected from the scanned image without comparing the scanned image with a correct image. Therefore, depending on the printed content of the printed matter, normally printed areas may be erroneously detected as candidate image abnormalities.

[0020] FIG. 3 is a flowchart showing the flow of operational control of the image forming apparatus 102 in this embodiment. Hereinafter, the step numbers of each process included in the flowchart are indicated by numbers beginning with "S." In this embodiment, when a user finds an image abnormality in a printed document printed by the printer 210 of the image forming apparatus 102, the scanned image obtained by reading the printed document using the scanner 202 of the image forming apparatus 102 is used for image diagnosis. When image diagnosis is selected from a menu (not shown) displayed on the operation panel 207 and an instruction to start image diagnosis is received from the image diagnosis execution screen, the CPU 201 of the image forming apparatus 102 starts processing from step S301. The user places the printed document with the image abnormality in the scanner 202 as the printed document to be diagnosed, and issues an instruction to start image diagnosis.

[0021] In S301, the CPU 201 reads a user's printed material that is the subject of image diagnosis using the scanner 202, and acquires user image data 401. The user image data 401 is stored in the storage device 208.

[0022] FIG. 4A shows an example of user image data 401, which is the target of image diagnosis and has been converted by the CPU 201 in S301. Image anomaly 402 is a linear image anomaly (e.g., a black line or a line of another color) that occurred on a user printout in the image forming apparatus 102. User image content 403 and 404 are examples of content that were originally included as image content to be printed on the user printout and have shapes similar to point-like and linear image anomalies. For example, in the example shown in FIG. 4A, user image content 403 is a mole on a person's face. User image content 404 is a prolonged vowel in the example shown in FIG. 4A.

[0023] In S302, the CPU 201 transmits the user image data 401 to the image diagnostic apparatus 103 via the network interface card 205, and requests image diagnosis.

[0024] In S303, the CPU 201 acquires the image abnormality detection result (for example, the detection result data shown in FIG. 5) from the image diagnostic apparatus 103 via the network interface card 205. The image abnormality detection result is stored in the storage device 208 by the CPU 201.

[0025] FIG. 5 is table-format data showing the detection results of image abnormalities output when the diagnostic imaging device 103 performs image diagnosis on the user image data 401, obtained in S303. Hereinafter, a location detected as a candidate for an image abnormality when the diagnostic imaging device 103 performs image diagnosis on the user image data 401 will be referred to as a detected image abnormality. The detected image abnormality ID is a unique identifier assigned to each detected image abnormality. The image abnormality type indicates the type of image abnormality determined, such as a linear image abnormality (hereinafter referred to as a streak) or a point-like image abnormality (hereinafter referred to as a spot). The fault location is a part name indicating the fault location determined by the diagnostic imaging device 103 to be the cause of the image abnormality based on the type and position of the image abnormality. The countermeasure / message is information on how to deal with the fault location determined by the diagnostic imaging device 103 to be the cause of the image abnormality, and information on a message to be notified to the user. The "Notify sales company" is information indicating whether or not the sales company needs to be notified of the fault location and countermeasure information determined by the diagnostic imaging device 103. The X and Y coordinates are the coordinates (pixel values) of the top left corner of the detected image anomaly. The height and width are the height and width (pixel values) of the detected image anomaly. The certainty factor indicates the likelihood of the detection result for each detected image anomaly, and is expressed, for example, as a value between 0 and 1, with the closer to 1 the higher the probability of an image anomaly. In other words, the certainty factor indicates the likelihood that an image anomaly has occurred within the candidate area for an image anomaly (the area indicated by the X and Y coordinates, width, and height above) as a result of analyzing the scanned image with an imaging diagnostic device. Note that while Figure 5 shows the image anomaly detection results in a table format, the data format is not limited to a table format.

[0026] 4(B) is a diagram for explaining which positions in FIG. 4(A) correspond to the image abnormality detection results in FIG. 5 obtained in S303. Bounding boxes 412, 413, and 414 on user image data 411 indicate the positions of areas (detected image abnormality IDs: 1 to 3 in FIG. 5) detected as image abnormality candidates by the image diagnostic device 103 analyzing the user image data 401. Bounding box 412 indicates the position of image abnormality 402 detected by the image diagnostic device 103. Bounding box 413 indicates the position of user image content 403 erroneously detected as an image abnormality by the image diagnostic device 103. Bounding box 414 indicates the position of user image content 404 erroneously detected as an image abnormality by the image diagnostic device 103.

[0027] Typically, a user recognizes the original data printed on a printed matter, and can identify where an image abnormality occurs when viewing the printed matter. Therefore, in S304, the CPU 201 displays a position input screen 601 on the operation panel 207 to allow the user to specify the location on the scanned image where the user has determined that an image abnormality has occurred. The input coordinates entered by the user are stored in the storage device 208 by the CPU 201.

[0028] FIG. 6 shows an example of a display of a position input screen 601 that allows the user to input the position where the user determined that an image anomaly has occurred with a single touch on the screen in S304. The image anomaly position input screen 601 includes a preview screen 602 that displays a preview of the user image data 401, a Done button 603, and a Retry button 604. When the user inputs the position of the image anomaly with a single touch on the preview screen 602, a red cross appears at the input coordinate position, making it recognizable to the user. Note that the mark that appears at the input coordinate position where the user touched the screen does not have to be a red cross; any format is acceptable as long as the user can recognize the input coordinate position. The Done button 603 cannot be selected until the user inputs the position of the image anomaly with a single touch. It becomes selectable after the user inputs the position of the image anomaly with a single touch. When the Done button 603 is selected, the input coordinate position is confirmed as the position of the image anomaly that the user wants to point out, and is stored in the storage device 208 by the CPU 201. The Redo button 604 cannot be selected until the user inputs the position of the image abnormality with a single touch, and becomes selectable after the user inputs the position of the image abnormality with a single touch. When the Redo button 604 is selected, the red cross mark indicating the input coordinate position on the preview screen 602 disappears, and the position of the image abnormality can be input again with a single touch. Note that the Done button 603 and the Redo button 604 may be initially displayed grayed out, and may be configured to accept operation from the user after the user inputs the position of the image abnormality.

[0029] In S305, the CPU 201 performs a process of identifying a candidate image abnormality designated by the user from among the detected image abnormalities, based on the detection result of the image abnormality acquired in S303 and the input coordinates input by the user accepted in S304. That is, from among the candidate image abnormalities detected by the image diagnostic device (FIG. 5), it identifies a candidate that is located near the input coordinates designated by the user and has a high degree of certainty as an image abnormality.

[0030] FIG. 7 is a flowchart detailing the process of identifying image anomaly candidates in step S305 of this embodiment. When identifying image anomaly candidates, an error (hereinafter referred to as an input error) may occur between the image anomaly that the user intends to designate on the preview screen 602 displaying the user's image and the input coordinates that the user actually designates with a single touch. Therefore, the input coordinates may fall outside any of the multiple image anomaly candidate areas (bounding boxes 412 to 414 in FIG. 4A) detected by the image diagnostic device. On the other hand, if a candidate image anomaly detected by the image diagnostic device exists near the input coordinates designated by the user, that candidate image anomaly is likely to be the one the user intends to designate. However, if multiple image anomaly candidates detected by the image diagnostic device exist near the input coordinates designated by the user, it is necessary to determine which of the multiple candidates is the image anomaly the user intends to designate. Therefore, in this embodiment, the candidate image anomaly candidate area that the user intends to designate is identified using the input error between each of the multiple image anomaly candidates detected by the image diagnostic device and the input coordinates, as well as the confidence level of each of the multiple detected image anomaly candidates. In this embodiment, the input error is described as the shortest distance between a bounding box indicating a candidate area for an image abnormality and the input coordinates. However, this is not limited to this, and it may be, for example, the distance between the center of gravity of the bounding box and the input coordinates. If the input coordinates are within the bounding box, the input error is set to 0. Since a candidate area for an image abnormality located significantly far from the input coordinates is considered not to be the image abnormality intended by the user, it is sufficient to determine candidate image abnormalities within a predetermined range from the input coordinates. Hereinafter, this predetermined range will be referred to as the maximum input error. The maximum input error may be a default value preset in the image forming device 102 or a value set by the user.

[0031] FIG. 8 is a diagram for explaining the relationship between input coordinates and input error. The input coordinate 801 is an input coordinate input by a user with a single touch. The circle 802 indicates a range centered on the input coordinate 801 and with a radius of the maximum input error r0. Also, assuming that the distances (i.e., input errors) between each of the bounding boxes 803 to 804 indicating the positions of candidates for image abnormalities detected by the image diagnostic apparatus and the input coordinate 801 are r1 to r3, respectively. In FIG. 8, the input error between each bounding box and the input coordinate is taken as the shortest distance between the bounding box and the input coordinate. Also, in FIG. 8, assuming that the magnitude relationship of the four distances is r2 < r1 < r0 < r3. That is, in FIG. 8, the input errors of the bounding boxes 803 and 804 are within the maximum input error r0, and the input error of the bounding box 805 is greater than the maximum input error r0. In this case, the bounding box 805 located at a position farther than the maximum input error r0 is considered not to be the image abnormality that the user intended to specify.

[0032] Also, even if the input coordinate is within the bounding box indicating the position of the candidate for the image abnormality detected by the image diagnostic apparatus, if the confidence level of the bounding box is low, the possibility that the detected image abnormality is the image abnormality that the user intended to specify is low. Therefore, in this embodiment, the candidates for the image abnormality detected by the image diagnostic apparatus with a confidence level equal to or higher than a predetermined minimum confidence level are set as the processing targets in S702 to S703, and those with a confidence level lower than the minimum confidence level are excluded. The minimum confidence level may be a default value preset in the image forming apparatus 102 or a value set by the user.

[0033] In S701, the CPU 201 extracts all the detected image abnormalities from the detected image abnormalities detected by the image diagnostic apparatus that have a confidence level equal to or higher than the minimum confidence level and an input error within the maximum input error, and makes them the processing targets in S702 to S703. The extracted detected image abnormalities are stored in the storage device 208 by the CPU 201.

[0034] In S702, for each detected image anomaly processed in S701, a confidence distance R is calculated as an index for determining whether or not the detected image anomaly is the image anomaly intended by the user. In this embodiment, evaluation is performed taking into consideration both the confidence level and the input error of each detected image anomaly. For example, the confidence distance R is calculated using the following formula, where p is the confidence level, r is the input error, and a is a predetermined constant. Note that the input error may be in pixels and may be calculated using Pythagoras' theorem, for example. The smaller the value of the confidence distance R shown in the formula below, the more likely it is that the image anomaly intended by the user is.

[0035]

number

[0036] Here, the constant a is calculated using the following formula, where the minimum confidence is p0 and the maximum input error is r0.

[0037]

number

[0038] In this embodiment, the above-mentioned confidence distance R is defined as an index for determining which of the multiple detected image abnormalities detected by the imaging diagnostic device the image abnormality that the user is attempting to specify is, but another judgment index may also be used.

[0039] 9 shows an example of the confidence distance calculation result when the processes of S701 and S702 are performed on the image anomaly detection results of FIG. 5 with a minimum confidence level of 0.5 and a maximum input error of 50 pixels. In S701, the distance (input error) from the input coordinate is calculated for each of detected image anomaly IDs 1 to 3, and detected image anomaly IDs 1 to 2, which have a maximum input error of 50 or less and a minimum confidence level of 0.5 or more, are determined to be the target for processing in S702. Then, in S702, the confidence distance is calculated for each detected image anomaly, resulting in the results shown in FIG. 9. In FIG. 9, the information on the input error calculated in S701 and the information on the confidence distance calculated in S702 are added to the information on detected image anomaly IDs 1 to 2 in FIG. 5, and the information on the X coordinate, Y coordinate, width, and height in FIG. 5 is omitted.

[0040] In S703, the CPU 201 adds the detected image anomaly with the smallest confidence distance from the confidence distance calculation results to the image anomaly candidate to be displayed. The image anomaly candidate to be displayed is stored in the storage device 208 by the CPU 201. Note that if the confidence distances of the detected image anomaly with the smallest confidence distance and the detected image anomaly with the next smallest confidence distance are close to each other, both may be added to the image anomaly candidate to be displayed. For example, all detected image anomalies with confidence distances within 1.05 times the smallest confidence distance may be added to the image anomaly candidate to be displayed.

[0041] In S306, the CPU 201 determines whether there is one or more image abnormality candidates to be displayed as determined in S305 (S701 to S703). If it is determined in S306 that there is one or more image abnormality candidates to be displayed (yes), the process proceeds to S307. If it is determined in S306 that there is no image abnormality candidate to be displayed (no), the process proceeds to S312.

[0042] In S307, the CPU 201 displays a confirmation screen 1001 on the operation panel 207, including the location of the image anomaly candidate to be displayed, to prompt the user for confirmation. FIG. 10A shows an example of the confirmation screen 1001 displayed in S307. The confirmation screen 1001 includes an area confirmation screen 1002, a YES button 1003, and a NO button 1004. The area confirmation screen 1002 displays a bounding box indicating the location of the area of ​​the image anomaly candidate to be displayed, superimposed on the user image data 401. When the YES button 1003 is selected, the image anomaly candidate to be displayed corresponding to the displayed bounding box is confirmed as the image anomaly intended by the user, and data of the image anomaly candidate to be displayed is stored in the storage device 208. Hereinafter, the detected image anomaly confirmed as the image anomaly intended by the user will be referred to as a confirmed image anomaly. When the NO button 1004 is selected, the image anomaly candidate to be displayed is determined not to be the image anomaly intended by the user.

[0043] If S306 determines that there are multiple image anomaly candidates to be displayed, the positions of all of the image anomaly candidates to be displayed are displayed on the confirmation screen in S307. However, this does not necessarily mean that all of the image anomalies are the ones the user intended to specify. Therefore, the user may be able to exclude any image anomaly that he or she does not desire from the multiple image anomaly candidates displayed in S307. FIG. 10B shows an example of a confirmation screen 1011 that presents multiple image anomaly candidates to be displayed to the user in S307 and prompts the user to confirm and select one. The confirmation screen 1011 includes a region confirmation screen 1012, a YES button 1013, and a NO button 1014. The region confirmation screen 1012 displays bounding boxes indicating the positions of the multiple image anomaly candidates to be displayed determined in S305 on the user image data. When the user selects one of the displayed bounding boxes that he or she determines is not an image anomaly, the border of the selected bounding box may be displayed in a lighter color, indicating that the bounding box has been excluded from the selection. Note that if the user selects a bounding box whose border color has faded and been excluded from the target list, the border color of the bounding box may be restored to its original color, allowing it to be returned to the target list. In the above example, the confirmation screen 1012 changes the border color of the bounding box in response to a user selection operation. However, the border color is not limited to the color of the border; any other method may be used to indicate whether the bounding box is a target or not. For example, when a bounding box is selected, the border of the bounding box may be dotted. If the YES button 1013 is selected after the user confirms or selects a bounding box on the confirmation screen of FIG. 10(B), the detected image anomaly corresponding to the bounding box selected on the region confirmation screen 1012 is confirmed as a confirmed image anomaly. The CPU 201 then stores the confirmed image anomaly data in the storage device 208. On the other hand, if the NO button 1014 is selected, the multiple image anomaly candidates to be displayed are determined not to be the image anomaly the user intended to specify.

[0044] In the example of FIG. 10(A) described above, the image abnormality candidates identified in S305 are displayed. However, a button (not shown) for confirming all of the image abnormality detection results detected by the diagnostic imaging device may be provided separately on the screen of FIG. 10(A). When this button is operated, a confirmation screen 1101 is displayed, as shown in FIG. 11, on which all of the detected image abnormalities detected by the diagnostic imaging device and acquired from the diagnostic imaging device in S303 can be confirmed. The confirmation screen 1101 is composed of a detected image abnormality display screen 1102 and an OK button 1103. The detected image abnormality display screen 1102 displays the user image data 401 superimposed with bounding boxes indicating the positions of the detected image abnormalities detected by the diagnostic imaging device. When the OK button 1103 is selected, the display of the detected image abnormality confirmation screen 1101 of FIG. 11 is terminated.

[0045] In S308, the CPU 201 determines whether or not the result of the check in S307 is stored as a final image abnormality in the storage device 208. If it is determined in S308 that there is a final image abnormality (yes), the process proceeds to S309. If it is determined in S308 that there is no final image abnormality (no), the process proceeds to S312.

[0046] In S309, the CPU 201 acquires information on a method of dealing with the confirmed image abnormality and a message, and displays an image diagnosis result screen, with reference to Fig. 9. The image diagnosis result screen may be a screen that shows the content of future measures based on the image diagnosis result.

[0047] For example, if the solution to the confirmed image abnormality is to send a part, the CPU 201 displays an image diagnosis result screen 1201 notifying the user that a replacement part will be sent on the operation panel 207, as shown in Fig. 12(A). For example, the image diagnosis result screen 1201 displays the name of the part that needs to be replaced as a result of the image diagnosis, and a message indicating that the part will be sent from the sales company.

[0048] Furthermore, when the remedy for the confirmed image abnormality is cleaning, the CPU 201 displays an image diagnosis result screen 1211 on the operation panel 207 to notify the user of the cleaning locations and cleaning procedures, as shown in FIG. 12(B). For example, the image diagnosis result screen 1211 displays the cleaning locations determined to need cleaning by the image diagnosis and a cleaning method display button 1212. When the cleaning method display button 1212 is selected, a cleaning method for the locations determined to need cleaning by the image diagnosis is displayed. Note that, instead of the cleaning method display button 1212, the image diagnosis result screen 1211 may display a message indicating the cleaning method, or may display the URL of a website where the cleaning method can be viewed.

[0049] In S310, the CPU 201 determines, with reference to FIG. 9, whether or not it is necessary to notify the sales company of the image diagnosis result for the confirmed image abnormality. For example, if the image abnormality with detected image abnormality ID=2 in FIG. 9 is determined to be a confirmed image abnormality, the solution is for the user to clean the image, and notification to the sales company is set to be unnecessary. Therefore, in this case, the CPU 201 determines that it is unnecessary to send the image diagnosis result to the sales company, and ends the process. On the other hand, if the image abnormality with detected image abnormality ID=1 in FIG. 9 is determined to be a confirmed image abnormality, the solution is to send a part, and the part must be sent from the sales company, and notification to the sales company is set to be necessary. Therefore, in this case, the CPU 201 determines that it is necessary to send the image diagnosis result to the sales company, and the process proceeds to S311.

[0050] In S311, the CPU 201 transmits the image diagnosis results (information on the location of the failure and how to deal with it) to the management server 104 managed by the sales company via the network interface card 205. It is desirable to control the transmission of user image data to the management server as the image diagnosis results in addition to information on confirmed image abnormalities (information on the failed part, whether or not a part needs to be sent, whether or not a service technician needs to be dispatched to the user's home, etc.).

[0051] In S312, the CPU 201 notifies the sales company that an image abnormality has occurred, and displays an image diagnosis result screen 1221 on the operation panel 207, as shown in Figure 12 (C), which includes a message indicating that the sales company will arrange for a visit or contact with the user.

[0052] Then, in S313, CPU 201 transmits information indicating that an image abnormality has occurred to sales company's management server 104 via network interface card 205. When notifying the sales company, it is desirable that the user image data and information on the input coordinates input by the user in S304 be transmitted to the sales company's management server so that the sales company's personnel can view the image data in which the image abnormality has occurred in advance.

[0053] According to this embodiment, image diagnosis can be performed without printing a test pattern. Furthermore, even if the user does not precisely specify the area where the image abnormality occurs, the image abnormality that the user intended to specify can be identified based on the area determined by the image diagnostic device as a candidate for the image abnormality. In particular, since the operation panel of the image forming device is small, the user does not need to precisely specify the location of the image abnormality area on the operation panel. Furthermore, if the solution to the image abnormality can be achieved by sending a replacement part or by the user cleaning the part, this can be notified on the screen, thereby reducing the cost of having a sales company service technician visit the user.

[0054] In the above example, when the user touches the position of an image abnormality in one position in S304 of Fig. 3, the input coordinates are accepted. The acceptance of this input coordinate is not limited to one position, and when image abnormalities occur in multiple positions on the scanned image, input of multiple positions may be accepted from the user. When input of image abnormalities in multiple positions is accepted, the process of S305 (S701 to S703) may be performed for each of the multiple input coordinates.

[0055] In addition, in the above-described embodiment, an example was described in which input coordinates were accepted by touch input from the user, but this is not limited to touch input, and input coordinates may be accepted by operation via a pointing device such as a mouse. [Example]

[0056] In the first embodiment, when an image abnormality occurs in the image forming apparatus, the CPU 201 calculates the confidence distance of the detected image abnormality in S702, and adds the detected image abnormality with the smallest confidence distance to the image abnormality candidates to be displayed in S703. However, since there are cases where image abnormalities that appear in printed matter are caused by the same cause (for example, streak-like image abnormalities) and appear side by side at extremely close distances, in the second embodiment, control is performed so that these are all added to the image abnormality candidates to be displayed.

[0057] 13 shows an example of a user's printout 1301 in which streaks of an image anomaly caused by the same cause appear side by side at extremely close distances. Image anomaly 1302 is an image anomaly in which streaks of an image anomaly caused by the same cause appear side by side at extremely close distances.

[0058] In the first embodiment, the CPU 201 adds only the detected image anomaly with the smallest confidence distance among the plurality of detected image anomalies to the image anomaly candidates to be displayed in S703. On the other hand, in the second embodiment, the CPU 201 groups the detected image anomaly with the smallest confidence distance among the plurality of detected image anomalies with a detected image anomaly that is located nearby and is determined to have the same fault location, and adds the resulting group to the image anomaly candidates to be displayed. That is, in the second embodiment, the processing content in S305 (processing to identify which of the detected image anomalies detected by the imaging diagnostic device corresponds to the image anomaly that the user is attempting to specify) is different.

[0059] FIG. 14 is a flowchart showing details of the process of S305 in the second embodiment.

[0060] In S1401, the CPU 201 extracts all detected image abnormalities whose certainty is equal to or greater than the minimum certainty and whose input error is equal to or less than the maximum input error from among the detected image abnormalities detected by the image diagnostic device, and sets them as the processing target for S1402. The extracted detected image abnormalities are stored in the storage device 208 by the CPU 201.

[0061] In S1402, for each of the detected image anomalies that were processed in S1401, a confidence distance is calculated as an index for determining whether or not the image anomaly is the one that the user intended to specify. The processes of S1401 to S1402 may be the same as the processes of S701 to S702 in the first embodiment.

[0062] In S1403, based on the calculation result of the confidence distance, the CPU 201 adds the detected image anomaly with the smallest confidence distance to a new group (same-cause image anomaly group) for identifying a group of image anomalies with the same cause. That is, the CPU 201 provides a buffer in the storage device 208 for collectively managing groups of same-cause image anomalies, and adds information on the detected image anomaly with the smallest confidence distance to the buffer. Note that, as long as it is possible to identify the group to be collectively displayed, the method of adding the detected image anomaly to a buffer for group identification and managing it is not limited to this. For example, the detected image anomaly may be managed by assigning a group identifier for identifying the group to the detected image anomaly. In this case, a new group identifier may be assigned to the detected image anomaly with the smallest confidence distance. Note that, in order to exclude the detected image anomaly added to the same-cause image anomaly group from the processing target of S1404 to S1407, the detected image anomaly added to the same-cause image anomaly group may be excluded from the detected image anomalies detected by the imaging diagnostic device.

[0063] S1404 is the end of a loop for performing the processes of S1405 to S1407, sequentially determining detected image anomalies other than the detected image anomaly with the smallest confidence distance. Note that the detected image anomalies to be determined in S1404 are preferably determined in order of proximity to the detected image anomaly already added to the same image anomaly group.

[0064] In S1405, the CPU 201 determines whether the failure location diagnosed for the detected image anomaly currently being evaluated is the same as the failure location of the detected image anomaly that has been added to the group of same-cause image anomalies. That is, based on the detection results shown in FIG. 5 detected by the image diagnostic device, the CPU 201 determines whether the failure location diagnosed for the detected image anomaly currently being evaluated is the same as the failure location diagnosed for the detected image anomaly that was determined to have the smallest confidence distance in S1403. If it is determined in S1405 that the failure location of the detected image anomaly currently being evaluated is the same as the failure location of the detected image anomaly that has been added to the group of same-cause image anomalies (yes), the process proceeds to S1406. On the other hand, if it is determined in S1405 that the failure location of the detected image anomaly currently being evaluated is not the same as the failure location of the detected image anomaly that has been added to the group of same-cause image anomalies (no), and if there remains a detected image anomaly to be evaluated next as a result of the loop end processing in S1404, the detected image anomaly is treated as the next evaluation target and the process returns to S1405.

[0065] In S1406, the CPU 201 determines whether the distance between the detected image anomaly currently being determined and the detected image anomaly already added to the group of image anomalies with the same cause is within a predetermined threshold. If it is determined in S1406 that the distance between the detected image anomaly currently being determined and the detected image anomaly already added to the group of image anomalies with the same cause is within the predetermined threshold (yes), the process proceeds to S1407. On the other hand, if the distance between the detected image anomaly currently being determined and the detected image anomaly already added to the group of image anomalies with the same cause is not within the predetermined threshold (no), the process returns to S1405, with the detected image anomaly being the next target of determination, if any. Note that the predetermined threshold may be a default value for the image forming apparatus 102 or a value set by the user.

[0066] In S1407, the CPU 201 adds the detection image abnormality currently being determined to the same cause image abnormality group. Then, if there is a remaining detection image abnormality to be the next determination target by the process at the loop end paired with the loop end of S1404, the process returns to S1405 with the detection image abnormality as the next determination target.

[0067] FIG. 15 is an example for explaining the processes of S1404 to S1407. Here, it is assumed that three streak-like image abnormalities 1302 are detected by analyzing a scanned image of a user's printed matter as shown in FIG. 13 with an image diagnostic apparatus. Further, in S304, it is assumed that coordinate input 801 by the user is performed in the vicinity of the bounding boxes 1501 to 1503 corresponding to the detected detection image abnormalities. The circle 802 indicates the range of the maximum input error r0 from the input coordinate 801. The bounding box 1501 is a bounding box whose distance from the input coordinate 801 is r1 (that is, the input error is r1). The bounding box 1501 is assumed to be added to the same cause image abnormality group in S1403 as the detection image abnormality with the smallest confidence distance from the input coordinate 801. Here, it is assumed that the distance between the bounding boxes 1501 and 1502 is d1. Also, it is assumed that the distance between the bounding boxes 1502 and 1503 is d2. Note that the detection image abnormalities corresponding to the bounding boxes 1501, 1502, and 1503 are image abnormalities caused by the same cause and are diagnosed as the same failure location in the diagnosis of the image diagnostic apparatus. Also, in FIG. 15, it is assumed that the magnitude relationship between the distances r0 and r1 is r1 < r0. Also, when a predetermined threshold value used in the determination process of S1406 is d0, it is assumed that both the distances d1 and d2 are not more than d0.

[0068] The bounding box having the smallest distance from the image anomaly already added to the group of common-cause image anomalies (i.e., the detected image anomaly corresponding to bounding box 1501) is bounding box 1502. In S1404, CPU 201 determines bounding box 1502 as the determination target and acquires information about the detected image anomaly corresponding to bounding box 1502. Then, in S1405, CPU 201 compares the failure location of the cause of the detected image anomaly corresponding to bounding box 1502 with the failure location of the cause of the image anomaly already added to the group of common-cause image anomalies. Because the failure location of the detected image anomaly corresponding to bounding box 1502 is the same as the failure location of the image anomaly already added to the group of common-cause image anomalies (the detected image anomaly corresponding to bounding box 1501), processing proceeds to S1406. Furthermore, in S1406, CPU 201 compares the distance d1 between the detected image anomaly corresponding to bounding box 1502 and the image anomaly already added to the group of common-cause image anomalies with a predetermined threshold d0. 15, since the distance d1 is equal to or less than d0, the process proceeds to S1407. Then, in S1407, the CPU 201 adds the detected image anomaly corresponding to the bounding box 1502 to the same-cause image anomaly group.

[0069] Then, by processing the loop end in S1404, the bounding box 1503 that is the closest to the group of common-cause image anomalies (the detected image anomalies corresponding to the bounding boxes 1501 and 1502) is set as the next target for determination, information on the detected image anomaly corresponding to this bounding box 1503 is acquired, and the process proceeds to S1405. In S1405, the CPU 201 compares the failure location of the detected image anomaly corresponding to the bounding box 1503 with the failure locations of the image anomalies that have already been added to the group of common-cause image anomalies. Because the failure location of the detected image anomaly corresponding to the bounding box 1503 is the same as the failure location of the image anomaly that has already been added to the group of common-cause image anomalies (the detected image anomalies corresponding to the bounding boxes 1501 and 1502), the process proceeds to S1406. Furthermore, in S1406, the CPU 201 compares the distance d2 between the detected image anomaly corresponding to the bounding box 1503 and the image anomaly that has already been added to the group of common-cause image anomalies with a predetermined threshold d0. 15, since the distance d2 is equal to or less than d0, the process proceeds to S1407. Then, in S1407, the CPU 201 adds the detected image anomaly corresponding to the bounding box 1503 to the same-cause image anomaly group.

[0070] In S1408, the CPU 201 collectively adds the detected image anomalies that have been added to the group of image anomalies with the same cause to the image anomaly candidate to be displayed. The image anomaly candidate to be displayed is stored in the storage device 208 by the CPU 201. In the example of FIG. 15, the detected image anomalies corresponding to the bounding boxes 1501 to 1503 are collectively stored in the image anomaly candidate to be displayed. Therefore, in the example of FIG. 15, it is determined in S306 that there is an image anomaly candidate to be displayed, and in S307, the areas of the bounding boxes 1501 to 1503 are displayed superimposed on the user image data.

[0071] As described above, in a user image in which multiple image abnormalities caused by the same cause appear close together, by simply touching a single point on the screen to specify coordinates, the multiple image abnormalities can be collectively identified and displayed as the image abnormality that the user intended to specify.

Claims

1. a receiving means for receiving information about a position designated by a user via a screen on which an image obtained by scanning a printed matter is displayed; an identification means for identifying at least one of the candidate regions for image abnormalities as the region of image abnormality that the user is attempting to specify, based on one or more candidate regions for image abnormalities and their certainties obtained by analyzing the image, and information on the position specified by the user; a display means for controlling the display of information corresponding to at least one candidate area identified by the identification means; An image forming apparatus comprising:

2. 2. The image forming apparatus according to claim 1, wherein the certainty factor indicates a probability that an image abnormality has occurred in each of the one or more candidate regions for the image abnormality.

3. 2. The image forming apparatus according to claim 1, wherein the image is analyzed by an image diagnostic apparatus different from the image forming apparatus, to determine the one or more candidate regions for the image abnormality and their certainty factors.

4. 2. The image forming apparatus according to claim 1, wherein the display unit controls the display unit to display the position of at least one candidate area identified by the identification unit so as to be superimposed on the image.

5. 2. The image forming apparatus according to claim 1, wherein the display unit controls the display unit to display information for dealing with an image abnormality occurring in at least one of the candidate areas identified by the identification unit.

6. 2. The image forming apparatus according to claim 1, wherein the display unit is controlled to display information about a part that caused an image abnormality that occurred in at least one candidate area identified by the identification unit.

7. 2. The image forming apparatus according to claim 1, wherein the display unit is controlled to display information regarding a method for cleaning parts as information for dealing with an image abnormality that has occurred in at least one candidate area identified by the identification unit.

8. 2. The image forming apparatus according to claim 1, wherein the display unit is controlled to display information indicating that a sales company will be notified in order to deal with an image abnormality that has occurred in at least one candidate area identified by the identification unit.

9. The image forming apparatus according to claim 8, further comprising a transmitting means for transmitting, when notifying the sales company, information on the image diagnosis results relating to at least one candidate area identified by the identifying means to a management server of the sales company.

10. 2. The image forming apparatus according to claim 1, wherein the information about the position designated by the user is input coordinates designated by a single touch on the screen by the user.

11. The image forming apparatus according to claim 1, wherein the identification means collectively identifies, as the area of ​​the image abnormality that the user intended to specify, a plurality of candidate areas of the image abnormality identified based on the plurality of candidate areas of the image abnormality obtained by analyzing the image and their certainty levels and information on the position specified by the user, as well as candidate areas of the image abnormality that have the same fault location causing the image abnormality and are closer than a predetermined threshold.

12. A program for causing a processor to function as each of the units of the image forming apparatus according to any one of claims 1 to 11.

13. 1. A control method for controlling an apparatus, comprising: a step in which a receiving means of the device receives information about a position designated by a user via a screen on which an image obtained by scanning a printed material is displayed; a step in which the specifying means of the device specifies at least one of the candidate regions for image abnormalities as the region of image abnormality that the user is attempting to specify, based on one or more candidate regions for image abnormalities and their certainties obtained by analyzing the image, and information on the position specified by the user; controlling a display means of the device to display information corresponding to the identified at least one candidate region; A control method comprising:

Images