Image diagnostic apparatus, image diagnostic method, and program

Abstract

To provide an image diagnostic apparatus which uses, for learning of a learning model for estimating a treatment content, a treatment which a user has actually done for image abnormality.SOLUTION: An image diagnostic apparatus according to the present invention has reception means for receiving information of a treatment carried out by a user for image abnormality, and learning means for learning a learning model for use in estimation of a treatment content from the image abnormality by using, as learning data, the image abnormality and the received information of the treatment.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image diagnostic device, an image diagnostic method, and a program for an image forming device, and in particular to the collection of learning data used for machine learning of an inference model when performing maintenance on an image forming device, in which image abnormalities in output images from the image forming device and the details of the treatments performed by the user are collected. [Background technology]

[0002] When an error, breakdown, or other abnormality occurs in an image forming device such as a multifunction peripheral, a service technician or other maintenance person is dispatched upon notification of the abnormality. The maintenance person checks the manual and performs maintenance such as replacing parts.

[0003] Cloud computing has also become increasingly popular in recent years. Its main feature is that it uses many computing resources to distribute data conversion and processing, and processes requests from many clients in parallel using distributed parallel processing. By using cloud computing, system developers can easily procure the necessary computing resources, allowing them to focus on developing system functions.

[0004] One of the elements that is highly compatible with cloud computing is AI (Artificial Intelligence). Machine learning is one of the core technologies that realizes AI. Machine learning analyzes large amounts of data (big data) using learning algorithms to create learning models that extract data features (traits, patterns, trends, etc.). Safely storing and analyzing such large amounts of data requires a large amount of computing resources, so it is often implemented in cloud computing environments.

[0005] In the maintenance of the image forming device mentioned above, various methods have been proposed that involve learning data collected from multiple image forming devices and using the resulting trained model to assist in maintenance work such as part replacement.

[0006] For example, Patent Document 1 proposes a maintenance system that determines a visit route for a worker by specifying the device to be visited and instructing the worker to perform replacement work. The proposed maintenance system identifies the worker who will perform the part replacement work based on part replacement requests from multiple printing devices such as MFPs, and part information and work status held by the worker, such as a service technician. This allows the worker, such as a service technician, to perform the replacement work efficiently. Furthermore, Patent Document 2 proposes that when an image abnormality occurs in an image forming device, an example of an abnormal image is displayed, the user's selection of the abnormal image is accepted, and the location of the failure is determined based on the selected abnormal image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-211940 [Patent Document 2] Japanese Patent Application Publication No. 2019-102843 Summary of the Invention [Problem to be solved by the invention]

[0008] However, Patent Document 1 does not identify parts to be replaced based on image anomalies, so it is not possible to obtain information on part replacement or repair details to resolve the image anomalies. Furthermore, Patent Document 2 can provide the operator with information on the location of the fault to resolve the image anomalies, but does not link and record or collect the image anomalies with the location of the fault or the details of the repair. Therefore, there is no mechanism for inferring the location of the fault or the details of the repair based on the location of the fault corresponding to the image anomaly and the details of the repair.

[0009] An object of the present invention is to provide an image diagnostic apparatus that utilizes treatments actually performed by a user for image abnormalities in training a learning model that estimates the treatment content. [Means for solving the problem]

[0010] The imaging diagnostic device of the present invention is characterized by comprising: a receiving means for receiving information on treatments performed by a user for image abnormalities; and a learning means for using the image abnormalities and the information on the received treatments as learning data to train a learning model for estimating the treatment content from the image abnormality. [Effects of the Invention]

[0011] This invention makes it possible to link image anomalies that occur in an image forming apparatus with the faulty parts that caused the image anomalies and the corrective actions that were taken. This provides a mechanism for selecting information corresponding to actions taken by a user such as a maintenance technician from the candidate faulty parts and candidate corrective actions that correspond to the image anomalies, and for using the image anomalies and the selected information as learning data for identifying the candidate faulty parts and candidate corrective actions. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing the configuration of an image diagnostic system for carrying out the present invention. [Figure 2] 1 is a hardware configuration diagram of an image diagnostic system according to the present invention. [Figure 3] FIG. 3 is a sequence diagram of image diagnosis processing in one embodiment. [Figure 4] 1 illustrates an example of a scanned image containing an anomaly, according to one embodiment. [Figure 5] An example of the screen of an information input terminal. [Figure 6] FIG. 2 is a software configuration diagram of an image diagnosis result notification server according to an embodiment. [Figure 7] 10 is a flowchart showing a learning data collection process performed by an image diagnosis result notification server according to an embodiment. [Figure 8] 10 is a flowchart of a machine learning process according to an embodiment. [Figure 9] 10 is a flowchart of a learning data collection process according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] Hereinafter, the data generated in this proposal to create a machine learning model will be referred to as training data, and the data input to an already created machine learning model for estimation will be referred to as input data. Data for retraining an already created machine learning model will also be referred to as training data, and retraining will also be included in the term "learning." When we say that a learning model is trained using training data, this includes both creating a learning model using training data and retraining a learning model using training data.

[0015] The imaging diagnostic device of this embodiment collects learning data including faulty parts and treatment details corresponding to image abnormalities, creates a machine learning model using the learning data, and uses the created learning model to estimate treatment details upon receiving a notification of an image abnormality from an image forming device.

[0016] The user who performs the treatment may be a maintenance person who performs service, or may be a user of the image forming apparatus. Replacement of consumables may be performed not only by a maintenance person but also by the user of the image forming apparatus.

[0017] The imaging diagnostic apparatus of this embodiment is configured to include an imaging diagnosis result notifying server 104. The imaging diagnostic apparatus may further include an imaging abnormality information collecting server 102 and a replacement part information collecting server 105. The imaging abnormality information collecting server 102, the imaging diagnosis result notifying server 104, and the replacement part information collecting server 105 may be configured in any desired manner, and may be configured as a single unit or may be configured as separate units. The result of estimation of the treatment content by the imaging diagnostic apparatus is displayed on a Web UI (Web based user interface) of the portal site of the information input terminal 103. The user views the estimation result and uses it as a reference for the treatment to be actually performed. [Example]

[0018] The configuration of an image diagnosis system for providing online image diagnosis services for implementing the present invention will be described with reference to Figure 1. The image diagnosis system of the present invention includes an image abnormality information collection server 102, an information input terminal 103, an image diagnosis result notification server 104, a replacement parts information collection server 105, an image forming device 106, and a network 101.

[0019] The image forming devices 106 are, for example, a digital multifunction peripheral, a facsimile machine, a laser beam printer, a scanner, and the like. The image anomaly information collection server 102 is a server that collects information from the image forming apparatuses 106. It collects and stores image anomaly information from multiple image forming apparatuses 106 via the network 101. The replacement part information collection server 105 stores information on replacement parts that have been replaced by users during maintenance work. The information on replacement parts may be acquired from the image forming apparatuses 106 or may be input by the user. The information input terminal 103 is an information input terminal used by the user. The user inputs the details of the measures that have actually been taken from the information input terminal 103 and transmits the information to the image diagnosis result notification server 104 via the network 101.

[0020] The image diagnosis result notifying server 104 is a server that creates and stores learning data based on image anomaly information and user responses, and performs diagnosis. The image diagnosis result notifying server 104 receives various information via the network 101. The various information includes feedback information sent by the information input terminal 103, the image anomaly information held by the image anomaly information collecting server 102, and information on replacement parts held by the replacement part information collecting server 105. Learning data is created and stored based on the various received information.

[0021] In the following, as an example, the image anomaly information collecting server 102, the image diagnosis result notifying server 104, and the replacement part information collecting server 105 will be described as separate servers. The servers 102, 104, and 105 may be configured as a single server, or may be configured as multiple servers with the functions of the servers 102, 104, and 105 distributed. The server configuration is not limited to the form shown in FIG. 1.

[0022] 2A, the hardware configuration of an image forming apparatus 106 according to an embodiment of the present invention will be described. The image forming apparatus 106 includes a CPU 201, a ROM 203, a RAM 204, a network interface card 205, an external memory 206, an operation panel 207, a storage device 208, a device interface 209, a printer 210, and a scanner 202. These components are connected via a system bus 200.

[0023] The CPU 201 performs overall control of access to various devices connected to the system bus 200. The CPU 201 performs control by reading out control programs stored in a ROM 203 or control programs and resource data (resource information) stored in an external memory 206 connected via a disk controller or the like into a RAM 204 and executing them.

[0024] The ROM 203 stores various data such as programs such as the basic I / O program, font data used in document processing, template data, etc. The RAM 204 functions as the main memory, work area, etc. of the CPU 201, and is configured so that the memory capacity can be expanded by an optional RAM connected to an expansion port (not shown).

[0025] The network interface card 205 is an interface with external devices, and the image forming apparatus 106 exchanges data with external devices via the network interface card 205. The operation panel 207 displays the screen and accepts user operation instructions via the screen. It also has buttons for setting the operation mode of the printing apparatus, displaying the operating status of the printing apparatus, and performing operations such as copy designation, and a display unit such as an LCD panel.

[0026] The storage device 208 is an external storage means that functions as a large-capacity memory. The device interface 209 is a connection interface with external devices that can be connected via USB or the like. The printer 210 uses known printing technology, and suitable implementations include electrophotography (laser beam), inkjet, and dye sublimation (thermal transfer) methods. The printer 210 prints image data converted from PDL (Page Description Language), PDF (Portable Document Format), or the like as print data onto paper.

[0027] The scanner 202 uses known image reading technology to optically scan a paper document placed on a transparent top plate and convert it into an image, and also successively reads multiple paper documents placed on an automatic document feeder (ADF) and converts them into images.

[0028] 2(B), the hardware configurations of the image anomaly information collection server 102, the image diagnosis result notification server 104, and the replacement part information collection server 105 according to the embodiment of the present invention will be described. The hardware configurations of the servers 102, 104, and 105 are basically the same.

[0029] The servers 102, 104, and 105 each include a CPU 221, a GPU 222, a ROM 223, a RAM 224, a network interface card 225, an external memory 226, an input / output interface 227, a storage device 228, and a device interface 229. The respective components are connected to each other via a system bus 220.

[0030] The CPU 221 controls the entire device and comprehensively controls access to various devices connected to the system bus 220. The CPU 221 performs control by loading control programs stored in a ROM 223 or control programs and resource data (resource information) stored in an external memory 226 connected via a disk controller or the like into a RAM 224 and executing them. The GPU 222 is a computing device specialized for vector operations such as image processing and machine learning.

[0031] The ROM 223 is a storage means and stores various data such as basic I / O programs. The RAM 224 functions as the main memory, work area, etc. of the CPU 221 and GPU 222, and is configured so that the memory capacity can be expanded by an optional RAM connected to an expansion port (not shown).

[0032] The network interface card 225 is an interface with external devices, and the server exchanges data with external devices via the network interface card 225 . The input / output interface 227 can display screens and receive user operation instructions via devices such as a display, keyboard, mouse, smartphone, or tablet. The storage device 228 is an external storage means that functions as a large-capacity memory. The device interface 229 is a connection interface with an external device that can be connected via USB or the like.

[0033] In this embodiment, the CPU 201 of the image forming apparatus 106 transmits a scanned image read by the scanner 202 to the image anomaly information collection server 102 via the network 101. The CPU 221 of the image diagnosis result notification server 104 is capable of executing image diagnosis processing to identify the type and position information of the image anomaly contained in the scanned image, a candidate faulty part indicating the faulty part that caused the image anomaly, and candidate treatment contents indicating treatment contents for the faulty part. Treatment contents for the faulty part include replacement, cleaning, adjustment, repair, etc. of the faulty part.

[0034] An overview of the image diagnosis processing executed in this embodiment will be described below. Fig. 3 is a sequence diagram relating to the image diagnosis processing shown in this embodiment. In the following, the step numbers of each process included in the sequence diagram are indicated by numbers beginning with "S."

[0035] First, in S301, the CPU 201 reads a paper document using the scanner 202. The scanner 202 outputs an actual image on a paper document printed by the image forming apparatus 106, or a printed test chart, as a scanned image 401. For example, the operation panel 207 performs an abnormal image reception function to receive a scanned image containing an image abnormality. The operation panel 207 displays a message prompting the user to set the paper document on the scanner 202 and then press the start button. This causes the user to set the paper document on the scanner 202 and press the start button on the operation panel 207. Note that, normally, the abnormal image reception function is used when there is an image abnormality in the paper document. It is also assumed that the user will scan a paper document containing an image abnormality from the image forming apparatus 106 using the abnormal image reception function.

[0036] In S302, the CPU 201 acquires a scanned image 401 obtained by reading a paper document. In S303, the CPU 201 transmits the scanned image 401 and a timestamp (scan date and time information) to the image anomaly information collection server 102. Fig. 4A shows an example of the scanned image 401 for explaining an abnormal image according to this embodiment. Image anomalies 402 and 403 are image anomalies contained in the scanned image 401. In S304, the CPU 221 of the image diagnosis result notifying server 104 acquires the scanned image 401 of the diagnosis target from the image abnormality information collecting server 102.

[0037] In S305, the CPU 221 of the image diagnosis result notification server 104 uses the trained learning model to estimate position information of the image abnormality included in the scanned image 401, information about the faulty part that caused the image abnormality, and the details of the remedial action to be taken for the faulty part. That is, simply by reading a paper document in the image forming apparatus 106, the position information of the image abnormality, the faulty part, and the details of the remedial action can be estimated. Here, machine learning may be performed using deep learning or other known object detection algorithms (object detection models).

[0038] Machine learning uses images containing image anomalies as training data for learning. A maintenance technician visually inspects an image containing an image anomaly, such as the scanned image 401 sent as an image anomaly, and identifies the area containing the anomaly by surrounding the anomaly with a pointing device or the like. Then, the type of image anomaly is indicated for each area containing an anomaly. Types of image anomalies include, for example, circular anomalies (stains (dots)) and streak-like anomalies (stains (streaks)). In this way, a learning model that can identify anomalies from an image can be created.

[0039] Identifying abnormalities can also be done through image recognition processing. For example, abnormalities can be identified by comparing an abnormal image with a properly printed image or RIP image. If there is only one abnormal area in an image, the user simply needs to specify the type of image abnormality. If there are multiple abnormal areas, the user selects the areas and identifies the type of abnormality for each area.

[0040] FIG. 4(B) shows an example of a diagnostic image 411 for explaining the results of executing machine learning according to this embodiment. The CPU 221 of the image diagnosis result notification server 104 outputs a bounding box 412 indicating an image abnormality 402, an image abnormality type "image abnormality," and a certainty factor. Furthermore, because the scanned image 401 contains multiple image abnormalities, a bounding box 413 indicating an image abnormality 403, an image abnormality type "image abnormality," and a certainty factor are output. The bounding boxes 412 and 413 contain area information that identifies an area indicating an image abnormality to be detected. For example, they include position information of the image abnormality and area information indicating the type of the image abnormality. Furthermore, the certainty factor is the likelihood of the detection result and is expressed as a numerical value between 0 and 100, for example.

[0041] If the bounding box or image anomaly type included in the diagnostic image is incorrect, the user can correct it and have the learning model learn using feedback.In addition, by providing feedback that the answer is correct, the confidence level of the image diagnosis can be increased.

[0042] The training data for the treatment details corresponding to the abnormal image is data that associates the type of image abnormality with the treatment details taken by a user such as a maintenance technician. The user selects a bounding box included in the abnormal image and inputs the treatment details. If the abnormal image contains only one abnormal location, there is no need to select a bounding box. By associating the abnormal image with the treatment taken to resolve the abnormal image, the type of abnormality can be identified, and the treatment details corresponding to the type of abnormality can be identified.

[0043] The display of the information input terminal 103 will be described with reference to FIG. The display screen of the information input terminal 103 displays a screen of a portal site 501. The portal site 501 is an example of a portal screen for inputting information to be transmitted to the image diagnosis result notification server 104. The portal site 501 includes a target image abnormality detail information display section 510, recommended treatment content display sections 511 and 513, feedback input sections 512 and 514, a scanned image display section 515, and image abnormality position information display sections 516 and 517.

[0044] The information input terminal 103 is a terminal used by a user. It may be a tablet terminal or a smartphone carried by a maintenance person when performing maintenance on the image forming apparatus 106. The user can check the image abnormality and the recommended remedial action on the information input terminal 103, which is a tablet terminal, by scanning a paper document having an image abnormality through the abnormal image reception in the image forming apparatus 106 to be maintained.

[0045] The image abnormality detailed information display section 510 displays the product name, machine number, and date and time when the image abnormality occurred. The product name is the product type of the image forming device 106. The machine number is a unique ID assigned to each image forming device 106. The image abnormality occurrence date and time is the date and time when the image abnormality occurred.

[0046] The recommended action content display unit 511 or 513 displays, for each image abnormality, candidate faulty parts and candidate action content that are estimated by the image diagnosis result notification server 104 to have a high probability of resolving the image abnormality. If one scanned image 401 contains multiple image abnormalities, candidate faulty parts and candidate action content are displayed for all image abnormalities. The recommended action content may include replacement, cleaning, repair, etc. of a specific part. Furthermore, the display of candidate faulty parts and candidate action content may indicate the likelihood that the processing will resolve the event based on the estimation results.

[0047] In the example of the recommended action content display section 511, the image abnormality position information display section 516 of the scanned image display section 515 indicates that the probability that the abnormality will be resolved by replacing part A is 80%, by replacing part B it is 15%, and by cleaning part C it is 5%.

[0048] The feedback input section 512 is a screen for inputting feedback information. The details of the measures taken to resolve the actual abnormal event are input as feedback information. The input method may be a selection format such as a check box, or a free input format using text. If multiple image abnormalities exist in one scanned image 401, feedback information is input for all of the image abnormalities.

[0049] In the example of the feedback input section 512, the part A displayed in the recommended treatment content display section 511 is replaced with the abnormal part 516 in the image displayed in the scan image display section 515.

[0050] The scanned image display section 515 is a display section that displays the scanned image 401 sent by the user, and if an image abnormality is present, the abnormal area is displayed as a rectangle. If multiple image abnormalities are present in one scanned image 401, rectangles are displayed in the areas of all image abnormalities. The example of the scanned image display section 515 in Figure 5 shows the presence of bounding boxes 516 and 517, which display the abnormal areas included in the scanned image display section 515 as rectangles.

[0051] The software configuration of the image diagnosis result notifying server 104 of the present invention will be described with reference to Fig. 6. The programs of the image diagnosis result notifying server 104 are read from the RAM 224, the storage device 228, a secondary storage device connected via a device interface 229, etc., and are executed by the CPU 221 and GPU 222 of the image diagnosis result notifying server 104. Access to the external devices such as the image abnormality information collecting server 102 and the information input terminal 103 is performed via a network interface card 225.

[0052] The image diagnosis result notification server 104 has, as data storage units, an image abnormality information storage unit 601, a part replacement information storage unit 602, and a feedback storage unit 603. The image diagnosis result notification server 104 also has, as software functional units, a learning / input data management unit 604, a learning execution unit 605, a machine learning model management unit 606, an estimation execution unit 607, and an estimation result storage unit 608.

[0053] The image abnormality information storage unit 601 stores image abnormality information of the image forming apparatus 106 and the like, which the CPU 221 of the image diagnosis result notification server 104 receives from the image abnormality information collection server 102 via the network 101 .

[0054] The part replacement information storage unit 602 stores the replacement part information that the CPU 221 of the image diagnosis result notification server 104 receives from the replacement part information collection server 105 via the network 101 . The feedback storage unit 603 receives and stores the feedback information input by the CPU 221 of the image diagnosis result notifying server 104 via the feedback input units 512 and 514 of the information input terminal 103 via the network 101 .

[0055] The learning / input data management unit 604 creates and stores learning data and input data based on each element information stored in the image diagnosis result notifying server 104. Each element information includes the following information. Image anomaly information stored in the image anomaly information storage unit 601 Replacement part information stored in the part replacement information storage unit 602 Feedback information stored in the feedback storage unit 603 The learning and input data management unit 604 creates and stores learning data when the CPU 221 of the image diagnosis result notification server 104 learns a machine learning model, and creates and stores input data when making an estimation using the machine learning model.

[0056] The learning execution unit 605 creates a machine learning model by having the CPU 221 of the image diagnosis result notifying server 104 acquire learning data from the learning and input data management unit 604 and execute learning based on a pre-specified machine learning algorithm. The created machine learning model is stored in the machine learning model management unit 606 by the CPU 221 of the image diagnosis result notifying server 104. Note that, as the learning data stored in the learning and input data management unit 604 changes, learning may be executed repeatedly to recreate (re-learn) the machine learning model.

[0057] The machine learning model management unit 606 stores the machine learning model created by the CPU 221 of the image diagnosis result notification server 104 using the learning execution unit 605. Note that the machine learning model used for estimation may be replaced when triggered by reception of a machine learning model from the learning execution unit 605 or a condition determination within the machine learning model management unit 606. For example, the current machine learning model may be replaced when the accuracy rate of the new machine learning model exceeds a certain level.

[0058] The estimation execution unit 607 executes estimation by the CPU 221 of the image diagnosis result notification server 104 acquiring input data from the learning / input data management unit 604 and inputting it into the machine learning model stored in the machine learning model management unit 606. The estimation result storage unit 608 stores the result of estimation executed by the estimation execution unit 607 by the CPU 221 of the image diagnosis result notification server 104. The estimation result storage unit 608 also transmits the estimation result to the information input terminal 103 via the network 101. Alternatively, the estimation result storage unit 608 may receive a request from the information input terminal 103 via the network 101 and return the estimation result. The information input terminal 103 displays the estimation result on the portal screen as shown in recommended action contents 511 and 513 and the scanned image display unit 515. Taking the bounding box 516 and recommended action contents 511 of the scanned image display unit 515 as an example, the recommended action contents of the bounding box 516 indicate that the likelihood of part A is 80%, the likelihood of part B is 15%, and the likelihood of part C is 5%.

[0059] The proposed method shown in this embodiment will be described with reference to FIGS. Fig. 7(A) is an overall flowchart of the image diagnosis result notifying server 104 in this embodiment. The processes of Fig. 7(A), Fig. 7(B), and Fig. 8 are realized, for example, by the CPU 221 or GPU 22 of the image diagnosis result notifying server 104 reading out a program stored in the ROM 223 or the external memory 226 into the RAM 224 and executing it. In the following, the step numbers of each process included in the flowcharts are indicated by numbers beginning with "S". The same applies to the subsequent flowcharts.

[0060] First, in S701, the CPU 221 of the image diagnosis result notifying server 104 collects learning data using the learning / input data management unit 604. Next, in S702, the CPU 221 of the image diagnosis result notifying server 104 creates a machine learning model using the collected learning data using the learning execution unit 605. Finally, in S703, the CPU 221 of the image diagnosis result notifying server 104 stores the created machine learning model in the machine learning model management unit 606.

[0061] When an image abnormality occurs in the image forming apparatus 106, the image diagnosis result notifying server 104 receives a notification from the image forming apparatus 106. The CPU 221 of the image diagnosis result notifying server 104 estimates the treatment content using the machine learning model created in the flow of FIG. 7(A) by the estimation execution unit 607. The CPU 221 of the image diagnosis result notifying server 104 transmits the estimation result to the information input terminal 103. The information input terminal 103 displays a portal site 501, allowing the user to check the portal site 501. The user can input feedback information via the portal site 501 displayed on the information input terminal 103. For example, the portal site 501 accepts input of feedback information 512 for the bounding box 516 and feedback information 514 for the bounding box 517. The CPU 221 of the image diagnosis result notifying server 104 stores the accepted feedback information 512 and 514 in the feedback storage unit 603.

[0062] FIG. 7B is a flowchart showing details of the learning data collection by the image diagnosis result notifying server 104 in this embodiment in S701 of FIG. 7A.

[0063] In S751, the CPU 221 of the image diagnosis result notification server 104 acquires image anomaly information from the image anomaly information collection server 102 and stores it in the image anomaly information storage unit 701. An example of image anomaly information collected by the image anomaly information collection server 102 is shown in Table 1. The product name is the product type of the image forming device 106. The machine number is a unique ID that identifies each image forming device 106. The scanned image transmission date and time is the date and time when the user transmitted the scanned image. The scanned image ID is a unique ID assigned to each transmitted scanned image. The image anomaly ID is a unique character string code for recognizing image anomalies contained in the transmitted scanned image.

[0064] For example, the first line of Table 1 means that "The user of the device with product name PRO1001 and serial number DEV0001 sent the scanned image ID SCA0001 at 10:00 on 2022 / 02 / 01. The image abnormality ID is IMA1001." [Table 1]

[0065] Table 2 also shows an example of bounding box information for image anomalies. The scan image ID is a unique ID assigned to each scanned image sent. The image anomaly ID is a unique string code used to identify image anomalies contained in the scanned image sent. The X and Y coordinates are the coordinates (pixel values) of the top left vertex of the bounding box, and the height and width are the height and width (pixel values) of the bounding box. In the example in Table 2, the image data with scan image ID "SCA0001" contains two image anomalies and has two pieces of bounding box information corresponding to the two image anomalies. [Table 2]

[0066] Next, in S752, the CPU 221 of the image diagnosis result notifying server 104 acquires feedback information of the machine number from the feedback storage unit 603. Table 3 shows an example of the feedback information. The product name is the product type of the image forming device 106. The machine number is a unique ID assigned to each image forming device 106. The treatment date and time is the date and time when the user performed the treatment. The scanned image ID is a unique ID assigned to each scanned image sent. The image abnormality ID is a unique character string code for recognizing image abnormalities contained in the scanned image sent. The treatment content is the name of the treatment content actually performed by the user. The faulty part is the name of the part that the user actually repaired or replaced.

[0067] For example, the first line of Table 3 means that "Part A was replaced for image abnormality ID IMA1001 on scan image ID SCA0001 on machine number DEV0001 with product name PRO1001 at 14:02 on 2022 / 02 / 03." [Table 3]

[0068] In S753, the CPU 221 of the image diagnosis result notification server 104 links the scanned image ID and image abnormality ID with the faulty part and the treatment content from the feedback information in the feedback storage unit 404. Table 4 shows the failure treatment data resulting from linking the scanned image ID and image abnormality ID with the faulty part and the treatment content from the feedback information. The product name in the fault correction data in Table 4 is the product type of the image forming device 106. The scanned image ID is a unique ID assigned to each transmitted scanned image. The image abnormality ID is a unique character string code for recognizing image abnormalities in the transmitted scanned image. The X and Y coordinates are the coordinates (pixel values) of the top left vertex of the bounding box, and the height and width are the height and width (pixel values) of the bounding box. The correction content is the name of the correction content actually performed by the user. The faulty part is the name of the part actually repaired or replaced by the user. [Table 4]

[0069] The present invention makes it possible to link image abnormalities that occur in the image forming apparatus 106 with the details of the measures taken to resolve the image abnormalities and the faulty parts. This provides a mechanism for using fault correction data, including information on the measures taken by the user in response to image abnormalities, as learning data.

[0070] FIG. 8 is a flowchart showing an example of a step of creating a machine learning model in the image diagnosis result notifying server 104 in this embodiment, which corresponds to S702 in FIG. 7(A). In S801, the CPU 221 of the image diagnosis result notification server 104 collects the scanned image stored in the image abnormality information storage unit 601, the replacement part information stored in the part replacement information storage unit 602, and the fault correction data derived from Table 4 as collected data. Either the replacement part information or the fault correction data derived from Table 4 may be used.

[0071] In S802, the CPU 221 of the image diagnosis result notifying server 104 classifies the collected data acquired in S801 by product. In S803, the CPU 221 of the image diagnosis result notification server 104 classifies the collected data classified by product in S802 by abnormality type. The learning data is the scanned image, and the abnormality position information and abnormality type of the abnormality contained in the scanned image. The learning data is also the replacement part information held by the part replacement information storage unit 602. This makes it possible to associate the scanned image with the replacement part. The learning data is also the failure treatment data derived from Table 4, which is the treatment content associated with the abnormality type of the scanned image and the failure information of the failed part.

[0072] In S804 to S806, the CPU 221 of the image diagnosis result notifying server 104 uses the learning data generated in S803 to train a learning model for each abnormality type of each product. In S804, it is determined whether learning of learning models for all abnormality types has been completed. If learning of learning models for all abnormality types has not been completed (No), the flow proceeds to S805. If learning of learning models for all abnormality types has been completed (Yes), the flow ends. The learning of S805 and S806 is repeated until learning of learning models for all abnormality types has been completed. In this embodiment, learning of a learning model is performed for each abnormality type of each product, but learning models may also be trained collectively for multiple abnormality types of each product.

[0073] In S805, the CPU 221 of the image diagnosis result notification server 104 uses an object detection algorithm to train a learning model for the scan images of each product for each type of image abnormality generated in S803. The object detection algorithm can use a well-known algorithm called DETEction TRansformer (DETR). For example, a convolutional neural network is used to estimate a bounding box, which is a frame that contains an object. Then, the confidence that the bounding box contains the object and the probability of each type of object being detected when the bounding box contains the object are predicted. During the learning process, the learning results are evaluated using cross-validation, which randomly divides the training data into analysis data and validation data. This allows for the creation of a learning model that estimates the diagnostic image 411 shown in Figure 4(B).

[0074] Here, there are various types of machine learning algorithms. Various object detection models can be used to detect rectangular areas that indicate an image of an object of interest (e.g., an image abnormality). Examples include YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and R-CNN (Region Based Convolutional Neural Networks). Hyperparameters of machine learning algorithms also differ depending on the machine learning algorithm. In this embodiment, the machine learning algorithm, the method for evaluating the learning results, and the method for optimizing the hyperparameters of the machine learning algorithm can be changed as appropriate.

[0075] Furthermore, machine learning involves learning to associate the types of abnormalities in diagnostic images with the parts that caused the abnormalities and the actions taken to resolve the abnormalities. In the above, the fault action data from Table 4 was described as being used as learning data. However, in the initial state where the feedback information in Table 3 has not yet been accumulated, the user must create fault action data that associates the types of abnormalities in diagnostic images with the parts that caused the abnormalities and the actions taken to resolve the abnormalities. The created fault action data is then used as learning data.

[0076] The creation of learning data by the user will now be described. The user identifies the type and location of the anomaly in a scanned image containing an anomaly, and associates the faulty part and the corrective action for that type of anomaly. If there is only one abnormality in a scanned image, the system can automatically associate the faulty part and the corrective action for that type of anomaly, or information on the replacement part.

[0077] When a scanned image contains multiple abnormalities, the types of abnormalities may differ. Different types of abnormalities may result in different components causing the abnormalities, and multiple faulty or replacement components may correspond to a single scanned image. In such cases, the user creates fault correction data by associating the faulty components and correction details, or replacement component information, with the abnormal image areas of the scanned image while checking the scanned image. The created correction data is then used as learning data for creating a learning model.

[0078] In S806, the CPU 221 of the image diagnosis result notification server 104 saves the trained model trained in S805 in a file and registers it in the machine learning model management unit 606. Here, the file in which the trained model is saved includes the type of learning algorithm and the values ​​of the hyperparameters of the learning algorithm. [Example]

[0079] In the first embodiment, it is assumed that a user provides feedback on an image anomaly via the web UI of a portal site. However, there may be cases where the user incorrectly inputs the feedback record. Since the feedback information is the most important information in the first embodiment, using incorrectly input feedback information may reduce the accuracy of the machine learning model.

[0080] In the second embodiment, when a user mistakenly inputs a completely unrelated part as feedback information, the replacement part included in the feedback information is not included in the replacement part information actually replaced by the user. For example, in the feedback information of Table 5 and the replacement part information of Table 6, it can be seen that the replacement part (part F) included in the feedback information of Table 5 is not included in the replacement parts (part A, part D) included in the user's replacement part information of Table 6. [Table 5] [Table 6] In Example 2, if the replacement part information does not include a replacement part included in the feedback information in Example 1, it is determined that the reliability of the replacement part information included in the feedback information is low, and the feedback information is not used for machine learning.

[0081] The system configuration diagram (FIG. 1), hardware configuration diagram (FIG. 2), input terminal screen example diagram (FIG. 5), and software configuration diagram (FIG. 6) of the second embodiment are the same as those of the first embodiment, and therefore descriptions thereof will be omitted. Also, the overall flowchart (FIG. 7(A)) of the image diagnosis result notification server 104 is the same as that of the first embodiment, and therefore descriptions thereof will be omitted.

[0082] Fig. 9 is a flowchart showing details of the learning data collection step S701 of the image diagnosis result notifying server 104 in Example 2. The processing in Fig. 9 is realized, for example, by the CPU 221 or the GPU 22 of the image diagnosis result notifying server 104 reading a program stored in the ROM 223 or the external memory 226 into the RAM 224 and executing the program.

[0083] 9 are the same as the processes of S751 and S752 shown in Fig. 7(B). In S901, the CPU 221 of the image diagnosis result notifying server 104 acquires replacement part information from the replacement part information collecting server 105 and stores it in the part replacement information storage unit 602.

[0084] In S902, it is determined whether the replacement part included in the feedback information exists in the replacement part information. If the replacement part included in the feedback information exists in the replacement part information (Yes), the process proceeds to S753. The processing of S753 in FIG. 9 is the same as S753 shown in FIG. 7(B). If the replacement part included in the feedback information does not exist in the replacement part information (No) in S902, it is determined that there was an input error in the user's feedback information, and the feedback information is not used as data for machine learning.

[0085] In the second embodiment, by comparing the feedback information and the replacement part information, it is possible to appropriately link the image abnormalities that have occurred in the image forming apparatus and the measures to resolve the image abnormalities. This makes it possible to prevent incorrect linking due to incorrect input by the user. This makes it possible to provide a mechanism for promoting the collection of learning data including appropriate responses to image abnormalities.

[0086] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0087] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0088] The disclosure of this specification includes the following imaging diagnostic apparatus, imaging diagnostic method, and program. (Item 1) a receiving means for receiving information on a treatment performed by a user for an image abnormality; a learning means for learning a learning model for estimating a treatment content from the image abnormality using information on the image abnormality and the accepted treatment as learning data; An imaging diagnostic device comprising: (Item 2) an estimation means for estimating a treatment content for an image abnormality using the learning model; Furthermore, the receiving means receives information on the treatment performed by the user from the estimated treatment content candidates. Item 1. The imaging diagnostic device according to item 1. (Item 3) The content of the treatment includes information on the faulty part. Item 3. The imaging diagnostic apparatus according to item 1 or 2. (Item 4) The content of the measures includes information on the faulty part and a certainty of the content of the measures. Item 2. The diagnostic imaging apparatus according to Item 3. (Item 5) Further, a means for transmitting the estimated treatment content candidates to an information input terminal is provided, the receiving means receives information on the action taken by the user from the information input terminal; 5. The diagnostic imaging device according to any one of items 2 to 4. (Item 6) Further, a determination means is provided for determining whether or not information on a replacement part included in the accepted action exists in information on parts that have actually been replaced, If the determination means determines that the data does not exist, the data is not used as the learning data. 6. The diagnostic imaging device according to any one of items 1 to 5. (Item 7) The receiving means receives information on the treatment performed by the user for the image abnormality; a learning means for learning a learning model for estimating a treatment content from an image abnormality using information on the image abnormality and the accepted treatment as learning data; Diagnostic imaging methods. (Item 8) The computer of the imaging diagnostic device, a receiving means for receiving information on a treatment performed by a user for an image abnormality; a learning means for learning a learning model for estimating a treatment content from the image abnormality using information on the image abnormality and the accepted treatment as learning data; A program to function as a [Explanation of symbols]

[0089] 102: Image abnormality information collection server, 103: Information input terminal, 104: Image diagnosis result notification server, 105: Replacement part information collection server, 106: Image forming apparatus

Claims

1. a receiving means for receiving information on a treatment performed by a user for an image abnormality; a learning means for learning a learning model for estimating a treatment content from the image abnormality using information on the image abnormality and the accepted treatment as learning data; An imaging diagnostic device comprising:

2. an estimation means for estimating a treatment content for an image abnormality using the learning model; Furthermore, the receiving means receives information on the treatment performed by the user from the estimated treatment content candidates. The diagnostic imaging device according to claim 1 .

3. The content of the treatment includes information on the faulty part. The diagnostic imaging device according to claim 1 .

4. The content of the measures includes information on the faulty part and a certainty of the content of the measures. The diagnostic imaging device according to claim 2 .

5. Further, a means for transmitting the estimated treatment content candidates to an information input terminal is provided, the receiving means receives information on the action taken by the user from the information input terminal; The diagnostic imaging device according to claim 2 .

6. Further, a determination means is provided for determining whether or not information on a replacement part included in the accepted action exists in information on parts that have actually been replaced, If the determination means determines that the data does not exist, the data is not used as the learning data. The diagnostic imaging device according to claim 1 .

7. The receiving means receives information on the treatment performed by the user for the image abnormality; a learning means for learning a learning model for estimating a treatment content from an image abnormality using information on the image abnormality and the accepted treatment as learning data; Diagnostic imaging methods.

8. The computer of the imaging diagnostic device, a receiving means for receiving information on a treatment performed by a user for an image abnormality; a learning means for learning a learning model for estimating a treatment content from the image abnormality using information on the image abnormality and the accepted treatment as learning data; A program to function as a

Images