Inspection system, inspection method, and inspection program

Abstract

This system provides an inspection system that can suppress the decline in inspection accuracy due to changes in inspection conditions and reduce the cost of training deep learning models. [Solution] An inspection system comprising: a data acquisition unit that acquires data relating to the shape of a bundle containing multiple processed recording media; a condition acquisition unit that acquires inspection conditions; a machine learning model unit that selectively uses one of several different machine learning models; a processing unit that selectively performs one of several different processes on at least one of the data before input to the machine learning model unit or after output to the machine learning model unit; and a selection unit that selects a machine learning model to be used by the machine learning model unit and a process to be performed by the processing unit based on the inspection conditions, wherein the machine learning model unit and the processing unit inspect the bundle based on the above data.

Description

Technical Field

[0001] The present invention relates to an inspection system, an inspection method, and an inspection program.

Background Art

[0002] After an image is formed on each recording medium by an image forming apparatus, a printed matter in the form of a bundle of recording media is manufactured by performing a processing such as binding on the plurality of recording media on which the image is formed.

[0003] Regarding the inspection of a printed matter in the form of a bundle of recording media, the following prior art is disclosed in Patent Document 1 below. Information regarding the shape of a bundle including a plurality of processed recording media to be inspected is acquired, and based on the acquired information, using a deep learning model corresponding to the combination of the basis weight of the cover of the bundle and the number of recording media, the inspection of the bundle is performed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in inspection using a deep learning model, the inspection accuracy varies depending on the inspection conditions. On the other hand, it is not practical to train a deep learning model to be able to handle all inspection conditions because the cost such as the man-hours for collecting learning data and the learning time increases. The above prior art cannot address such problems.

[0006] This invention was made to solve these problems. Specifically, it aims to provide an inspection system, inspection method, and inspection program that can suppress the decrease in inspection accuracy due to changes in inspection conditions and reduce the training cost of deep learning models. [Means for solving the problem]

[0007] The above-mentioned problems of the present invention are solved by the following means.

[0008] (1) An inspection system comprising: a data acquisition unit that acquires data relating to the shape of a bundle containing a plurality of processed recording media; a condition acquisition unit that acquires inspection conditions; a machine learning model unit that selectively uses one of several different machine learning models; a processing unit that selectively performs one of several different processes on at least one of the data before input to or after output to the machine learning model unit; and a selection unit that selects the machine learning model to be used by the machine learning model unit and the processing to be performed by the processing unit based on the inspection conditions acquired by the condition acquisition unit, wherein the machine learning model unit and the processing unit inspect the bundle based on the data relating to the shape of the bundle acquired by the data acquisition unit.

[0009] (2) The inspection system described in (1) above, wherein the machine learning model is a deep learning model.

[0010] (3) The inspection system according to (1) above, wherein the processing unit performs the rule-based processing corresponding to the inspection conditions.

[0011] (4) The inspection system described in (1) above, wherein the inspection conditions include information on the printed content on the cover of the bundle.

[0012] (5) The inspection system described in (1) above, wherein the inspection conditions include the types of abnormalities to be inspected.

[0013] (6) The inspection system according to (1) above, wherein the data relating to the shape of the bundle is imaging data of the bundle, and the inspection conditions include at least one of the type of imaging unit that captures the imaging data and the settings of the imaging unit.

[0014] (7) The inspection system according to (1) above, having a condition input unit that accepts manual input of the inspection conditions by a user.

[0015] (8) The inspection system according to (1) above, wherein the condition acquisition unit acquires information as inspection conditions from an image forming apparatus that forms an image on the recording medium, or a post-processing machine that processes the plurality of recording mediums.

[0016] (9) The inspection system according to (1) above, wherein the condition acquisition unit acquires environmental information of the environment in which a post-processing machine that processes the plurality of recording media, or a detection unit that detects data relating to the shape of the bundle, is installed, as the inspection conditions.

[0017] (10) The inspection system according to (1) above, further comprising a notification unit that notifies the system to create a machine learning model suitable for the inspection conditions based on the selection result by the selection unit.

[0018] (11) The inspection system according to (1) above, further comprising a selection setting unit that accepts a selection to create a machine learning model suitable for the inspection conditions based on the selection result by the selection unit.

[0019] (12) The inspection system according to (1) above, further comprising a notification unit that notifies the time required according to the selection made by the selection unit.

[0020] (13) The inspection system according to (1) above, further comprising a change input unit that accepts manual input from a user of a change in the selection result by the selection unit, wherein the selection unit changes the selected machine learning model and the processing performed by the processing unit based on the accepted change.

[0021] (14) The inspection system according to (1) above, which includes the machine learning model unit and the processing unit, and has an analysis unit for inspecting the bundle by the machine learning model unit and the processing unit, wherein the analysis unit is composed of a combination of a plurality of deep learning models.

[0022] (15) The inspection system according to (1) above, which has a re-learning unit for re-learning the machine learning model, and the re-learning unit selects settings for re-learning the machine learning model according to the inspection conditions acquired by the condition acquisition unit.

[0023] (16) An inspection method having: a step (a) of acquiring data regarding the shape of a bundle including a plurality of processed recording media; a step (b) of acquiring inspection conditions; a step (c) of selecting, based on the inspection conditions acquired in step (b), either a machine learning model to be used for inspection or any one of a plurality of different processes to be performed on at least one of the data before input to or after output from the machine learning model; and a step (d) of inspecting the bundle by the machine learning model and the process selected in step (c) based on the data regarding the shape of the bundle acquired in step (a).

[0024] (17) An inspection program for causing a computer to execute a process having: a step (a) of acquiring data regarding the shape of a bundle including a plurality of processed recording media; a step (b) of acquiring inspection conditions; a step (c) of selecting, based on the inspection conditions acquired in step (b), either a machine learning model to be used for inspection or any one of a plurality of different processes to be performed on at least one of the data before input to or after output from the machine learning model; and a step (d) of inspecting the bundle by the machine learning model and the process selected in step (c) based on the data regarding the shape of the bundle acquired in step (a).

Effect of the Invention

[0025] Data regarding the shape of a bundle including a plurality of processed recording media is obtained, and inspection conditions are acquired. Based on the inspection conditions, a machine learning model and at least one of the data before input to or after output from the machine learning model unit are selected for processing. Then, the bundle is inspected based on the data by the selected processing and the machine learning model. This can suppress a decrease in inspection accuracy due to the inspection conditions and reduce the cost of re-learning the deep learning model.

Brief Description of the Drawings

[0026] The advantages and features provided by one or more embodiments of the present invention are more fully understood from the following detailed description and the accompanying drawings. However, these are for illustrative purposes only and are not intended to limit the present invention. [Figure 1] It is a schematic diagram showing the configuration of an inspection system. [Figure 2] It is an explanatory diagram for explaining an example of case-binding bookbinding. [Figure 3] It is an explanatory diagram for explaining an example of non-binding bookbinding. [Figure 4] It is a block diagram showing the hardware configuration of an inspection unit. [Figure 5] It is a block diagram showing the functions of the control unit of an inspection unit. [Figure 6] It is a diagram showing a table in which the relationship between print content as inspection conditions and pre-processing is set. [Figure 7] It is a diagram showing a table in which the relationship between inspection abnormality types as inspection conditions and pre-processing is set. [Figure 8] It is a diagram showing a table in which the relationship between camera settings as inspection conditions and pre-processing is set. [Figure 9] It is a diagram showing a flowchart of the inspection operation of an inspection unit. [Figure 10] It is a diagram showing a flowchart of the re-learning operation of the machine learning model of an inspection unit.

Embodiments for Carrying Out the Invention

[0027] Hereinafter, an inspection system, inspection method, and inspection program according to embodiments of the present invention will be described with reference to the drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0028] Figure 1 is a schematic diagram showing the configuration of the inspection system 10. The inspection system 10 includes a paper feeding unit 200, an image forming unit 300, an image inspection unit 400, a post-processing unit 500, a bundle transport unit 700, an discharge unit 800, an acquisition unit 600, and an inspection unit 100. Arrow A in the figure indicates the transport direction of the recording medium. The recording medium includes paper 1, resin film, etc. Paper 1 can include various types of paper. For the sake of simplicity, the following explanation will use the case where the recording medium is paper 1 as an example.

[0029] The paper feeding unit 200 supplies the paper 1 stored in a tray or the like to the image forming unit 300 one sheet at a time.

[0030] The image forming unit 300 forms an image on one or both sides of the supplied paper 1. The image forming unit 300 is composed of, for example, an MFP (MultiFunction Peripheral).

[0031] The image inspection unit 400 inspects the image on the paper 1 formed by the image forming unit 300. The image inspection unit 400 includes a first image inspection sensor 400a that inspects the image formed on the first surface of the paper 1 in a known manner, and a second image inspection sensor 400b that inspects the image formed on the second surface of the paper 1 in a known manner.

[0032] A media sensor 20 may be placed in the transport path 19 leading from the paper feeding section 200 to the image forming section 300. The media sensor 20 detects information about the paper 1. The information about the paper 1 includes the basis weight of the paper 1.

[0033] The image forming unit 300 includes a control unit 310 and an operation display unit 320.

[0034] The operation display unit 320 accepts various information input from the user. The operation display unit 320 displays various information to the user. The information accepted by the operation display unit 320 includes information about print jobs. A print job is a general term for print commands to the image forming unit 300, and includes print data and print settings. Print data is the data of the document to be printed, and may include various types of data such as image data, vector data, and text data. Specifically, print data may be PDL (Page Description Language) data, PDF (Portable Document Format) data, or TIFF (Tagged Image File Format) data. Print settings are settings related to image formation on the recording medium, and include various settings such as the number of pages, the number of copies to print, the type of recording medium, the selection of color or monochrome, double-sided printing, and page layout. The information about print jobs includes the number of sheets of paper 1 that make up a stack containing multiple sheets of paper 1 (hereinafter also referred to as "paper stack 2"), and the type of processing treatment for the paper 1. The paper stack 2 is manufactured by processing treatment by the post-processing unit 500.

[0035] The control unit 310 controls the overall operation of the inspection system 10. Specifically, the control unit 310 controls the paper feeding unit 200, the image forming unit 300, the image inspection unit 400, the post-processing unit 500, the bundle transport unit 700, the discharge unit 800, the acquisition unit 600, and the inspection unit 100. The control unit 310 may be configured by a CPU.

[0036] The post-processing unit 500 produces a paper bundle 2, which is a stack containing multiple sheets of paper 1, through processing. The post-processing unit 500 is located downstream of the image forming unit 300 and processes the multiple sheets of paper 1 on which images have been formed by the image forming unit 300. Paper 1 that is determined to be defective as a result of image inspection by the image inspection unit 400 can be transported to a paper discharge unit (not shown) and excluded from processing.

[0037] Paper stack 2 includes (A) a stack of multiple sheets of paper 1 that are not folded, (B) a stack of multiple sheets of paper 1 that are individually folded and then stacked, or a stack of multiple sheets of paper 1 that are folded together, and (C) a stack of a single sheet of paper 1 that has been folded.

[0038] The post-processing unit 500 processes multiple sheets of paper 1 according to the conditions specified in the print job. Processing includes stapling, punching, perfect binding, folding, saddle stitching, unstitched binding, and perfect binding. Booklet binding is one form of processing. Booklets are obtained by binding the bundles of (A) to (C) above into a booklet. Booklets to be bound are broadly classified into booklets that require binding and booklets that do not require binding. One example of a booklet that requires binding is a saddle-stitched booklet. One example of a booklet that does not require binding is an unstitched booklet. In addition, there are bundles of paper obtained by stapling or punching the bundle of (A) above.

[0039] Figure 2 is an explanatory diagram illustrating an example of saddle-stitched binding.

[0040] In saddle stitching, multiple sheets of paper 1 are stacked on top of each other, and multiple staples 3 are driven along the center line of each sheet of paper 1. Then, the multiple sheets of paper 1 are folded along the center line. This results in a saddle-stitched booklet 4.

[0041] Figure 3 is an explanatory diagram illustrating an example of unbound bookbinding.

[0042] In unbound bookbinding, multiple sheets of paper 1, folded in half, are stacked in page order to form a booklet. This results in an unbound booklet 5.

[0043] Unbound booklets 5 may have defects such as folding or tearing of the paper 1. Saddle-stitched booklets 4 may have defects such as folding or tearing of the paper 1, as well as defects related to stapling, such as dry staples 3 being used or buckling of staples 3.

[0044] The bundle conveying unit 700 conveys the paper bundle 2, which is a bundle containing multiple sheets of paper 1 that have been processed by the post-processing unit 500. The bundle conveying unit 700 is composed of, for example, multiple belt conveyors.

[0045] The discharge unit 800 switches the destination of the paper stack 2 according to the inspection results from the inspection unit 100. Specifically, the discharge unit 800 switches the destination of the paper stack 2 according to a control signal based on the inspection results.

[0046] The discharge unit 800 can assume one of two positions in Figure 1: a first position shown by a dashed line, or a second position shown by a solid line. When the discharge unit 800 assumes the first position, the paper stack 2 is transported toward a good product collection unit (not shown) and collected as good product. When the discharge unit 800 assumes the second position, the paper stack 2 is transported toward a defective product collection unit (not shown) and collected as defective product.

[0047] The inspection unit 100 determines whether the stack of paper 2 to be inspected is good or defective. Details of the inspection unit 100 will be described later.

[0048] The acquisition unit 600 acquires data relating to the shape of the processed paper stack 2. The data relating to the shape of the processed paper stack 2 may be data indicating the three-dimensional shape of the processed paper stack 2. The three-dimensional shape data includes information about the shape of the paper stack 2 in the planar direction and information about the shape of the paper stack 2 in the height direction. The shape of the paper stack 2 in the planar direction is the shape when the paper stack 2 is viewed from above. In other words, the shape of the paper stack 2 in the planar direction is the shape that defines the size of the paper stack 2. The height direction of the paper stack 2 can be rephrased as the thickness direction of the paper stack 2. The three-dimensional shape data can be rephrased as data indicating a three-dimensional shape.

[0049] The acquisition unit 600 includes a first acquisition unit 600a and a second acquisition unit 600b. The first acquisition unit 600a and the second acquisition unit 600b constitute a detection unit. The first acquisition unit 600a acquires data on the shape of the paper stack 2 from the front side of the paper stack 2. The second acquisition unit 600b acquires data on the shape of the paper stack 2 from the back side of the paper stack 2. The first acquisition unit 600a is positioned above the stack transport unit 700. When the first acquisition unit 600a is positioned in this manner, the front side of the paper stack 2 corresponds to the upper side of the paper stack 2 that is transported by the stack transport unit 700. The second acquisition unit 600b is positioned below the stack transport unit 150. When the second acquisition unit 600b is positioned in this manner, the back side of the paper stack 2 corresponds to the lower side of the paper stack 2 that is transported by the stack transport unit 700.

[0050] The acquisition unit 600 acquires data relating to the shape of the paper stack 2 by utilizing the light emitted from the paper stack 2. The acquisition unit 600 is configured, for example, as a light section type laser displacement meter. In this case, the acquisition unit 600 irradiates the paper stack 2 with a strip-shaped laser beam and acquires data relating to the shape of the paper stack 2 by receiving the reflected light from the paper stack 2. By configuring the acquisition unit 600 as a light section type laser displacement meter, data relating to the shape of the paper stack 2 can be acquired with high accuracy without distorting the shape of the paper stack 2. The data relating to the shape of the paper stack 2 may be imaging data obtained by the acquisition unit 600, in which data showing the three-dimensional shape acquired by the light section type laser displacement meter is converted into a two-dimensional image. The two-dimensional image includes information about the shape of the paper stack 2 in the height direction. The two-dimensional image may include a two-dimensional heat map, two-dimensional contour lines, etc.

[0051] The acquisition unit 600 may acquire data regarding the shape of the paper stack 2 using a ToF (Time of Flight) sensor, a stereo camera, a pattern illumination method, etc. The light section type laser displacement meter, ToF sensor, stereo camera, and camera used in the pattern illumination method constitute an imaging device. The acquisition unit 170 may be configured to acquire data regarding the shape of the paper stack 2 by utilizing the sound emitted from the paper stack 2.

[0052] Figure 4 is a block diagram showing the hardware configuration of the inspection unit 100. Figure 5 is a block diagram showing the functions of the control unit 110 of the inspection unit 100.

[0053] The inspection unit 100 inspects the stack of paper 2 and outputs the inspection results.

[0054] As shown in Figure 4, the inspection unit 100 includes a control unit 110, a storage unit 120, a communication unit 130, and an operation display unit 140. The operation display unit 140 constitutes a condition input unit and a change input unit. The control unit 110 includes a processor such as a GPU (Graphics Processing Unit) or CPU (Central Processing Unit). The storage unit 120 includes ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The communication unit 130 is an interface for communicating with, for example, an image forming unit 300, an output unit 800, and an acquisition unit 600 via a network, and may be, for example, a LAN card. The operation display unit 140 is, for example, a touch panel, and a touch sensor is provided on a display surface made of liquid crystal or the like. The inspection unit 100 may be configured by a computer. The control unit 110 constitutes a notification unit, a selection setting unit, and a relearning unit.

[0055] The inspection unit 100 functions as a data acquisition unit 111, an analysis unit 11, a recording unit 115, a condition acquisition unit 116, and a selection unit 117 when the control unit 110 executes a program. The analysis unit 11 includes a pre-processing unit 112, a machine learning model unit 113, and a post-processing unit 114. The machine learning model unit 113 may include a machine learning model 113a. Preferably, the machine learning model 113a is a deep learning model. The pre-processing unit 112 and the post-processing unit 114 constitute a processing unit. Note that at least one of the pre-processing unit 112 and the post-processing unit 114 may be included.

[0056] The data acquisition unit 111 acquires data relating to the shape of the paper stack 2 processed by the post-processing unit 500 from the acquisition unit 600. The data acquisition unit 111 can acquire the data relating to the shape of the paper stack 2 processed by the post-processing unit 500 by receiving it from the acquisition unit 600 via the communication unit 130.

[0057] The condition acquisition unit 116 acquires inspection conditions manually entered by the user into the operation display unit 140 from the operation display unit 140. The inspection conditions are various conditions related to inspection and include print content, inspection abnormality type, and camera settings. The print content is image data formed on each sheet 1 of the paper stack 2. The condition acquisition unit 116 may also acquire print jobs entered by the user into the operation display unit 140 as print content. The condition acquisition unit 116 may also acquire data related to the shape of the paper stack 2 acquired by the acquisition unit 111 as print content. The print content may include data that has been processed by analysis of the image data formed on each sheet 1 of the paper stack 2. The print content may be the print content on the cover of the paper stack 2. The print content on the cover of the paper stack 2 may constitute a relatively significant proportion of the data related to the shape of the paper stack 2. The inspection abnormality type is the type of abnormality to be detected as requested by the user and includes, for example, folds, buckling of staples 3, and unevenness of the edges of the paper stack 2. The camera settings refer to the settings for an imaging device, such as a camera, used to acquire data on the shape of the paper stack 2 that is to be inspected. For example, this includes at least one of the following: the type of imaging device and the settings for imaging the imaging device. The imaging device may include, for example, the light section type laser displacement meter described above.

[0058] The condition acquisition unit 116 may acquire necessary inspection conditions from the image forming unit 300 or the post-processing unit 500.

[0059] The condition acquisition unit 116 may acquire environmental information of the environment in which the first acquisition unit 170a and the second acquisition unit 170b, which detect data related to the post-processing unit 500 or the shape of the paper stack 2, are installed, as inspection conditions. The environmental information may include the installation location, temperature, and humidity of the first acquisition unit 170a and the second acquisition unit 170b, which constitute the imaging unit.

[0060] The preprocessing unit 112 preprocesses the data relating to the shape of the paper stack 2. Specifically, the preprocessing unit 112 selectively performs one of several different processes on the data relating to the shape of the paper stack 2, which is the data before input to the machine learning model unit 113. As will be described later, the preprocessing performed by the preprocessing unit 112 is selected by the selection unit 117 based on the inspection conditions. The preprocessing may be a rule-based process corresponding to the inspection conditions.

[0061] The machine learning model unit 113 selectively uses one of the different machine learning models 113a to detect abnormalities in the paper stack 2 based on pre-processed data regarding the shape of the paper stack 2. As will be described later, the machine learning model 113a used by the machine learning model unit 113 is selected by the selection unit 117 based on inspection conditions.

[0062] The machine learning model 113a can be constructed, for example, by an autoencoder. An autoencoder is a model that is trained so that the input data and output data are the same, and is trained by unsupervised learning. The autoencoder comprises an encoder that converts the input data into low-dimensional data by dimensionality reduction, and a decoder that reconstructs the image from the low-dimensional data. The encoder and decoder are each constructed by a neural network. The machine learning model 113a is trained using only data about the shape of good stacks of paper 2. The output image of the trained machine learning model 113a is an image that reconstructs only the main features of the input image. Since the trained machine learning model 113a is trained using only data about the shape of good stacks of paper 2, the image generated by the machine learning model 113a is an image that reconstructs only the main features of the shape of good stacks of paper 2. Therefore, when data about the shape of good stacks of paper 2 is input to the machine learning model 113a, the input image and the output image become virtually the same image. That is, the difference between the input data and the output data becomes almost zero. When data relating to the shape of a defective stack of paper 2 is input to the machine learning model 113a, the machine learning model 113a only reconstructs the main features of the shape of a good stack of paper 2, and cannot reconstruct the defective areas in the input data image. This is because the machine learning model 113a is trained using only data relating to the shape of a good stack of paper 2. Therefore, when data relating to the shape of a defective stack of paper 2 is input to the machine learning model 113a, the input data image and the output data image will be different images. In other words, the difference between the input data and the output data will not be zero. In this case, when there is a difference between the input data and the output data, the machine learning model unit 113 identifies the area with the difference as a defective area and also identifies the type of anomaly, thereby inspecting the stack of paper 2.

[0063] The post-processing unit 114 performs post-processing on the detection results (inference results) from the machine learning model unit 113. Specifically, the post-processing unit 114 selectively performs one of several different processes on the detection results, which are the output data from the machine learning model unit 113. As will be described later, the post-processing performed by the post-processing unit 114 is selected by the selection unit 117 based on the inspection conditions.

[0064] Based on the functions of the pre-processing unit 112, the machine learning model unit 113, and the post-processing unit 114 described above, the analysis unit 11 inspects the paper stack 2 using the pre-processing unit 112, the machine learning model unit 113, and the post-processing unit 114, based on data relating to the shape of the paper stack 2.

[0065] The selection unit 117 selects the machine learning model 113a to be used by the machine learning model unit 113, as well as the pre-processing and post-processing to be performed by the pre-processing unit 112 and the post-processing unit 114, respectively, based on the inspection conditions acquired by the condition acquisition unit 116.

[0066] Figure 6 shows a table that defines the relationship between the print content, which is the inspection condition, and the pre-processing steps.

[0067] In the example shown in Figure 6, a correspondence is established between the amount of edges calculated by analyzing the printed content and the preprocessing. Specifically, the preprocessing corresponding to the case where the amount of edges in the printed content is greater than or equal to a predetermined threshold is filtering with a noise filter with high filtering intensity. The preprocessing corresponding to the case where the amount of edges in the printed content is less than or equal to a predetermined threshold is filtering with a noise filter with low filtering intensity. The amount of edges may be the number of pixels in the image formed on the paper 1 of the paper stack 2 where the difference in pixel value between adjacent pixels is greater than or equal to a predetermined threshold. This predetermined threshold can be determined experimentally from the viewpoint of inspection accuracy.

[0068] The selection unit 117 can use the above table to select a filtering process using a noise filter corresponding to the inspection condition "amount of edges in printed content" as a preprocessing step. That is, the preprocessing unit 112 can perform a rule-based filtering process as a preprocessing step in response to the inspection condition. In particular, if the data regarding the shape of the paper stack 2 is image data from a light section type laser displacement meter, the inspection accuracy may decrease due to noise caused by the edges of the printed content. For example, if the edge of the printed content is the boundary between white and black formed on the paper 1, the reflected light from the black part is weaker than the reflected light from the white part, causing the centroid corresponding to the edge in the detected light to shift. This can cause errors in the detected height, potentially leading to the detection of steps where there are no steps. Such false detection noise reduces the inspection accuracy. By performing a filtering process as a preprocessing step in response to the inspection condition, noise in the preprocessed data regarding the shape of the paper stack 2, which is input to the machine learning model unit 113, can be removed. Therefore, the need to prepare a separate machine learning model 113a for each size of the inspection condition, "amount of edges in printed content," is reduced, thereby lowering the cost of training each machine learning model 113a. In addition, the optimization of preprocessing reduces the processing cost of preprocessing.

[0069] The selection unit 117 can select a filtering process using a noise filter corresponding to the inspection condition "amount of edges in the printed content" as a preprocessing step, and can also further select a machine learning model 113a corresponding to the inspection condition "amount of edges in the printed content". This can further improve inspection accuracy.

[0070] Figure 7 shows a table that defines the relationship between inspection conditions, specifically the types of inspection defects, and pre-processing.

[0071] In the example shown in Figure 7, a correspondence between the type of inspection abnormality and the preprocessing is established. Specifically, the preprocessing maintains the resolution of the imaging data, which is data related to the shape of the paper stack 2, when the inspection abnormality type is "fold," and reduces the resolution of the imaging data, which is data related to the shape of the paper stack 2, when the inspection abnormality type is anything other than "fold." "Fold" is an example of a minor abnormality.

[0072] The selection unit 117 may, using the above table, select a preprocessing step to maintain high resolution of the image data, which is data related to the shape of the paper stack 2, if the inspection abnormality type, which is an inspection condition, is "folded". The selection unit 117 may, if the inspection abnormality type is "other than folded", select a preprocessing step to reduce the resolution of the image data, which is data related to the shape of the paper stack 2, if the inspection abnormality type is "other than folded". In other words, the preprocessing unit 112 may perform preprocessing based on rules in response to the inspection conditions.

[0073] If the inspection condition for inspection abnormality is "folded," the selection unit 117 may further select a machine learning model 113a that corresponds to high-resolution image data, corresponding to maintaining the resolution of the image data which is data related to the shape of the paper stack 2. If the inspection condition for inspection abnormality is "other than folded," the selection unit 117 may further select a machine learning model 113a that corresponds to low-resolution image data, corresponding to reducing the resolution of the image data which is data related to the shape of the paper stack 2.

[0074] Depending on the type of inspection anomaly, the data input to the machine learning model unit 113, which is pre-processed data relating to the shape of the paper stack 2, is adjusted to a resolution suitable for detecting the type of inspection anomaly, and the learning model 113a corresponding to that resolution is used. This allows for high-precision detection of even minute anomalies such as "folds". Furthermore, for anomalies other than minute ones, the machine learning model 113a detects the anomaly based on low-resolution imaging data, thereby reducing the processing time for inspection.

[0075] Figure 8 shows a table that defines the relationship between camera settings, which are inspection conditions, and pre-processing.

[0076] In the example shown in Figure 8, a correspondence between camera settings and preprocessing is established. Specifically, preprocessing involves performing pixel interpolation of the image data, which is data related to the shape of the paper stack 2, if the exposure time is less than a predetermined threshold, and not performing the interpolation process if it is greater than or equal to the predetermined threshold. The predetermined threshold can be determined experimentally from the viewpoint of inspection accuracy. Interpolation is a process that compensates for missing pixel data. Interpolation can be performed using a known algorithm.

[0077] The selection unit 117, using the table above, may select, as a preprocessing step, pixel interpolation processing of the image data, which is data related to the shape of the paper stack 2, if the inspection condition "camera setting" is "exposure time is less than a predetermined threshold". The selection unit 117 may also select, as a preprocessing step, not to perform pixel interpolation processing of the image data, which is data related to the shape of the paper stack 2, if the inspection condition "camera setting" is "exposure time is greater than or equal to a predetermined threshold". In other words, the preprocessing unit 112 may perform preprocessing based on rules in response to the inspection conditions. Insufficient exposure time may result in the loss of pixel data caused by the printed content. Depending on the camera setting, the preprocessing pixel interpolation processing of the data related to the shape of the paper stack 2, which is input to the machine learning model unit 113, is performed appropriately. This reduces the need to prepare a machine learning model 113a for each camera setting of the inspection conditions, and reduces the cost of training each machine learning model 113a.

[0078] For example, if the inspection condition "camera settings" includes the installation position of the imaging unit, the selection unit 117 may select different machine learning models 113a depending on whether the installation position is flat or at an angle.

[0079] The control unit 110 may notify the user via the communication unit 130 or the operation display unit 140 to create a machine learning model 113a suitable for the inspection conditions based on the selection result by the selection unit 117. Specifically, if the selection unit 117 determines that there is no machine learning model 113a suitable for the inspection conditions, the control unit 110 may send a notification to the user prompting them to create a machine learning model 113a.

[0080] The control unit 110 may, based on the selection result by the selection unit 117, accept a selection via the operation display unit 140 to create a machine learning model 113a suitable for the inspection conditions.

[0081] The control unit 110 may notify the user of the required time according to the selection made by the selection unit 117 via the communication unit 130 or the operation display unit 140. The required time can be calculated, for example, using calculation formulas obtained in advance through experiments for each of the pre-processing unit 112, the machine learning model 113a, and the post-processing unit 114.

[0082] The control unit 110 accepts manual input from the user regarding changes to the selection result made by the selection unit 117. The control unit 110 may then modify the machine learning model 113a selected by the selection unit 117, as well as the pre-processing and post-processing performed by the pre-processing unit 112 and post-processing unit 114, respectively, based on the accepted changes.

[0083] The analysis unit 11 may be composed of a combination of multiple deep learning models. For example, the analysis unit 11 may consist of a machine learning model 113a and part or all of the preprocessing unit 112, which may be composed of different deep learning models. Specifically, the noise filter of the preprocessing unit 112 described above may be composed of a deep learning model.

[0084] The control unit 110 retrains the machine learning model 113a. The control unit 110 selects settings for retraining the machine learning model 113a according to the inspection conditions acquired by the condition acquisition unit 116. The settings for retraining include, for example, (1) selection of data to be used for retraining, (2) type of image augmentation during retraining, and (3) generation of data to be used for retraining by simulation. Specifically, regarding (1), for example, the control unit 110 selects data relating to the shape of the paper stack 2 on which printed content is formed, where the difference from the amount of edge of printed content formed on the cover of the paper stack 2 to be inspected is below a predetermined threshold, as data to be used for retraining. This improves the inspection accuracy of the machine learning model 113a for the paper stack 2 to be inspected. Regarding (2), for example, when inspecting for angular deviation of specific printed content formed on the cover of the paper stack 2 to be inspected, the control unit 110 obtains data to be used for retraining by rotational image augmentation based on data relating to the shape of the paper stack 2 to be inspected. Furthermore, regarding (3), for example, if the printed content formed on the cover of the paper stack 2 to be inspected is a black and white image, the control unit 110 obtains black and white data as data to be used for retraining by simulation based on data regarding the shape of the paper stack 2 to be inspected.

[0085] The post-processing unit 114 performs a thresholding process as a post-processing step, for example, to binarize the continuous value prediction result output by the machine learning model unit 113 into two values: normal or abnormal. The post-processing unit 114 may also perform a thresholding process that multi-levels the continuous value prediction result output by the machine learning model unit 113 into, for example, normal or one of several abnormal ranks. In such thresholding processes, the user may want to flexibly set the threshold according to the inspection conditions. Specifically, for example, the user may want to increase the sensitivity of abnormality detection in order to reduce the chances of missing an abnormality. In this case, a post-processing step with a relatively relaxed threshold for detecting abnormalities may be selected by the selection unit 117 based on the inspection conditions.

[0086] Furthermore, the post-processing unit 114 may perform further rule-based judgments on the judgment results output by the machine learning model unit 113 to output the final inspection result. Specifically, for example, after the machine learning model unit 113 has determined that an anomaly is classified into classes for each numerical range, the post-processing unit 114 may determine that an anomaly is finalized if it is classified into a class with a numerical range greater than a set value. In this case, the set value, which is the magnitude of the anomaly that the user wants to detect, is set as the inspection condition, and the rule-based post-processing is selected by the selection unit 117 according to that set value.

[0087] The recording unit 115 stores the data after post-processing by the post-processing unit 114 in the storage unit 120.

[0088] Figure 9 is a flowchart showing the inspection operation of the inspection unit 100. This flowchart can be executed by the control unit 110 according to a program.

[0089] The control unit 110 acquires data regarding the shape of the paper stack 2 to be inspected by receiving it from the acquisition unit 600 via the communication unit 130 (S101).

[0090] The control unit 110 obtains the inspection conditions from the operation display unit 140 (S102).

[0091] The control unit 110 selects pre-processing, machine learning model 113a, and post-processing based on the inspection conditions (S103).

[0092] The control unit 110 performs inspection of the paper stack 2 to be inspected by performing selected pre-processing, inference by the machine learning model 113a, and post-processing on data relating to the shape of the paper stack 2 to be inspected (S104).

[0093] Figure 10 is a flowchart showing the retraining operation of the machine learning model 113a in the inspection unit 100. This flowchart can be executed by the control unit 110 according to the program.

[0094] The control unit 110 obtains the inspection conditions from the operation display unit 140 (S201).

[0095] The control unit 110 selects settings for retraining the machine learning model 113a according to the inspection conditions (S202).

[0096] The control unit 110 performs retraining of the machine learning model 113a with the selected settings (S203).

[0097] The embodiment produces the following effects. Data regarding the shape of a bundle containing multiple processed recording media, along with inspection conditions, are acquired. Based on these inspection conditions, a machine learning model and a process to be performed on at least one of the data before input to or after output to the machine learning model are selected. Then, the bundle is inspected based on the data using the selected process and machine learning model. This suppresses the decrease in inspection accuracy due to inspection conditions and reduces the cost of retraining the deep learning model.

[0098] Furthermore, the machine learning model will be upgraded to a deep learning model. This will allow for flexible and easy improvement of inspection accuracy.

[0099] Furthermore, the system performs the rule-based processing described above, corresponding to the inspection conditions. This allows for a simple and effective reduction in the cost of retraining the deep learning model.

[0100] Furthermore, the inspection criteria include information about the printed content on the cover of the bundle. This effectively suppresses the decrease in inspection accuracy due to the inspection criteria and effectively reduces the cost of retraining the deep learning model.

[0101] Furthermore, the inspection conditions include the types of defects to be inspected. This effectively suppresses the decrease in inspection accuracy due to inspection conditions and effectively reduces the cost of retraining the deep learning model.

[0102] Furthermore, data relating to the shape of the bundle is used as image data of the bundle, and the inspection conditions include at least one of the type of imaging unit that captures the image data and its settings. This effectively suppresses the decrease in inspection accuracy due to inspection conditions and effectively reduces the cost of retraining the deep learning model.

[0103] Furthermore, the system accepts manual input of inspection conditions from users. This makes it easy to suppress the decrease in inspection accuracy due to inspection conditions, and also easily reduces the cost of retraining the deep learning model.

[0104] Furthermore, information is acquired as inspection conditions from an image forming apparatus that forms an image on a recording medium, or from a post-processing machine that processes multiple recording media. This makes it easier to suppress the decrease in inspection accuracy due to inspection conditions, and also makes it easier to reduce the cost of retraining the deep learning model.

[0105] Furthermore, environmental information of the environment where a post-processing machine that processes multiple recording media, or a detection unit that detects data related to the shape of the bundle, is installed, is acquired as inspection conditions. This effectively suppresses the decrease in inspection accuracy due to inspection conditions and effectively reduces the cost of retraining the deep learning model.

[0106] Furthermore, based on the selection results of the machine learning model and the processing to be performed on at least one of the data before input or after output to the machine learning model, the system notifies the user to create a machine learning model suitable for the inspection conditions. This prevents a decrease in inspection accuracy due to the use of a machine learning model unsuitable for the inspection conditions.

[0107] Furthermore, the system accepts the option to create a machine learning model suitable for inspection conditions based on the selection of a machine learning model and the results of processing performed on at least one of the data before input or after output to the machine learning model. This prevents a decrease in inspection accuracy due to the use of a machine learning model unsuitable for the inspection conditions.

[0108] Furthermore, the system notifies the user of the time required for processing the machine learning model and at least one of the data before or after input to the machine learning model. This provides the user with information to help them decide on the trade-off between inspection accuracy and increased processing time.

[0109] Furthermore, the system accepts manual user input for changes to the machine learning model and the processing performed on at least one of the data before or after input to the machine learning model. The selected machine learning model and processing are then modified based on the accepted changes. This allows for a more effective improvement in inspection accuracy.

[0110] Furthermore, it includes a machine learning model unit that selectively uses one of several machine learning models, and a processing unit that selectively performs one of several different processes on at least one of the data before input to or after output to the machine learning model unit. It also includes an analysis unit that inspects the bundles using the machine learning model unit and the processing unit. The analysis unit is composed of a combination of multiple deep learning models. This allows for greater flexibility, suppression of the decrease in inspection accuracy due to inspection conditions, and reduction of the cost of retraining the deep learning models.

[0111] Furthermore, settings for retraining the machine learning model are selected according to the inspection conditions. This allows for effective retraining of the machine learning model to improve inspection accuracy.

[0112] The present invention is not limited to the embodiments described above.

[0113] For example, in this embodiment, some or all of the processing performed by the program may be replaced by hardware such as circuits.

[0114] Furthermore, the steps shown in the flowchart may be executed in parallel, and their order may also be changed.

[0115] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are for illustrative purposes only and are not limiting. The scope of the present invention should be interpreted in accordance with the language of the appended claims. [Explanation of Symbols]

[0116] 1 sheet, 2 stacks of paper, 3 staples, 4. Saddle-stitched booklet, 5. Unbound booklet, 10. Inspection system, 100 Inspection Department, 110, 310 Control Unit, 111 Data acquisition unit, 112 Pre-processing unit, 113 Machine Learning Models Department, 113a Machine learning models, 114 Post-processing unit, 115 Records Department, 116 Condition acquisition unit, 117 Selection section, 120 storage section, 130 Communications Department, 140, 320 operation display section, 200 Paper feed section, 300 Image forming unit, 400 Image Inspection Department, 500 Post-processing department, 600 Acquisition Department; 700 bundle conveying section, 800 Discharge section.

Claims

1. A data acquisition unit that acquires data regarding the shape of a bundle containing multiple processed recording media, A condition acquisition unit for acquiring inspection conditions, A machine learning model unit that selectively uses one of two different machine learning models, A preprocessing unit that selectively performs one of several different preprocessing steps on the data before it is input to the machine learning model unit, The system includes a selection unit that selects the machine learning model to be used by the machine learning model unit and the preprocessing to be performed by the preprocessing unit, based on the inspection conditions acquired by the condition acquisition unit. An inspection system in which, after the preprocessing selected by the selection unit is performed on the data acquired by the data acquisition unit, the machine learning model unit analyzes the data using the machine learning model selected by the selection unit and inspects the bundle.

2. The system further includes a post-processing unit that selectively performs one of several different post-processing operations on the output data from the machine learning model unit. The selection unit further selects the post-processing to be performed by the post-processing unit based on the inspection conditions acquired by the condition acquisition unit. The inspection system according to claim 1, wherein the preprocessing unit performs the preprocessing selected by the selection unit on the data acquired by the data acquisition unit, the machine learning model unit analyzes the preprocessed data using the machine learning model selected by the selection unit, and then the postprocessing unit performs the postprocessing selected by the selection unit to inspect the bundle.

3. The inspection system according to claim 1, wherein the machine learning model is a deep learning model.

4. The inspection system according to claim 1, wherein the preprocessing unit performs the rule-based preprocessing corresponding to the inspection conditions.

5. The inspection system according to claim 1, wherein the inspection conditions include information on the printed content on the cover of the bundle.

6. The inspection system according to claim 1, wherein the inspection conditions include the type of abnormality to be inspected.

7. The data relating to the shape of the bundle is the imaging data of the bundle. The inspection system according to claim 1, wherein the inspection conditions include at least one of the type of imaging unit that captures the imaging data and the settings of that unit.

8. The inspection system according to claim 1, further comprising a condition input unit for receiving manual input of the inspection conditions by a user.

9. The inspection system according to claim 1, wherein the condition acquisition unit acquires information as inspection conditions from an image forming apparatus that forms an image on the recording medium, or from a post-processing machine that processes the plurality of recording mediums.

10. The inspection system according to claim 1, wherein the condition acquisition unit acquires environmental information of the environment in which a post-processing machine that processes the plurality of recording media, or a detection unit that detects data relating to the shape of the bundle, is installed, as the inspection conditions.

11. The inspection system according to claim 1, further comprising a notification unit that notifies the system to create a machine learning model suitable for the inspection conditions based on the selection result by the selection unit.

12. The inspection system according to claim 1, further comprising a selection setting unit that accepts a selection to create a machine learning model suitable for the inspection conditions based on the selection result by the selection unit.

13. The inspection system according to claim 1, further comprising a notification unit that notifies the time required according to the selection made by the selection unit.

14. The system includes a change input unit that accepts manual input from the user to change the selection result made by the selection unit, The inspection system according to claim 1, wherein the selection unit modifies the selected machine learning model and the preprocessing performed by the preprocessing unit based on the received change.

15. The machine learning model unit and the preprocessing unit are included, and the analysis unit is used to inspect the bundle by the machine learning model unit and the preprocessing unit, The inspection system according to claim 1, wherein the analysis unit is composed of a combination of multiple deep learning models.

16. It has a retraining unit that retrains the aforementioned machine learning model, The inspection system according to claim 1, wherein the retraining unit selects settings for retraining the machine learning model according to the inspection conditions acquired by the condition acquisition unit.

17. (a) A step of obtaining data relating to the shape of a bundle containing multiple processed recording media, Step (b) to obtain inspection conditions, Step (c) involves selecting a machine learning model to be used for inspection and one of several different preprocessing steps to be performed on the data before input to the machine learning model, based on the inspection conditions obtained in step (b). Step (d) involves performing the preprocessing selected in step (c) on the data obtained in step (a), then analyzing it using the machine learning model selected in step (c), and inspecting the bundle. An inspection method having the following characteristics.

18. Step (c) further selects one of several different post-processing steps to be performed on the output data from the machine learning model, based on the inspection conditions obtained in step (b). The inspection method according to claim 17, wherein step (d) is to perform the preprocessing selected in step (c) on the data acquired in step (a), analyze the preprocessed data using the machine learning model selected in step (c), and then perform the postprocessing selected in step (c) to inspect the bundle.

19. (a) A step of obtaining data relating to the shape of a bundle containing multiple processed recording media, Step (b) to obtain inspection conditions, Step (c) involves selecting a machine learning model to be used for inspection and one of several different preprocessing steps to be performed on the data before input to the machine learning model, based on the inspection conditions obtained in step (b). Step (d) involves performing the preprocessing selected in step (c) on the data obtained in step (a), then analyzing it using the machine learning model selected in step (c), and inspecting the bundle. An inspection program that causes a computer to perform a process that includes [specific characteristics / features].

20. Step (c) further selects one of several different post-processing steps to be performed on the output data from the machine learning model, based on the inspection conditions obtained in step (b). The inspection program according to claim 19, wherein step (d) is to perform the preprocessing selected in step (c) on the data acquired in step (a), analyze the preprocessed data using the machine learning model selected in step (c), and then perform the postprocessing selected in step (c) to inspect the bundle.

Images