Image Processing Device

Abstract

To provide an image processing apparatus that properly detects an anomaly in a target image.SOLUTION: An anomaly detection unit 12 (a) generates a first characteristic map obtained by performing a filter process for a target image and a second characteristic map obtained by performing the filter process for a reference image, (b) derives a correction amount on the basis of a deviation between an object in the first characteristic map and an object in the second characteristic map, and (c) corrects the target image or the reference image with the correction amount and thereafter compares the target image and the reference image and thereby detects an anomaly in the target image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] One image inspection device compares a target image obtained by scanning a printout of a reference image with the reference image, and inspects the target image based on the difference between the two (see, for example, Patent Document 1).

[0003] At that time, registration marks or other alignment markers are printed on the printed matter, and the target image and the reference image are aligned based on the markers, and the difference between the two is generated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-206691 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, as described above, when using a marker to align the target image with the reference image, when deriving the difference between the target image and the reference image, the marker portion must be excluded to derive the difference. However, if the reference image contains an object similar to the marker near the marker, alignment will not be performed correctly, and abnormalities in the target image may not be detected correctly.

[0006] The present invention has been made in consideration of the above problems, and has an object to provide an image processing device that correctly detects an abnormality in a target image. [Means for solving the problem]

[0007] The image processing device of the present invention is an image processing device that compares a target image with a reference image to detect an abnormality in the target image, and includes an anomaly detection unit that (a) generates a first feature map obtained by performing a filter process on the target image and a second feature map obtained by performing the filter process on the reference image, (b) derives a correction amount based on a deviation between an object in the first feature map and an object in the second feature map, and (c) corrects the target image or the reference image with the correction amount, and then compares the target image with the reference image to detect an abnormality in the target image. The apparatus further includes the following configuration (A) or (B): (A) the anomaly detection unit generates a plurality of the first feature maps and a plurality of the second feature maps by a filter process corresponding to a plurality of object types, derives a deviation between an object in the first feature map and an object in the second feature map for each of the object types, and derives the correction amount based on the derived deviation, and the anomaly detection unit (a) corrects the first feature map or the second feature map for each of the object types with the correction amount, and then generates a difference image between the first feature map and the second feature map, and (b) detects an anomaly in the target image based on the difference image for the plurality of object types. (B) the anomaly detection unit derives the correction amount by excluding an object for which a similarity between an object in the first feature map and an object in the second feature map is less than a predetermined threshold. Effect of the Invention

[0008] According to the present invention, an image processing device that correctly detects an abnormality in a target image is obtained.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of a feature map for a plurality of object types. [Diagram 3] FIG. 3 is a diagram illustrating the coordinate difference of an object in a feature map. [Figure 4] FIG. 4 is a diagram illustrating the removal of objects in a feature map based on the rotation angle. [Diagram 5] FIG. 5 is a diagram illustrating the similarity of objects in a feature map. [Figure 6] FIG. 6 is a diagram showing an example of a difference image between the feature map of the target image and the feature map of the reference image. [Figure 7] FIG. 7 is a diagram illustrating the detection of an abnormality based on a difference image. [Figure 8]FIG. 8 is a diagram illustrating the correspondence relationship between the feature amount and the cause of anomaly. [Figure 9] FIG. 9 is a flowchart illustrating the operation of the image processing device shown in FIG. [Figure 10] FIG. 10 is a flowchart illustrating the object similarity determination process in FIG. [Figure 11] FIG. 11 is a diagram showing an example of a reference image and a corresponding target image. [Figure 12] FIG. 12 is a diagram showing a difference image corresponding to the reference image and the target image shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] Fig. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. The image processing device shown in Fig. 1 is an information processing device such as a personal computer or a server, or an electronic device such as a digital camera or an image forming device (scanner, multifunction machine, etc.), and includes an arithmetic processing device 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, etc.

[0013] The arithmetic processing device 1 includes a computer, and executes an image processing program on the computer to operate as various processing units. Specifically, the computer includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and operates as a specific processing unit by loading a program stored in the ROM or storage device 2 into the RAM and executing the program on the CPU. The arithmetic processing device 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0014] The storage device 2 is a non-volatile storage device such as a flash memory, and stores an image processing program and data necessary for the processing described below. The image processing program is stored in, for example, a non-transitory computer-readable recording medium, and is installed in the storage device 2 from the recording medium.

[0015] The communication device 3 is a device that performs data communication with an external device, such as a network interface, a peripheral device interface, etc. The display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display, etc. The input device 5 is a device that detects user operations, such as a keyboard, a touch panel, etc.

[0016] The internal device 6 is a device that executes a predetermined function of the image processing device. For example, if the image processing device is an image forming device, the internal device 6 is an image reading device that optically reads an original image from an original, a printing device that prints an image on a printing paper, or the like.

[0017] Here, the arithmetic processing device 1 operates as the target image acquisition unit 11 and the abnormality detection unit 12, which are the above-mentioned processing units.

[0018] The target image acquisition unit 11 acquires a target image (image data) from the storage device 2, the communication device 3, the internal device 6, etc., and stores it in a RAM, etc. Note that the target image is obtained, for example, by scanning a printout obtained by printing a reference image.

[0019] The anomaly detection unit 12 (a) generates a first feature map obtained by performing a filter process on the target image and a second feature map obtained by performing the same filter process on the reference image, (b) derives a correction amount based on the deviation between an object in the first feature map and an object in the second feature map (the coordinate difference and angle difference described below), and (c) corrects the target image or the reference image with the correction amount, and then compares the target image with the reference image to detect an anomaly in the target image.

[0020] Specifically, the anomaly detection unit 12 generates a plurality of first feature maps and a plurality of second feature maps by using a plurality of filter processes corresponding to a plurality of object types, respectively, derives the deviation between the object in the first feature map and the object in the second feature map for each object type, and derives a correction amount (for the entire target image or reference image) based on the derived deviation.

[0021] For this filtering process, for example, a second-order differential filter, a Gabor filter, or the like is used.

[0022] Fig. 2 is a diagram showing an example of feature maps of a plurality of object types. For example, as shown in Fig. 2, the plurality of object types include vertical lines, horizontal lines, points, etc., and feature maps of each object type are obtained for each of the reference image and the target image. For example, a one-dimensional horizontal second-order differential filter is used for filtering vertical lines, a one-dimensional vertical second-order differential filter is used for filtering horizontal lines, and a one-dimensional horizontal second-order differential filter and a one-dimensional vertical second-order differential filter are used for filtering points.

[0023] Furthermore, in this embodiment, the correction amount is derived based on the coordinate difference and the angle difference for each pair of corresponding objects between the first feature map and the second feature map.

[0024] Fig. 3 is a diagram for explaining the coordinate difference of objects in a feature map. For the coordinate difference, for example, as shown in Fig. 3, rectangular regions (e.g., circumscribing rectangles of objects) including corresponding objects i in the feature maps of the reference image and the target image are specified, and the horizontal (X-axis direction) and vertical (Y-axis direction) coordinate values ​​(X11i, X12i, X21i, X22i, Y11i, Y12i, Y21i, Y22i) of two vertices P11i, P12i, P21i, P22i on the diagonal line of the rectangular regions are specified, and the average value of the coordinate difference for all pairs of objects i is calculated as the positional deviation (ΔX, ΔY) between the reference image and the target image.

[0025] Similarly, for the angle difference, the angle difference for each pair of corresponding objects i is identified, and the average value of the angle differences for all pairs of objects i is calculated as the angle shift Δθ between the reference image and the target image. For example, when scanning a target image from a print, an angle shift may occur due to the skew of the print.

[0026] At this time, an object that has rotated independently is excluded from the calculation of the angle deviation. Fig. 4 is a diagram for explaining the exclusion based on the rotation angle for each object in the feature map. For example, as shown in Fig. 4, an object having an angle difference of a predetermined deviation (here, for example, three times the variance) or more from the average value θav of the angle difference (rotation angle) for each object is excluded from the calculation of the angle deviation.

[0027] Furthermore, the abnormality detection unit 12 derives the similarity between the objects in the first feature map and the objects in the second feature map, and derives the above-mentioned correction amount by excluding objects whose similarity is less than a predetermined threshold.

[0028] FIG. 5 is a diagram illustrating the similarity of objects in a feature map. For example, as shown in FIG. 5, the similarity of each object is derived, and when the shapes of the objects are completely identical, the similarity is 1, and the more similar the objects are, the closer the similarity is to 1, and when the shapes of the objects are completely different, the similarity is 0. This similarity is derived by an existing predetermined method. For example, an open source library such as OpenCV (Open Source Computer Vision Library) can be used to derive this similarity and the above-mentioned feature map.

[0029] As for the above-mentioned angle difference, object i is rotated by a predetermined angle at a time, and the rotation angle at which the similarity becomes equal to or greater than a predetermined threshold value is regarded as the angle difference of object i.

[0030] FIG. 6 is a diagram showing an example of a difference image between the feature map of the target image and the feature map of the reference image. FIG. 7 is a diagram explaining detection of anomalies based on the difference image. For example, as shown in FIG. 6, the anomaly detection unit 12 (a) corrects the first feature map or the second feature map with the correction amount for each object type, and then generates a difference image between the first feature map and the second feature map, and (b) detects anomalies in the target image based on the difference images for the above-mentioned multiple object types. Here, the anomalies are detected as objects (hereinafter referred to as abnormal objects) in the difference image, as shown in FIG. 7, for example.

[0031] Therefore, an abnormal object is detected when an object not present in the reference image appears in the target image, or when an object in the target image is missing (partially or entirely) or enlarged. Furthermore, a color unevenness analysis may be performed on the difference image to detect color unevenness as an abnormality, for example, as shown in Fig. 7. In the color unevenness analysis, for example, the median value of the pixel values ​​at the front end of the difference image is used as a reference, the deviation of the pixel value of each pixel from the median value is specified, and the sum of the deviations of all pixels in the difference image is derived as an unevenness level (feature amount described later), and when the unevenness level value is equal to or greater than a predetermined threshold value, it is determined that the color unevenness is abnormal.

[0032] Furthermore, the anomaly detection unit 12 identifies the cause of the detected anomaly based on the feature amount of the anomaly. Fig. 8 is a diagram for explaining the correspondence relationship between the feature amount and the cause of the anomaly.

[0033] For example, the feature amount of an abnormality includes the area, orientation, growth direction of an abnormal object, density of an abnormal object part, edge strength of an abnormal object part, color of an abnormal object part, period of an abnormal object, number of abnormal objects, etc., and the cause of an abnormality is identified from the value of the feature amount of an abnormality based on the correspondence relationship between the value range of each feature amount and the cause of an abnormality. Also, as shown in Fig. 8, for example, the cause of an abnormality may be identified based on a combination of the values ​​of a plurality of feature amounts. Note that this correspondence relationship is stored in advance in the image processing device as a table or the like.

[0034] For example, the orientation of the abnormal object is the longitudinal direction of the abnormal object, the growth direction is the growth direction of the abnormal object identified from the shape of the abnormal object obtained at a specific time interval, the density of the abnormal object part is the average or median of the density of the abnormal object part in the target image or the difference between the average or median of the part other than the abnormal object in the target image and the average or median of the density of the object part, the edge strength of the abnormal object part is the density gradient of the edge of the abnormal object part in the target image, the color of the abnormal object part is the color of the abnormal object part in the target image, the period of the abnormal object is the spatial period of multiple abnormal objects, and the number of abnormal objects is the number of abnormal objects of each object type.

[0035] In addition, a predetermined abnormality response process may be executed for the detected abnormality. The abnormality response process may be, for example, notification of each detected abnormality (notification by sending a message via the communication device or displaying a message on the display device 4 to an operator who identifies a defective part corresponding to the abnormality or performs maintenance), identification of a defective part corresponding to each detected abnormality, maintenance operation, etc.

[0036] Next, a description will be given of the operation of the image processing device shown in Fig. 1. Fig. 9 is a flow chart illustrating the operation of the image processing device shown in Fig. 1.

[0037] First, the target image acquisition unit 11 acquires image data of the target image and reads out image data of the reference image stored in advance (step S1). When the image data of the target image and the reference image are stored in advance in the storage device 2 or the like, the image data of the target image and the reference image are simply read out from the storage device 2 or the like.

[0038] Next, the anomaly detection unit 12 performs a predetermined filter process on the target image and the reference image to generate a feature map (filtered image) of one or more object types (vertical lines, horizontal lines, dots, etc.) (step S2). Note that, if image data of the feature maps of the target image and the reference image are stored in advance in the storage device 2 or the like, the image data of the feature maps is simply read out from the storage device 2 or the like.

[0039] The anomaly detection unit 12 performs an object similarity identification process (step S3). In the object similarity identification process, an object in the feature map of the reference image and a corresponding object in the feature map of the target image are identified, and the similarity between the two is identified, and a positional deviation and an angular deviation between the reference image and the target image are identified as correction amounts.

[0040] Then, the anomaly detection unit 12 corrects the position and angle of at least one of the reference image and the target image so as to eliminate the positional deviation and the angle deviation (step S4), generates a difference image between the corrected reference image and the target image, detects an abnormal object from the difference image, and identifies the cause of the anomaly based on the feature amount of the abnormal object (step S5). Note that the feature amount may be derived using an existing open source library.

[0041] Here, the object similarity specification process (step S3) will be described in detail. Fig. 10 is a flowchart illustrating the object similarity specification process in Fig. 9.

[0042] First, the abnormality detection unit 12 selects feature maps of a target image and a reference image for a certain object type as a feature map of interest (step S11).

[0043] Next, the anomaly detection unit 12 selects an object Xi (i=1,...,M, M is the number of objects Xi in the attention feature map) in the attention feature map of the reference image (step S12), and selects an object Yj (j=1,...,N, N is the number of objects Yj in the attention feature map) in the attention feature map of the target image (step S13). Note that the objects Yj are selected, for example, in the order of proximity to the same position in the target image as the position of Xi in the reference image.

[0044] Then, the anomaly detection unit 12 calculates the similarity between the selected objects Xi and Yj (step S14), and determines whether the similarity is equal to or greater than a predetermined threshold (step S15).

[0045] If the similarity is equal to or greater than a predetermined threshold, the anomaly detection unit 12 determines that the current object Xi and the current object Yj correspond to each other, and stores the coordinate difference between them (the difference ΔXi in the X-axis direction and the difference ΔYi in the Y-axis direction) and the rotation angle (angle difference) between them in a RAM or the like (step S16).

[0046] On the other hand, if the similarity is not equal to or greater than the predetermined threshold, the anomaly detection unit 12 determines whether the current rotation angle exceeds a predetermined upper limit, and if the current rotation angle does not exceed the predetermined upper limit, rotates the object Yj by a predetermined angle (step S18). The rotation angle of the object Yj is set to zero when the object Yj is selected, and is increased by a predetermined angle at a time in step S18. Then, returning to step S14, the anomaly detection unit 12 calculates the similarity between the current object Xi and the rotated object Yj, and determines whether the calculated similarity is equal to or greater than the predetermined threshold. Here, if the current rotation angle exceeds the predetermined upper limit, the anomaly detection unit 12 determines that the current object Yj does not correspond to the current object Xi, returns to step S13, selects the next object Yj, and performs the same process on the object Yj.

[0047] In this manner, the object Yj corresponding to the selected object Xi is searched for and determined, and the coordinate difference and angle difference between the two are stored.

[0048] Then, the anomaly detection unit 12 determines whether or not all objects Xi in the feature map of interest of the reference image have been processed (step S19), and if there are any unprocessed (unselected) objects Xi, the process returns to step S12, selects the next object Xi, and similarly executes the processes from step S13 onwards.

[0049] In this way, all corresponding objects Xi, Yj are identified in the salient feature maps of the reference image and the target image, and the similarities, coordinate differences, and angle differences between the two are identified and stored.

[0050] If it is determined that all objects Xi in the current feature map of interest have been processed, the anomaly detection unit 12 determines whether or not the feature maps of all object types have been processed (step S20), and if there is an object type that has not been processed (unselected), the process returns to step S11, selects the next object type, and similarly executes the processes from step S12 onwards.

[0051] In this manner, the coordinate and angular differences described above are identified and stored for the feature maps of all object types.

[0052] Then, the abnormality detection unit 12 calculates the average value of the stored coordinate differences as the positional deviation between the target image and the reference image for each coordinate X, Y, and calculates the average value of the stored angle differences as the angle deviation between the target image and the reference image (step S21).

[0053] At this time, the coordinate difference and the angle difference of the object whose similarity is lower than a predetermined threshold are excluded from the calculation of the position shift and the angle shift. Also, as described above, if the angle difference of a certain object is different from the average angle difference in the feature map that includes the object by a predetermined value or more, the object is also excluded from the calculation of the position shift and the angle shift.

[0054] In this manner, the positional deviation and the angular deviation are calculated, and the calculated positional deviation and the angular deviation are used as the correction amounts.

[0055] Here, a specific example will be described. Fig. 11 is a diagram showing an example of a reference image and a corresponding target image. The reference image and the target image are A4 size images with an image resolution of 300 dpi, and are 3507 x 2480 pixels, and the actual positional deviation between the two is 10px in the X coordinate, 50px in the Y coordinate, and the angle deviation is 2.5 degrees. The target image contains an abnormal object of one point.

[0056] Fig. 12 is a diagram showing a difference image corresponding to the reference image and the target image shown in Fig. 11. As shown in Fig. 12, the difference image obtained by the above-mentioned processing contains one abnormal object for each point, and the abnormal object is detected by being well distinguished from other objects (objects in the reference image).

[0057] In this specific example, the threshold value of similarity is set to 0.85, and objects with a threshold value less than 0.85 are excluded from the calculation of positional and angular deviations. As a result, the positional deviation error is zero pixels for both the X and Y coordinates, and the angular deviation error is 0.1 degrees. On the other hand, when no threshold value is set (i.e., when exclusion based on similarity is not performed), the positional deviation error is 2 pixels for both the X and Y coordinates, and the angular deviation error is 0.8 degrees. Therefore, by performing exclusion based on similarity, the errors in the correction amounts (positional and angular deviations) are reduced.

[0058] As described above, according to the above embodiment, anomaly detection unit 12 (a) generates a first feature map obtained by performing a filter process on the target image and a second feature map obtained by performing a filter process on the reference image, (b) derives a correction amount based on the deviation between an object in the first feature map and an object in the second feature map, and (c) corrects the target image or the reference image with the correction amount, and then compares the target image with the reference image to detect anomalies in the target image.

[0059] This allows the deviation between the target image and the reference image to be accurately identified without using markers for aligning the two images, and thus allows abnormalities in the target image to be correctly detected.

[0060] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, such changes and modifications are intended to be included within the scope of the claims. [Industrial Applicability]

[0061] The present invention is applicable to, for example, detection of abnormalities in an image forming apparatus or the like. [Explanation of symbols]

[0062] 12 Anomaly detection section

Claims

1. 1. An image processing device for detecting anomalies in a target image by comparing the target image with a reference image, comprising: (a) generating a first feature map obtained by performing a filtering process on the target image and a second feature map obtained by performing the filtering process on the reference image; (b) deriving a correction amount based on a deviation between an object in the first feature map and an object in the second feature map; and (c) comprising an anomaly detection unit that corrects the target image or the reference image with the correction amount and then compares the target image with the reference image to detect an anomaly in the target image, the anomaly detection unit generates a plurality of the first feature maps and a plurality of the second feature maps by filter processing corresponding to a plurality of object types, derives a deviation between an object in the first feature map and an object in the second feature map for each of the object types, and derives the correction amount based on the derived deviation; the anomaly detection unit (a) corrects the first feature map or the second feature map with the correction amount for each of the object types, and then generates a difference image between the first feature map and the second feature map; and (b) detects an anomaly in the target image based on the difference images for the plurality of object types. An image processing device comprising:

2. An image processing device for detecting anomalies in a target image by comparing the target image with a reference image, (a) generating a first feature map obtained by performing a filtering process on the target image and a second feature map obtained by performing the filtering process on the reference image; (b) deriving a correction amount based on a deviation between an object in the first feature map and an object in the second feature map; and (c) comprising an anomaly detection unit that corrects the target image or the reference image with the correction amount and then compares the target image with the reference image to detect an anomaly in the target image, The image processing device according to claim 1, wherein the anomaly detection unit derives the correction amount by excluding objects for which a similarity between an object in the first feature map and an object in the second feature map is less than a predetermined threshold.

3. 3. The image processing apparatus according to claim 1, wherein the abnormality detection unit identifies a cause of the abnormality based on a feature amount of the detected abnormality.

Images