Image processing method and apparatus thereof, and controller system using the same

Abstract

To highly accurately and quickly detect abnormality of an image.SOLUTION: An image processing device comprises: a division unit 31 which divides an image, into one or more first images and one or more second images, according to a characteristic of regularity owned by the image; a first processing unit 321 which analyzes the first image divided by the division unit; a second processing unit 322 which analyzes the second image divided by the division unit; and an output unit which performs output on the basis of processing of the first processing unit or the second processing unit.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing method, an apparatus therefor, and a controller system using the same, and particularly to an abnormality detection technique for an image of an article.

Background Art

[0002] In the fields of logistics and production, the need for full automation using AI (artificial intelligence) and robots is increasing. For example, an article moving on a belt conveyor is photographed by a camera, and it is required to recognize the article or detect an abnormality of the article from the photographed image by AI. As an abnormality detection of an image using AI, a method using a neural network is well known.

[0003] In image abnormality detection, in order to improve the abnormality detection accuracy, a method of dividing an image into a plurality of regions and analyzing them is known. This is because the ratio of abnormal parts in the analysis region increases, making detection easier. Regarding this type of technology, for example, Patent Document 1 discloses a defect inspection apparatus that divides an image into a predetermined image size and determines the presence or absence of a defect for each of the divided images using a neural network.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technology described in Patent Document 1, since the amount of analysis required is a multiple of the number of divided images, the analysis time for each divided image becomes long. As a result, it may be difficult to apply it to a production site or the like where rapid inspection is required. As one countermeasure, if the arithmetic processor used for analysis is enhanced, the analysis time of the image can be suppressed, but the device used for arithmetic processing becomes larger. In this case, it becomes difficult to apply it to the detection of abnormalities in articles flowing on a belt conveyor, which requires AI analysis using a compact device such as so-called edge AI.

[0006] Therefore, an object of the present invention is to provide an image processing technology capable of detecting image abnormalities with high accuracy and at high speed.

Means for Solving the Problems

[0007] A preferred example of the image processing apparatus according to the present invention includes a dividing unit that divides an image into one or more first images and one or more second images according to the regularity characteristics of the image, a first processing unit that analyzes the first image divided by the dividing unit, a second processing unit that analyzes the second image divided by the dividing unit, and an output unit that outputs based on the processing of the first processing unit or the second processing unit. It is an image processing apparatus having the above components.

[0008] A preferred example of the image processing method according to the present invention includes a dividing step of dividing an image into one or more first images and one or more second images according to the regularity characteristics of the image, a first processing step of analyzing the first image divided by the dividing step, a second processing step of analyzing the second image divided by the dividing step, and an output step of outputting based on the processing of the first processing step or the second processing step. It is an image processing method having the above components.

[0009] A preferred example of the controller system according to the present invention is a controller system having a controller for controlling the operation of an article and an image processing device for processing an image of the article, wherein the image processing device has a dividing unit that divides an image into one or more first images and one or more second images according to the regular characteristics of the image, a first processing unit that analyzes the first image divided by the dividing unit, a second processing unit that analyzes the second image divided by the dividing unit, and a determination unit that determines an abnormality of the image according to the processing of the first processing unit or the second processing unit, wherein the controller controls the operation of the article according to the determination by the determination unit. The controller system is characterized by this.

Effects of the Invention

[0010] According to the present invention, an abnormality of an image can be detected with high precision and high speed.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0012] Hereinafter, with reference to the drawings, preferred embodiments of the present invention will be described.

Examples

[0013] Example 1 will be described with reference to FIGS. 1 to 5.

[0014] FIG. 1 is a diagram showing the configuration of an image anomaly detection device. An image anomaly detection device 13 and a controller 14 are connected to form a system. This system may be referred to as an anomaly detection system, a controller system, or simply a controller. The controller 14 is a device that controls the operation of an article, for example, a device that controls a belt conveyor that moves the article. One or more cameras 15 for taking images of the article moving on the belt conveyor are installed near the belt conveyor. The controller 14 is an industrial controller that controls the stop and movement of the belt conveyor according to the image anomaly detection by the image anomaly detection device 13 (paragraph 121 of Patent Document 2). In one example, according to the output from the image anomaly detection device 13, the controller 14 controls, by executing a program on the arithmetic processor (CPU) it has, to move (exclude) the article determined to be abnormal from the belt conveyor.

[0015] The image abnormality detection device 13 has a plurality of arithmetic processors 11 and a memory 12, and is an information processing device that analyzes an image of an article to perform abnormality detection. In one example, the plurality of arithmetic processors 11 include a plurality of GPUs (Graphics Processing Units) and a plurality of CPUs (Central Processing Units), and execute a program for performing abnormality analysis and related processes. The memory 12 is a semiconductor storage device (storage unit), and stores the image 16 acquired by the camera 15. The memory 12 stores the acquired image 16 (original image and its divided images (divided images)), programs for performing abnormality analysis, data and programs 17 for AI analysis and rule-based analysis, various management information, and the like. As one of the management information, there is an image management table 18 (FIG. 5).

[0016] Although not shown in FIG. 1, an input device for a user such as an administrator to input various information and a display device for displaying the image 16 and the determination result of abnormality detection are connected to the image abnormality detection device 13. In addition, the image abnormality detection device 13 can be configured as a single information processing device by itself. As another example, it can be incorporated into a part of the controller 14 and configured as a controller or a controller system.

[0017] <Explanation of Image and Divided Image> FIG. 2 shows an example of an image to be subjected to abnormality detection. In a preferred example of the present invention, the image 16 is divided into a plurality of regions according to the presence or absence of regularity. In the example of FIG. 2A, an image region 21 with a pattern or characters is divided as a region without regularity. On the other hand, an image region 22 with a periodic lattice pattern and a plain region 23 are divided as regions with regularity. In the example of FIG. 2B, the image regions 24, 26, 28 are divided as regions without regularity, and the image regions 25, 27 are divided as regions with regularity.

[0018] Here, "with regularity" refers to an image such as a repeating pattern or a plain color, in other words, an image for which abnormality determination is relatively easy. "Without regularity" refers to an image that includes patterns, characters, or symbols, in other words, an image for which abnormality determination is relatively difficult. Regarding the possibility of determination, for example, when dirt or stains are attached to an image of an article or the label is damaged, it is difficult to determine whether these dirt, stains, damage, etc. are part of the characters or symbols or are original dirt, stains, etc. On the other hand, even if dirt or stains are attached to or there is damage to a repeating pattern or a plain color, it is easy to determine that these dirt, stains, etc. disrupt the regularity.

[0019] In a preferred example of the present invention, for an area without regularity, analysis is performed using AI. For example, AI analysis using a deep neural network (hereinafter referred to as DNN) or a type of neural network called deep learning is applied. On the other hand, for an area with regularity, instead of AI analysis, analysis with a pre-determined procedure (rule-based analysis) is applied. In the example of FIG. 2A, for the image area 21 without regularity, anomaly analysis by deep learning is performed, and for the image areas 22, 23 with regularity, instead of AI analysis, analysis with a pre-determined procedure (rule-based analysis) is applied. Similarly, in the example of FIG. 2B, for the image areas 24, 26, 28 without regularity, AI analysis is applied, and for the image areas 25, 27 with regularity, rule-based analysis is applied. For example, in an area with a periodic pattern, if the period of the RGB (Red-Green-Blue color model) intensity of the image photo is disrupted, it is determined as abnormal. Also, in a plain area, if the absolute value of the derivative with respect to the spatial coordinates of the RGB intensity exceeds a threshold value, it is considered abnormal.

[0020] Generally, rule-based analysis has a relatively small computational amount and a short calculation time, but DNN analysis has a significantly larger computational amount and a longer calculation time compared to rule-based analysis. Therefore, as in this embodiment, if the area to which deep learning analysis is applied and the area to which rule-based analysis is applied are separated for analysis, the calculation time can be significantly shortened compared to the conventional method of applying deep learning analysis to all the divided areas.

[0021] Also, in a preferred example of the present invention, an arithmetic processor according to the analysis method is used. For example, for the image region 21 with patterns or characters, deep learning analysis is performed using a GPU, which is an arithmetic processor suitable for deep learning analysis. For the image region 22 with a periodic lattice pattern and the plain region 23, rule-based analysis is performed using a CPU. In this way, image analysis is performed using an arithmetic processor suitable for the characteristics of each image region. Furthermore, by executing the analysis by the GPU and the analysis by the CPU in parallel, the calculation time can be shortened.

[0022] Note that, as examples of the arithmetic processors used for image analysis, a GPU and a CPU are mentioned, but the arithmetic processors that can be used in the present invention are not limited to these. For example, any arithmetic processor such as an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or an SoC (System on Chip) can be used.

[0023] <Functions of the Image Abnormality Detection Device> Figure 3 is a functional block diagram of the image abnormality detection device. The image abnormality detection device includes an image division unit 31 that divides the image 30 of an article into a plurality of parts, a plurality of image analysis units 32 that perform processing according to the characteristics (i.e., the presence or absence of regularity) of the divided images (divided images), and an abnormality determination unit 33 that determines an abnormality according to the analysis results for each divided image. Here, the functions of the image division unit 31 to the abnormality determination unit 33 are realized by the program 17 being executed by any one of the arithmetic processors 11. The image 30 is an image acquired by the camera 15 and is, for example, the image shown in Figure 2. An article ID for identifying the article is input to the image division unit 31, and the image 30 is divided according to the characteristics of the image. For this purpose, an image management table 18 for managing the processing of the divided images for each article is used.

[0024] Note that the image division unit 31 may also be called a region division unit or simply a division processing unit since it divides the region of the image. Or it may be called an image characteristic processing unit since it performs division according to the characteristics of the image 30.

[0025] The plurality of image analysis units 32 are functional units that perform image analysis by means of the plurality of respective arithmetic processors 11. (Specifically, they are parts that analyze the divided images, but hereinafter simply referred to as image analysis). Among the plurality of image analysis units 32, one or more image analysis units are deep learning analysis units (not shown but denoted by reference numeral 321) that perform image analysis using deep learning by means of one or more GPUs, and the other one or more image analysis units are rule-based analysis units (not shown but denoted by reference numeral 322) that perform rule-based analysis by means of one or more CPUs. Note that it is not necessary for all of the prepared GPUs and CPUs to be used to realize the functions of the image analysis units 321 and 322, and there may be unused GPUs and CPUs. For example, in the processing of the image in FIG. 2A, one GPU and two CPUs are used, and the corresponding image analysis units 321 and 322 function. In the processing of the image in FIG. 2B, three GPUs and two CPUs are used, and the corresponding image analysis units 321 and 322 function.

[0026] The abnormality determination unit 33 determines whether there is an abnormality based on the analysis results of the divided images by the image analysis units 321 and 322. An abnormality means, for example, a case where the divided image has dirt, stains, etc. and they exceed the allowable range. The determination result by the abnormality determination unit 33 is sent to the controller 14, and the controller 14 controls, for example, to exclude the target article from the line of the belt conveyor.

[0027] <Image Management Table> FIG. 5 shows a configuration example of the image management table 18. The image management table (hereinafter simply referred to as the management table) 18 defines, for each article, the divided areas of the article and the image analysis processing method for the divided areas. The management table 18 holds items related to the article ID for identifying an article, the area ID for identifying a divided area, the area coordinates defining the coordinates of each divided area, the presence or absence of regularity in the divided area, the type of image analysis method for the divided area, the assigned PU for identifying the arithmetic processor assigned for the image analysis of the divided area, and the intensity and threshold values used for the image analysis of the divided area. Here, the article ID is, for example, information for identifying the articles in FIGS. 2A and 2B. The area ID is, for example, information for identifying the divided areas 21, 22, 23 in FIG. 2A, such as 0021, 0022, 0023. The area coordinates are the X-Y coordinate values of each of the divided areas 21 to 23. Regarding the presence or absence of regularity, the area "0021" is indicated as "none" because it has no regularity, and the area 0022 is indicated as "yes" because it has regularity. The image analysis method manages whether AI or rule-based is used for the analysis of the divided area, that is, the type of AI used and the type of rule-based. The assigned PU defines the identification information of the arithmetic processor 11 assigned for the processing by AI, or the identification information of the arithmetic processor 11 assigned for the processing by rule-based. Note that the assigned PU may be information for identifying the image analysis unit 32 instead of the identification of the arithmetic processor 11 itself. The intensity and threshold values are values used in the rule-based image analysis.

[0028] Note that the items managed by the management table 18 are not limited to the items shown in FIG. 5. For example, other information can be added. For example, an item indicating that the analysis result of the divided image by AI or rule-based, that is, the determination result is "abnormal" or "normal" may be added. Also, when recording the daily image determination results as a log, it may be associated with the corresponding log. Also, the intensity and threshold values may be managed in association with the rule-based instead of being stored in the management table 18. Furthermore, the management table does not necessarily have to be in a single table configuration as long as the above items are associated. Also, instead of calling it the management table, it may simply be called image management information or management information structure, or database (DB).

[0029] <Abnormal determination processing operation> With reference to FIG. 4, the processing operation of abnormal determination by the image abnormality detection device will be described. For example, take the abnormal determination of an article moving on a belt conveyor at the production site of a product as an example. As a premise, in order to identify the article to be produced on the day, the administrator inputs the article ID from the input device. In addition, the camera 15 arranged near the belt conveyor sequentially captures images of a plurality of moving articles. The article ID and the image of the article are stored in the memory 12.

[0030] First, the region division unit 31 acquires the image stored in the memory 12 (S411), and also acquires the article ID (S412). Then, the region division unit 31 refers to the management table 18 stored in the memory 12 to perform a structure determination on the image 16 (S42). The structure determination is an operation for creating a divided image by determining which of the divided images constituting the target image 16 has a regular structure or an irregular structure with reference to the management table 18. For example, when the article ID is "0001", the divided images 21, 22, 23 are cut out based on the region coordinates. Each divided image is assigned a region ID and temporarily stored in the memory 12. Note that the divided image may also be called a partial image.

[0031] For each divided image, the presence or absence of regularity in the management table 18 is determined (S43). As a result of the determination, if there is regularity (regular structure), rule-based analysis is specified (S441), and if there is no regularity (irregular structure), AI analysis such as deep learning (DNN) is specified (S442). For example, for the divided image 21, one arithmetic processor 11 for performing AI analysis (that is, the image analysis unit 321 by AI) is specified, and for the divided images 22, 23, two arithmetic processors 11 for performing rule-based analysis (that is, two image analysis units 322 by rule-based) are respectively specified. Note that the above image analysis units 321, 322 may also be called an image processing unit or simply a processing unit.

[0032] In rule-based analysis (S441), the designated arithmetic processor 11 analyzes the image according to the rule base pre-stored in the memory 12. This will be described later with reference to FIGS. 6 and 7. In AI analysis (S442), the designated arithmetic processor 11 inputs the divided image 21 into a pre-trained neural network and calculates the output.

[0033] Next, the abnormality determination unit 33 determines whether there is an abnormality in the result of the rule-based analysis (S451) and determines whether there is an abnormality in the AI analysis result (S452). As a result of the determination of the presence or absence of an abnormality, if an abnormality is detected in the result of the rule-based analysis (S451: Y), or if an abnormality is detected in the output of the AI analysis (S452: Y) (S47), it is determined that there is an abnormality (S48). In the case of an abnormality, the abnormality determination unit 33 issues a command indicating the abnormality to the controller 14. The arithmetic processor of the controller 14 controls, for example, to exclude the target article on the belt conveyor from the normal route.

[0034] On the other hand, as a result of the determination of the presence or absence of an abnormality, if no abnormality is detected, the processes are terminated respectively (S461, 462). After that, if the next image 16 is stored in the memory 12, the image is read out and the same processing as the above operation is performed.

[0035] <Example of rule-based analysis (Example 1)> With reference to FIGS. 6 and 7, an example of analysis by the rule base (processing of S441) will be described. FIG. 6 shows an image to be the abnormality detection target. This example shows an example in which rule-based analysis is applied to an image in which a regular image (for example, an image area 23 that is a divided image) is a plain area 60 and there are abnormal areas 61 and 62 in the area 60. Assume that the RGB intensity is higher in the abnormal area 61 and lower in the abnormal area 62 than in the plain area 60.

[0036] Figure 7 shows a waveform diagram in the abnormal detection of an image. In the waveform diagram of Figure 7, the horizontal axis represents the position of the image, and the vertical axis represents the image intensity. That is, it shows the RGB intensity 71 of the image taken in the x direction along the dashed line in Figure 6. Reference numeral 72 is the average value of the RGB intensities obtained for the entire image shown in Figure 6. An upper threshold 73 for the RGB intensity and a lower threshold 74 for the RGB intensity are set above and below the average value 72, respectively. The set values of the upper threshold 73 and the lower threshold 74 can be easily determined by measuring the RGB intensities of a plurality of normal images and using the upper and lower limits of their fluctuations.

[0037] Based on such settings, when analyzing the image, it can be easily understood that the region where the RGB intensity 71 exceeds the upper threshold 73 of the RGB intensity is the abnormal region 75, which corresponds to the abnormal region 61 in Figure 6. Similarly, it can be easily understood that the region where the RGB intensity 71 is below the lower threshold 74 of the RGB intensity is the abnormal region 76, which corresponds to the abnormal region 62 in Figure 6. Note that the examples in Figures 6 and 7 are examples of abnormal detection of an image taken in the x direction, but abnormal detection can also be performed in the y direction of Figure 6 by a similar method.

[0038] Note that the data such as the RGB intensity 71, the average value 72, the upper threshold 73, the lower threshold 74, and the abnormal regions 75 and 76 calculated above are associated with the region ID of the region image 60 and stored in the memory 12. And they are used as needed in the rule-based analysis (S441).

[0039] <Comparative Example of Image Analysis> Figure 8 shows a time chart for explaining the comparison of image analysis. (1) in Figure 8 shows an example of conventional image analysis, and (2) shows an example by the image analysis of this embodiment. The horizontal axis represents the elapsed time. For example, assume that the image is divided into 16 parts, and among them, two region images (divided images) are images without regularity (irregular), and 14 divided images are images with regularity.

[0040] In the conventional example, as shown in (1), deep learning (DNN) is assigned to all 16 divided images for anomaly analysis. That is, after the CPU determines the structure of the image processing 51, the GPU continuously performs anomaly analysis 52 by deep learning on the 16 divided images.

[0041] On the other hand, in this embodiment, as shown in (2), after the CPU determines the structure of the image processing 51, the CPU continuously performs anomaly analysis 53 by a rule-based method (non-DNN) on 14 divided images having regularity. Here, in parallel with the anomaly analysis 53 by the rule-based method by the CPU, the GPU continuously performs anomaly analysis 52 by deep learning on two divided images having no regularity. Generally, the time required for image analysis is overwhelmingly shorter for the anomaly analysis 53 by the rule-based method than for the anomaly analysis 52 by deep learning. Therefore, the analysis according to this embodiment is significantly shorter than the conventional one as shown in the figure. Thus, according to the image analysis method of this embodiment, the anomaly detection accuracy for the divided images is improved and high-speed analysis becomes possible.

[0042] In the above, the case where rule-based analysis is performed using the CPU and AI analysis is performed using the GPU has been described as an example. However, rule-based analysis may be performed using the GPU and AI analysis may be performed using the CPU. Furthermore, for the rule-based analysis and AI analysis of the image anomaly detection device of the present invention, not only the CPU and GPU but also any arithmetic processor such as FPGA, ASIC, and SoC can be used. Also, in the image anomaly detection device of this embodiment, the CPU, GPU, or other arithmetic processor built into the controller may also be used for rule-based analysis or AI analysis.

Embodiment

[0043] The analysis of images by the rule-based method is not limited to the above Example 1, and several more examples can be given. Hereinafter, with reference to FIGS. 9 to 11, examples of the rule-based analysis will be described.

[0044] <Rule-based analysis example (Example 2)> Example 2 will be described with reference to FIGS. 6 and 9. FIG. 9 is a diagram showing an example (Example 2) of a waveform in the abnormal detection of an image by rule-based analysis. Also in Example 2, as shown in FIG. 6, a normal region as an example of an image having regularity is a plain region 60, and an image having abnormal regions 61 and 62 in that region 60 is targeted for abnormal detection.

[0045] In the waveform diagram of FIG. 9, the differential value 81 with respect to the x coordinate of the RGB intensity, the upper threshold value 82 of the differential value with respect to the x coordinate of the RGB intensity, and the lower threshold value 83 of the differential value with respect to the x coordinate of the lower RGB intensity of the image taken in the x direction along the dashed line in FIG. 6 are shown. In this example, the range from the point where the differential value 81 with respect to the x coordinate of the RGB intensity exceeds the upper threshold value 82 with respect to the x coordinate to the point where it falls below the lower threshold value 83 with respect to the x coordinate is determined to be the abnormal region 84. Since this abnormal region 84 first exceeds the upper threshold value, it can be seen that it corresponds to the abnormal region 61 in FIG. 6. Similarly, after the abnormal region 84, the range from the point where the differential value 81 with respect to the x coordinate of the RGB intensity falls below the lower threshold value 83 with respect to the x coordinate to the point where it exceeds the upper threshold value 82 with respect to the x coordinate is determined to be an abnormal region. Since this abnormal region 85 first falls below the lower threshold value, it can be seen that it corresponds to the abnormal region 62 in FIG. 6. Note that abnormal detection can also be performed in the y direction in FIG. 6 by a similar method.

[0046] The set values of the upper threshold value 82 and the lower threshold value 83 can be easily determined using the upper and lower limits of the fluctuations by obtaining the differential values of the RGB intensities of a plurality of normal images.

[0047] <Rule-based analysis example (Example 3)> Example 3 will be described with reference to FIGS. 10 and 11. As shown in FIG. 10, the normal region of the regular image (segmented image) is the lattice pattern region 90, and abnormal regions 91 and 92 are present in that region 90. In the lattice pattern region 90, sites with high RGB intensity and low RGB intensity are alternately arranged. The RGB intensity of the abnormal region 91 is higher than that of the sites with high RGB intensity in the lattice pattern region 90, and the RGB intensity of the abnormal region 92 is the same as that of the sites with low RGB intensity in the lattice pattern region 90.

[0048] In the waveform diagram of FIG. 11, the RGB intensity 101 of the image taken in the x - direction along the broken line in FIG. 10 is shown. The ranges indicated by L and H on the horizontal axis of FIG. 11 are the ranges where the RGB intensity of the normal lattice pattern becomes low and high, respectively. In FIG. 11, an upper threshold 102 for the RGB intensity in the range where the RGB intensity of the normal lattice pattern becomes low, a lower threshold 103 and an upper threshold 104 for the RGB intensity in the range where the RGB intensity of the normal lattice pattern becomes high are set. The abnormal regions can be detected using these thresholds 103, 104. That is, since the RGB intensity in the abnormal region 105 exceeds the respective upper thresholds for both the sites where the RGB intensity of the normal lattice pattern becomes low and high, it can be determined as the abnormal region 91. Also, since the RGB intensity in the abnormal region 106 is lower than the lower threshold for the sites where the RGB intensity of the normal lattice pattern included in that region becomes high, it can be determined as the abnormal region 92. Note that abnormal detection can also be performed in the same manner for the y - direction in FIG. 10. Also, the set values of the thresholds used in Example 3 can be easily determined by measuring the RGB intensities of a plurality of normal images and using the upper and lower limits of their fluctuations.

Example

[0049] With reference to FIG. 12, an example of the determination of image analysis will be described. In the image processing of Example 1, since a normal image is known, it is also known which regions of the image have regularity and which regions do not (irregular). Also, when detecting an abnormality in an image, it is often the case that an image of an object is acquired while keeping the positional relationship between the object and the camera constant. Therefore, in such a case, it is also known from which coordinates to which coordinates of the acquired image have regularity, so based on the coordinates of the image, the necessary region division in this example can be performed. At the same time, it is also possible to determine the abnormality analysis method to be applied to the divided regions, that is, whether it is deep learning analysis or rule-based analysis.

[0050] Incidentally, during the detection of an image of an article on a belt conveyor, due to some accident, a situation may occur where, for example, a large amount of articles continue to flow onto the belt conveyor with the arrangement of the articles shifted. This embodiment can present a method for confirming whether the region division is being performed normally even when such a situation occurs. For example, as shown in FIG. 12, consider an example where an image is composed of a plain region 111, a stripe region 112 with a single cycle, a region 113 where two types of periodic stripes coexist, and a region 114 having characters or patterns. FIG. 12 shows a spatial FFT (Fast Fourier Transform) image 115 when this image is arranged as expected. The horizontal axis of the graph in the spatial FFT image 115 is the frequency, and the vertical axis is the FFT intensity. Here, (1) to (4) correspond to the regions 111 to 114. In the plain region 111 of (1), noise with low FFT intensity occurs randomly with respect to the frequency. Also shown is an upper limit threshold value 116 of the FFT intensity for determining that it is a plain region. Since the stripe region 112 of (2) has a single stripe period, only one frequency with high FFT intensity is generated. Since the region 113 of (3) has two stripe periods, two frequencies with high FFT intensity are generated. Since the region 114 of (4) has no regularity, frequencies with high FFT intensity occur randomly. Note that the noise generated in (1) often overlaps with (2) to (4) as well, but they can be determined as noise by considering the threshold value 116. Whether the target article is arranged as expected can be determined by such a spatial FFT image. That is, if the spatial FFT results of each divided region are as described above, it can be determined that the article is arranged as expected.

[0051] On the other hand, when performing the abnormality detection according to this embodiment even when the arrangement is not as expected or the arrangement is not constant for each article, the above spatial FFT may be tried by changing the image division position. The predetermined methods may be used for analysis at the division positions where the spatial FFT intensities corresponding to the above (1) to (4) are obtained. By using such a method, it is possible to automatically determine the position of the region division of the image and the analysis method for each divided region in this embodiment. Note that the upper limit threshold value 116 of the FFT intensity for determining that it is the plain region 111 can be easily obtained from the fluctuations by measuring the RGB intensities of a plurality of normal plain regions.

Embodiment

[0052] Modification The present invention is not limited to the above embodiment and can be implemented with various modifications or applications. For example, in the first embodiment, an example in which the image abnormality detection device is applied to a controller that controls a belt conveyor at a production site was described. According to another example, it is applicable to the abnormality detection of articles in distribution. Since various types of articles are handled in distribution, all of their article IDs and divided images, etc. will be managed in an image management table. Also, at a distribution site, it is not practical to input the article IDs of various types of articles flowing on the belt conveyor one by one from an input device. Therefore, it is meaningful to identify an article using an image automatic recognition technology that recognizes an image acquired by a camera to identify the article and index the article ID in the image management table.

[0053] In addition, the present invention is not limited to the production field or the distribution field, and is also applicable to the analysis of general images. As an example, the abnormality determination unit 33 does not necessarily have to function as a determination unit that determines the abnormality of an image according to the analysis result of the image analysis unit 32. For example, not limited to abnormalities, it may function as an output unit that outputs the analysis results of the images by the image analysis units 321 and 322 each time. Alternatively, it may function as an output unit that temporarily stores the analysis results of the images in the memory 12 or the image management table 18.

Explanation of Reference Numerals

[0054] 11: Calculation processor 12: Memory 13: Image abnormality detection device 14: Controller 15: Camera 16: Image 17: Program 18: Image management table 31: Image segmentation unit 32, 321, 322: Image analysis unit 33: Abnormality determination unit

Claims

Claim 1: For one or more regions of an image of an article, a characteristic that the image of the region has regularity (characteristic with regularity) or a characteristic that the image of the region has no regularity (characteristic without regularity) is predetermined. A dividing unit that divides an image of an article to be processed into a first image that is one or more divided images having the characteristic with regularity and a second image that is one or more divided images having the characteristic without regularity for each of the regions. A first processing unit that performs analysis using a rule base on the first image divided by the dividing unit. A second processing unit that performs analysis using deep learning on the second image divided by the dividing unit. An abnormality determination unit that makes an abnormality determination based on the processing of the first processing unit or the second processing unit. It has The analysis of the first image by the first processing unit and the analysis of the second image by the second processing unit are executed in parallel using arithmetic processors with performance according to the analysis method respectively. An image processing apparatus.

2. The first processing unit is executed by a CPU. The second processing unit is executed by a GPU. The CPU and the GPU execute processing in parallel. The image processing apparatus according to Claim 1.

3. The first processing unit obtains a threshold value used for the rule base from the fluctuation of the RGB intensity of a normal image. The image processing apparatus according to Claim 1.

4. The first processing unit obtains a threshold value used for the rule base from the fluctuation of the spatial derivative of the RGB intensity of a normal image. The image processing apparatus according to Claim 1.

5. The dividing unit obtains a dividing position based on the spatial FFT of the image. The image processing apparatus according to Claim 1.

6. The dividing unit determines that a repetitive pattern or a plain image has the characteristic with regularity and determines it as the first image, and determines that an image including a pattern or a character or a symbol has the characteristic without regularity and determines it as the second image. The image processing apparatus according to Claim 1.

7. It has an image management table that manages an article ID for identifying an article having an image to be processed, a region ID for identifying an image of a plurality of regions constituting the image of the article, the presence or absence of the regularity defined for each of the region images, and an analysis type indicating the analysis of the first image or the second image applied to each of the region images. The dividing unit refers to the image management table, makes a structure determination of the image, and divides it into a plurality of region images. The image processing apparatus according to Claim 1. Claim 8: For one or more regions of an image of an article, a characteristic that the image of the region has regularity (characteristic with regularity) or a characteristic that the image of the region has no regularity (characteristic without regularity) is predetermined. A splitting step of splitting an image of an article to be processed into a first image which is one or more split images having the characteristic with regularity and a second image which is one or more split images having the characteristic without regularity for each of the regions; A first processing step of performing an analysis using a rule base on the first image split by the splitting step; A second processing step of performing an analysis using deep learning on the second image split by the splitting step; An abnormality determination step of performing an abnormality determination based on the processing of the first processing step or the second processing step, and The analysis of the first image by the first processing step and the analysis of the second image by the second processing step are executed in parallel using arithmetic processors with performance corresponding to the analysis methods respectively. An image processing method. Claim 9 The first processing step is executed by a CPU. The second processing step is executed by a GPU. The CPU and the GPU execute processing in parallel. The image processing method according to Claim 8. Claim 10 In the splitting step, a repeating pattern or a plain image is determined as the first image by determining it as a characteristic with regularity, and an image including a pattern or a character or a symbol is determined as the second image by determining it as a characteristic without regularity. The image processing method according to Claim 8.

Images