Image processing apparatus and computer-readable storage medium

Abstract

The image processing apparatus (1) (at the time of learning) converts the input image and the label image from the color space of RGB to the color space of CIELAB, and learns the color difference of the inference image and the label image as an error. The image processing apparatus (1) (at the time of inference) converts the input image from the color space of RGB to the color space of CIELAB, and generates the inference image using the input image of CIELAB.

Description

Technical Field

[0001] This invention relates to an image processing apparatus using a neural network and a computer-readable storage medium. Background Technology

[0002] In manufacturing sites such as factories, visual inspections of products manufactured on the production line are performed (e.g., Patent Document 1). To perform visual inspections of products, a machine learning model needs to be generated in advance, which classifies images of normal or abnormal products based on images obtained from photographs of the products.

[0003] When generating such a machine learning model, many images of normal products and many images of abnormal products are collected beforehand. Then, machine learning is performed using the collected images. For the images of abnormal products, there are also situations where it's desirable to pinpoint the location of the anomaly within the product.

[0004] In image classification and recognition, neural network technology is used. Patent document 2 discloses an information processing system comprising: a candidate extraction unit that extracts multiple region candidates from an input image; an image segmentation unit that segments the extracted multiple region candidates; an image classification unit that classifies the extracted multiple region candidates; and a region determination unit that determines the recognition target region by removing a portion of the multiple region candidates based on the segmentation results and the image classification results.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2014-190821

[0008] Patent Document 2: Japanese Patent Application Publication No. 2019-061505 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] A neural network is a mathematical model that represents the nerve cells in the human brain and the connections between them. A neural network consists of an input layer, an output layer, and hidden layers, with weights between these layers representing the strength of the connections between neurons.

[0011] Figure 9 This is a conceptual diagram of a previous image processing device that used a supervised neural network. Figure 9The image processing device generates an inference image from the input image using a neural network. Then, the image processing device compares the inference image with the label image, adjusting the weights of each layer of the neural network little by little to eliminate the error between the label image and the inference result.

[0012] The process of repeatedly updating weights to reduce the error with the correct label is called "learning." In "learning," a learned neural network (model) with optimally adjusted weights is generated. When the amount of learning data used and the learning results and inference performance are good, it can be evaluated as efficient learning.

[0013] In the field of image processing that uses neural networks, the goal is to improve learning efficiency.

[0014] Methods for solving problems

[0015] One aspect of the image processing apparatus disclosed herein includes: a color space conversion unit that converts an image represented in an RGB color space into an image in a color space having luminance as a component; and a learning unit that optimizes a neural network by using the color difference of the image converted in color space by the color space conversion unit as an error.

[0016] One aspect of the image processing apparatus disclosed herein includes: a color space conversion unit that converts the color space of an input image into an image having a color space with luminance as a component; and an inference unit that infers using a neural network that uses the color difference of the input image converted to the color space as an error.

[0017] One aspect of this disclosure is a storage medium that stores computer-readable commands, said commands being: by being executed by one or more processors, converting an image represented in an RGB color space to an image in a color space having luminance as a component, and using the color difference of the converted color space image as an error to optimize a neural network.

[0018] Invention Effects

[0019] According to one aspect of the present invention, the learning efficiency of image processing using neural networks can be improved. Attached Figure Description

[0020] Figure 1 This is a block diagram of the first disclosed image processing apparatus (during learning).

[0021] Figure 2 This is a block diagram of the first disclosed image processing apparatus (in the inference).

[0022] Figure 3 This is a diagram illustrating the color difference between RGB and CIELAB.

[0023] Figure 4 This is a diagram illustrating the differences in the composition of RGB and CIELAB.

[0024] Figure 5 This is a block diagram of the second disclosed image processing apparatus (during learning).

[0025] Figure 6 This diagram illustrates the methods for generating abnormal images and labeled images.

[0026] Figure 7 This is a block diagram of the second disclosed image processing apparatus (inferred).

[0027] Figure 8 It is a diagram showing the hardware structure of an image processing device.

[0028] Figure 9 This is a conceptual diagram of a previous image processing device that uses neural networks. Detailed Implementation

[0029] [First Public Announcement]

[0030] The following description, with reference to the accompanying drawings, outlines the general outline of this disclosure.

[0031] Figure 1 and Figure 2 This is a block diagram of an image processing apparatus 1, which is an example of an image processing apparatus 1. Figure 1 This is a block diagram of the image processing device 1 (during learning). Figure 2 This is a block diagram of image processing device 1 (during inference). Learning and inference can be performed by one image processing device 1, or by different image processing devices respectively.

[0032] Figure 1 The structure and operation of the image processing apparatus 1 (in learning mode) are described. The image processing apparatus 1 (in learning mode) includes: an input image storage unit 10 that stores input image data; a label image storage unit 11 that stores label images; a color space conversion unit 12 that converts the color space of the input image and the label image; and a learning unit 13 that learns to generate an inference image based on the input image and the label image converted by the color space conversion unit 12.

[0033] The input image storage unit 10 stores the image that becomes the learning object. The image that becomes the learning object can be captured by a sensor (described later) attached to the image processing device 1, or it can be an image captured by an external sensor. Alternatively, images stored in a fog computer (described later) or a cloud server (described later) can also be obtained.

[0034] The label image storage unit 11 stores label images. The label images are datasets paired with the input images. The label images serve as the reference for adjusting the inference images. The input and label images are image data represented using the RGB color space. The RGB color space is a color representation method that mixes the three primary colors: red (R), green (G), and blue (B) to represent colors.

[0035] The color space conversion unit 12 converts RGB image data into a color space with luminance as a component. In this disclosure, as an example of a color space with luminance as a component, it converts to CIELAB. CIELAB is a color space with three components: L, a, and b. The L component represents luminance, and the a and b components represent hue as complementary colors. The more positive the a direction, the stronger the red; the more negative the a direction, the stronger the green. Similarly, the more positive the b direction, the stronger the yellow; and the more negative the b direction, the stronger the blue.

[0036] Furthermore, the color space converted by the color space conversion unit 12 can be any color space that has luminance as a component, and there are no particular limitations. For example, in color spaces that also have luminance as a component, in addition to CIELAB, there are L*A*B* color space, L*C*h color space, Hunter Lab color space, etc.

[0037] The learning unit 13 includes a neural network 14. The neural network 14 generates an inference image based on the input image converted to CIELAB by the color space conversion unit 12. The color space of the generated inference image is CIELAB.

[0038] The learning unit 13 inputs the label image converted to CIELAB by the color space conversion unit 12 and compares it with the inference image generated by the neural network 14. The comparison of the label image and the inference image yields the error between the two images. This error is the color difference. Color difference is one of the indicators defined between two colors, defined by distance in a color space. Methods for calculating color difference in the CIELAB color space include Euclidean distance, CIE76, CIE94, and CIEDE2000.

[0039] The learning unit 13 adjusts the weights of the neural network 14 to reduce the error (color difference). The learning unit 13 repeatedly inputs and learns from the input image and the label image to optimize the weights between the nodes of the neural network 14, thereby generating a learned neural network.

[0040] Figure 2The structure and operation of the image processing apparatus 1 (in deduction) are described. The image processing apparatus 1 (in deduction) includes: an input image acquisition unit 15, which acquires an input image; a color space conversion unit 16, which converts the color space of the input image; a deduction unit 17, which generates a deduction image based on the input image; and a color space restoration unit 18, which restores the deduction image to the RGB color space.

[0041] The input image acquisition unit 15 acquires an input image. The input image is image data represented in the RGB color space. The color space conversion unit 16 converts the RGB image data to the CIELAB color space. Furthermore, Figure 2 The color space conversion unit 16 performs the conversion with Figure 1 The same processing is applied to the color space conversion section 12.

[0042] The inference section 17 includes a tuned neural network 14. The tuned neural network 14 is... Figure 1 The learning unit 13 generates a neural network 14. The adjusted neural network 14 generates an inference image based on the input image.

[0043] The color space of the inference image output by the adjusted neural network 14 is CIELAB. The color space restoration unit 18 restores the CIELAB inference image to the RGB color system and outputs an RGB output image.

[0044] As described above, the image processing apparatus 1 of this disclosure uses input images converted to CIELAB and label images to perform learning and inference of the neural network 14. CIELAB and other Lab color spaces are designed to approximate human vision. The L component is extremely close to human perception of brightness. Therefore, learning that approximates human vision is possible.

[0045] The differences in color between RGB and CIELAB are explained.

[0046] exist Figure 3 The image displays four colors: red, orange, green, and light green, along with their corresponding RGB and CIELAB values. The RGB value for red is [240, 110, 80], and the RGB value for orange is [240, 160, 30]. Additionally, the CIELAB value for red is [61.88, 48.04, 40.66], and the CIELAB value for orange is [72.07, 20.60, 71.32].

[0047] The color difference (Euclidean distance) between red and orange in the RGB values ​​is 70.71. The color difference (Euclidean distance) between red and orange in the CIELAB values ​​is 42.38, and the color difference (CIEDE2000) is 25.59.

[0048] The RGB values ​​for green are [150, 250, 30], and for light green are [100, 250, 80]. Additionally, the RGB values ​​for light green are [100, 250, 80]. Furthermore, the CIELAB values ​​for green are [89.37, -58.79, 82.64], and for light green are [87.80, -69.98, 66.69].

[0049] The color difference (Euclidean distance) between green and light green in RGB values ​​is 70.71, which is equal to the color difference between red and orange. The color difference (Euclidean distance) between green and light green in CIELAB values ​​is 19.54, and the color difference (CIEDE2000) is 6.63.

[0050] If judgment is made through human vision, then Figure 3 The difference between red and orange on the left is clear, but the green and light green on the right appear to be similar colors. When calculating color difference (Euclidean distance) using RGB, the color difference between red and orange and the color difference between green and light green are both equal to 70.71. In RGB, the color difference that humans can perceive is not expressed as a numerical value.

[0051] When calculating color difference using CIELAB, the color difference (Euclidean distance) between red and orange is 42.38, and the color difference (Euclidean distance) between green and light green is 19.54. The color difference (Euclidean distance) between red and orange is greater than that between green and light green; the color difference that humans can perceive is expressed as a numerical value.

[0052] In addition, in the CIELAB values, the color difference between red and orange (CIEDE2000) is 25.59, and the color difference between green and light green (CIEDE2000) is 6.63. The color difference between red and orange (CIEDE2000) is greater than the color difference between green and light green (CIEDE2000), and the color difference that humans can perceive is expressed as a numerical value.

[0053] like Figure 4 As shown, the components of RGB are R (red primary color), G (green primary color), and B (blue primary color). The components of CIELAB are L (luminance) and a / b (complementary color dimension). RGB does not include a luminance component. Human vision is highly sensitive to luminance. Human vision values ​​luminance differences more than chroma differences; therefore, a color space with luminance as a component is closer to human vision. That is, by learning using a color space with luminance as a component, it is possible to generate a neural network (model) that closely resembles human vision.

[0054] [Second Public Announcement]

[0055] In the second disclosure, an image processing apparatus 1 for artificially generating abnormal images and performing supervised learning is described. Figure 5This is a block diagram of the second disclosed image processing apparatus 1 (during learning). Figure 7 This is a block diagram of the second disclosed image processing apparatus 1 (in the deduction case).

[0056] Figure 5 The image processing apparatus 1 (in learning mode) includes: a normal image storage unit 30 that stores a normal RGB image obtained by taking a picture of the product; a color space conversion unit 12 that converts the color space of the input image; and a learning unit 21 that generates a neural network model based on the normal image.

[0057] The normal image storage unit 30 stores, for example, image data of a product captured by a sensor (described later) installed on industrial machinery (not shown). The image data stored in the normal image storage unit 30 includes image data of normal products (hereinafter referred to as normal images).

[0058] The color space conversion unit 12 converts the normal image stored in the normal image storage unit 30 from the RGB color system to the CIELAB color space.

[0059] The abnormal image generation unit 19 generates abnormal images based on normal images stored in the normal image storage unit 30. The abnormal images generated by the abnormal image generation unit 19 become input images for learning in a neural network. In product inspection and the like, the number of abnormal image samples is insufficient, so sometimes abnormal images are artificially generated to compensate for the lack of samples.

[0060] The abnormal image generation unit 19 can also generate an abnormal image by overlaying a specified graphic onto a portion of a product image within a normal image. Alternatively, the abnormal image generation unit 19 can generate an abnormal image by altering the hue, saturation, brightness, or adding mosaic effects to a portion of a normal image. Furthermore, the abnormal image generation unit 19 can generate an abnormal image, for example, by adding or reducing (distorting) a specified graphic to a product image within a normal image. The abnormal image is preferably an image encompassing a combination of defects, locations, sizes, colors, background materials, and light source settings that are imagined to occur in an actual location.

[0061] Figure 6This example illustrates how an anomalous images are generated by overlaying a prescribed graphic (geometric graphic) onto a portion of a product image within a normal image. The prescribed image data for the overlay can be stored beforehand, or the geometrically shaped graphic can be generated as the prescribed graphic during the anomalous image generation stage. The color of the prescribed graphic is preferably similar to, but different from, the color of the product. The location of the prescribed graphic to be overlaid on the product image can be determined, for example, by calculating random values. This additional prescribed graphic represents areas of reduced product quality, defects within the product, etc. The prescribed image overlaid on the normal image is saved in CIELAB or converted to CIELAB.

[0062] The color space of a normal image is CIELAB, and the overlapping image data is also CIELAB. Therefore, the input image (abnormal image) generated by the abnormal image generation unit 19 is also CIELAB.

[0063] The learning unit 21 performs learning based on the normal image (label image) converted to CIELAB in the color space conversion unit 12 and the input image (abnormal image) generated by CIELAB in the abnormal image generation unit 19. As a learning method, Pix2Pix or similar methods are used to transform an image into an image by learning a mapping on a pixel-by-pixel basis.

[0064] In the second disclosure, the learning unit 21 generates training data with abnormal images as input images and normal images as label images, and performs learning based on the generated training data.

[0065] The neural network 22 constituting the learning unit 21 can also be used for tasks such as taking an abnormal image as input and outputting a normal image (repair task), or for generating images that emphasize the position and size of abnormal parts of an abnormal image.

[0066] Figure 7 The structure and operation of the image processing apparatus 1 (in deduction) disclosed in the second disclosure are shown. The image processing apparatus 1 (in deduction) includes: an input image acquisition unit 15 that acquires input image data; a color space conversion unit 16 that converts the color space of the input image data; a deduction unit 23 that generates deduction image data based on the input image data; and a color space restoration unit that restores the deduction image data to the RGB color space.

[0067] The input image acquisition unit 15 acquires an input image. The input image is acquired, for example, from a sensor installed on industrial machinery. The color space of the input image is RGB.

[0068] The color space conversion unit 16 converts the input image, which is represented in RGB color space, into CIELAB.

[0069] The inference unit 23 includes a tuned neural network 22 generated by the learning unit 21. The inference unit 23 takes input image data converted to CIELAB as input and generates labeled images representing abnormal parts of the input image data.

[0070] Furthermore, the inference content of the inference unit 23 varies depending on the structure of the learning unit 21. The inference unit 23 may also generate a normal image after removing abnormal parts from the input image, or an image that emphasizes the position and size of the abnormal parts, instead of a labeled image. Regardless of the learning content, the color space of the image data output from the inference unit 23 is CIELAB.

[0071] The color space restoration unit 18 restores the CIELAB image data generated by the inference unit 23 to the RGB color space. Thus, the output image of the image processing device 1 (during restoration) becomes RGB.

[0072] As explained above, Figure 5 Image processing apparatus 1 generates anomalous images from normal image data and uses these anomalous images as samples for supervised learning. Typically, in manufacturing settings such as factories, it is difficult to prepare anomalous images suitable for learning. Therefore, the second disclosed image processing apparatus artificially generates anomalous images to ensure the samples required for learning.

[0073] In the image processing apparatus 1 of the second disclosure, abnormal images are generated using normal images converted to CIELAB. In the second disclosure, a model closer to human vision is generated through learning in the CIELAB color space.

[0074] [Hardware Structure of Image Processing Device]

[0075] Figure 8 This describes the hardware structure of the image processing device 1. The CPU 111 of the image processing device 1 is the processor that controls the entire image processing device 1. The CPU 111 reads the system program stored in the ROM 112 via the bus 122 and controls the entire image processing device 1 according to the system program. Temporary calculation data, display data, and various data input from external sources are temporarily stored in the RAM 113.

[0076] The non-volatile memory 114 is configured such as a memory backed up by a battery (not shown) or an SSD (Solid State Drive), and maintains its storage state even when the power supply to the image processing device 1 is disconnected. The non-volatile memory 114 stores data read from the external device 125 via the interface 115, data input via the input device 124, and data obtained from industrial machinery (not shown) via the network 5. The stored data may include, for example, image data of the product captured by sensors 4 such as vision sensors installed on the industrial machinery. The data stored in the non-volatile memory 114 can be expanded in RAM 113 during execution / use. Furthermore, various system programs, such as known parsing programs, are pre-written into ROM 112.

[0077] Interface 115 is an interface for connecting the CPU 111 of the image processing device 1 to external devices 125 such as USB devices. Data related to products manufactured by various industrial machines (e.g., image data of normal products, CAD data representing the shape of products, etc.) can be read from the external device 125. Furthermore, data edited by the image processing device 1 can be stored via the external device 125 in an external storage unit such as a CF card.

[0078] Interface 120 is used to connect the CPU 111 of the image processing device 1 to the wired or wireless network 5. The network 5 connects to industrial machinery, fog computers 6, cloud servers 7, etc., and exchanges data with the image processing device 1.

[0079] On the display device 123, data read into the memory and data obtained as a result of executing programs are output and displayed via the interface 117. In addition, the input device 124, which consists of a keyboard, indicator devices, etc., transmits instructions and data based on the operator's operation to the CPU 111 via the interface 118.

[0080] in addition, Figure 8 The image processing device is only equipped with a CPU111, but a GPU (Graphical Processing Unit) can also be used in neural network learning.

[0081] The present invention has been described above as an embodiment of the invention, but the present invention is not limited to the example of the above embodiment and can be implemented in various ways by applying appropriate modifications.

[0082] Explanation of reference numerals in the attached figures

[0083] 1 Image processing device

[0084] 5 Networks

[0085] 10 Input Image Storage Unit

[0086] 11-Tag Image Storage Unit

[0087] 12 Color Space Conversion Department

[0088] 13, 21 Study Department

[0089] 14, 22 Neural Networks

[0090] 15 Input Image Acquisition Unit

[0091] 16 Color Space Conversion Section

[0092] Inferences 17 and 23

[0093] 18 Color Space Restoration Department

[0094] 19 Abnormal Image Generation Unit

[0095] 20-label image generation unit

[0096] 24 Tag Storage Department

[0097] 30 Normal Image Storage Unit

[0098] 111 CPU

[0099] 112ROM

[0100] 113 RAM

[0101] 114 non-volatile memory.

Claims

1. An image processing apparatus, characterized in that, The image processing device has: The color space conversion unit converts an image represented in the RGB color space into an image in a color space with luminance as a component; and The learning unit uses the color difference of the image, which has been converted to a different color space by the color space conversion unit, as an error to optimize the neural network. The color space conversion unit converts the input image and label image, which are represented in the RGB color space, into an image with luminance as a component. The learning unit uses the color difference between the input image (which has been converted to a different color space by the color space conversion unit) and the label image as an error to optimize the neural network.

2. An image processing apparatus, characterized in that, The image processing device has: The color space conversion unit converts an image represented in the RGB color space into an image in a color space with luminance as a component; and The learning unit uses the color difference of the image, which has been converted to a different color space by the color space conversion unit, as an error to optimize the neural network. Images represented by the RGB color space are normal images. The image processing apparatus includes an abnormal image generation unit that generates an abnormal image based on the normal image whose color space has been converted by the color space conversion unit. The learning unit uses the normal image, whose color space has been converted by the color space conversion unit, as the label image, the abnormal image as the input image, and the color difference between the normal image and the abnormal image as the error, thereby optimizing the neural network.

3. The image processing apparatus according to claim 2, characterized in that, An image represented by the RGB color space is a normal image obtained by photographing a normal object.

4. An image processing method, characterized in that, Convert an image represented in the RGB color space to an image in a color space with luminance as a component. The color difference of the image converted to the color space is used as error to optimize the learning process of the neural network. By converting the image to the aforementioned color space, the input image and label image, represented in the RGB color space, are transformed into images with luminance as a component. Through the optimized learning, the color difference between the input image and the label image, which have been converted to the color space, is used as an error to optimize the neural network.

5. A storage medium storing computer-readable commands, characterized in that, The command is as follows: The process, executed by one or more processors, converts an image represented in the RGB color space into an image in a color space with luminance as a component. The color difference of the image converted to the color space is used as error to optimize the learning process of the neural network. By converting the image to the aforementioned color space, the input image and label image, represented in the RGB color space, are transformed into images with luminance as a component. Through the optimized learning, the color difference between the input image and the label image, which have been converted to the color space, is used as an error to optimize the neural network.

Images