Image processing device

Abstract

To provide an image processing apparatus for suppressing deterioration of liquid leakage in detection accuracy.SOLUTION: The image processing apparatus is provided with a control unit for performing the processes of: converting a first reference image in which a color is expressed in an RGB (Red, Green and Blue) format and does not include a liquid leakage to an HSV (Hue Saturation Value) format and extracting a second reference image of only an S-component; converting a first detection target image in which a color is expressed in the RGB format to the HSV format and extracting a second detection target image of only an S-component; generating a third reference image and a third detection target image obtained by smoothing the second reference image and the second detection target image, respectively; and detecting the liquid leakage based on a comparison result between the third reference image and the third detection target image.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an image processing apparatus.

Background Art

[0002] There is known an oil leakage detection device that detects oil leakage from oil-filled equipment. The oil leakage detection device includes a light source that irradiates the oil-filled equipment with light, an imaging device that images the oil-filled equipment, a control device that controls the operations of the light source and the imaging device, and an image processing device that processes the captured image. When observing a three-layer structure of brightness with different brightness levels and a dark part in the captured image, the image processing device recognizes the brightest part on the image and the dark part adjacent to the brightest part as the oil leakage adhesion part (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the detection of liquid leakage such as oil leakage or refrigerant leakage, when the above-described brightness is used, the detection accuracy of liquid leakage may decrease. For example, when observing the three-layer structure of brightness in a dark and narrow space and when observing the three-layer structure of brightness in a bright and wide space, different management of the light source is required to control the brightness. That is, the management of the light source changes.

[0005] In such a state where the management of the light source changes, if the management of the light source is incorrect, the brightness may not be appropriately controlled according to the state of the space, and the detection accuracy of liquid leakage may decrease.

[0006] Therefore, an object of the present invention is to provide an image processing apparatus that suppresses a decrease in the detection accuracy of liquid leakage. <[Means for solving the problem]

[0007] The image processing apparatus according to the present invention has a control unit that performs the following processes: converting a first reference image, in which the color is expressed in RGB (Red Green Blue) format and does not contain any liquid leakage, into HSV (Hue Saturation Value) format to extract a second reference image of the S component alone; converting a first detection target image, in which the color is expressed in RGB format, into HSV format to extract a second detection target image of the S component alone; generating a third reference image and a third detection target image by smoothing the second reference image and the second detection target image, respectively; and detecting the liquid leakage based on the comparison result between the third reference image and the third detection target image.

[0008] In the above configuration, the control unit may perform a process to detect a specific area where liquid leakage has occurred, based on a first background difference image which is the result of processing the third reference image and the third detection target image using the background difference method.

[0009] In the above configuration, the control unit may generate a second background difference image by removing noise from the first background difference image based on a predetermined noise reduction method for removing noise from an image, and perform a process to detect the specific area where the liquid leak occurred based on the second background difference image.

[0010] In the above configuration, the control unit may perform a process to determine the type of liquid leaking, based on a trained model that has been machine-learned to represent the oil color expressed in RGB format and the non-oil color expressed in RGB format as the color of the liquid excluding the oil, and the specific area.

[0011] In the above configuration, the control unit may perform a process of acquiring the first reference image and the first detection target image from an imaging device that images an object in which liquid circulates, and displaying the liquid type determination result on a display device. [Effects of the Invention]

[0012] According to the present invention, it is possible to suppress a decrease in the accuracy of liquid leakage detection. [Brief explanation of the drawing]

[0013] [Figure 1] This diagram illustrates an example of an image processing device application. [Figure 2] This is an example of the hardware configuration of an image processing device. [Figure 3] This is an example of the functional configuration of an image processing device. [Figure 4] (a) is an example of a reference image. (b) is an example of an image to be detected. [Figure 5] This is an example of a discriminant model. [Figure 6] This flowchart shows an example of the processing performed by the control unit. [Figure 7] (a) is an example of a background difference image relating to a comparative example. (b) is an example of a background difference image relating to an embodiment. [Modes for carrying out the invention]

[0014] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

[0015] First, an example of the application of the image processing device 100 will be explained with reference to Figure 1. The image processing device 100 is included in the terminal device 10. In Figure 1, a stationary PC (Personal Computer) is shown as an example of the terminal device 10. However, the terminal device 10 may also be a laptop PC or a portable terminal such as a smart device. Note that smart devices include, for example, smartphones and tablet devices.

[0016] An input device 11, a display device 12, and an imaging device 13 are connected to the image processing apparatus 100. The input device 11 includes, for example, a keyboard, a mouse, and a touch panel. The display device 12 includes, for example, a liquid crystal display. The imaging device 13 includes, for example, a video camera that captures video in color and a still camera that continuously captures still images in color. When the terminal device 10 is a mobile terminal, the imaging device 13 is built into the mobile terminal.

[0017] An imaging object 20 is arranged within the imaging range of the imaging device 13. The imaging object 20 includes, for example, an engine 21, a torque converter 22, and an automatic transmission 23. The torque converter 22 is connected to the engine 21. The automatic transmission 23 is connected to the torque converter 22. A propeller shaft 24 is connected to the automatic transmission 23. The automatic transmission 23 is provided with a hydraulic control circuit 25.

[0018] In FIG. 1, a part of the imaging object 20 is included in the imaging range, but all of the imaging object 20 may be included in the imaging range. The imaging device 13 images, for example, each side surface (specifically, the surface of the side wall) of the engine 21, the torque converter 22, and the automatic transmission 23. Engine oil and a liquid coolant circulate inside the engine 21. The liquid coolant may be cooling water or LLC (Long Life Coolant). A lubricating oil such as ATF (Automatic Transmission Fluid) circulates inside the torque converter 22 and the automatic transmission 23.

[0019] As will be described later, the image processing apparatus 100 acquires a captured image that is an image captured by the imaging apparatus 13, executes various image processes, and detects a liquid leak that occurs in any of the engine 21, the torque converter 22, and the automatic transmission 23. In the case of the engine 21, the liquid leak may be engine oil or liquid refrigerant. Therefore, when the image processing apparatus 100 detects a liquid leak, it discriminates the liquid type of the liquid leak and, if necessary, displays the discrimination result on the display device 12. For example, the image processing apparatus 100 displays the liquid type such as engine oil or LLC on the display device 12 together with the captured image including the liquid leak.

[0020] Next, referring to FIG. 2, the hardware configuration of the image processing apparatus 100 will be described. The image processing apparatus 100 includes a CPU (Central Processing Unit) 100A as a processor, a RAM (Random Access Memory) 100B and a ROM (Read Only Memory) 100C as memories. The image processing apparatus 100 includes a network I / F (interface) 100D and a HDD (Hard Disk Drive) 100E. Instead of the HDD (Hard Disk Drive) 100E, an SSD (Solid State Drive) may be adopted.

[0021] The image processing apparatus 100 may include at least one of an input I / F 100F, an output I / F 100G, an input / output I / F 100H, and a drive device 100I as necessary. From the CPU 100A to the drive device 100I are connected to each other by an internal bus 100J. That is, the image processing apparatus 100 can be realized by a computer.

[0022] An input device 11 is connected to input I / F 100F. A display device 12 is connected to output I / F 100G. An imaging device 13 is connected to input / output I / F 100H. If the image processing device 100 includes multiple input / output I / F 100H, semiconductor memory may be connected to input / output I / F 100H. Examples of semiconductor memory include USB (Universal Serial Bus) memory and flash memory. Input / output I / F 100H reads a predetermined program stored in the semiconductor memory. Input I / F 100F and input / output I / F 100H are equipped with, for example, USB ports. Output I / F 100G is equipped with, for example, a DisplayPort.

[0023] A portable recording medium 14 is inserted into the drive unit 100I. The portable recording medium 14 may be a removable disk such as a CD (Compact Disc)-ROM or DVD (Digital Versatile Disc). The drive unit 100I reads a predetermined program recorded on the portable recording medium 14. The network interface 100D includes, for example, a LAN (Local Area Network) port and communication circuits.

[0024] A predetermined program stored in at least one of the ROM 100C, HDD 100E, and semiconductor memory is temporarily stored in RAM 100B by the CPU 100A. A predetermined program recorded on the portable recording medium 14 is also temporarily stored in RAM 100B by the CPU 100A. By executing the stored predetermined program, the CPU 100A realizes various functions described later and also performs various processes described later. The predetermined program may be one that corresponds to the flowchart described later.

[0025] The functional configuration of the image processing device 100 will be described with reference to Figures 3 to 5. Figure 3 shows the main functional parts of the image processing device 100.

[0026] As shown in Figure 3, the image processing device 100 includes a storage unit 110, a control unit 120, an input unit 130, an output unit 140, and an input / output unit 150. The storage unit 110 can be implemented by the RAM 100B or HDD 100E described above. The control unit 120 can be implemented by the CPU 100A described above. The input unit 130 can be implemented by the input I / F 100F described above.

[0027] The output unit 140 can be implemented by the output I / F 100G described above. The input / output unit 150 can be implemented by the input / output I / F 100H described above. Therefore, the storage unit 110, the control unit 120, the input unit 130, the output unit 140, and the input / output unit 150 are connected to each other.

[0028] Here, the storage unit 110 includes a first image storage unit 111, a second image storage unit 112, and a discrimination model storage unit 113. The control unit 120 includes an image acquisition unit 121, an image processing unit 122, and a discrimination unit 123.

[0029] The image acquisition unit 121 acquires captured images from the imaging device 13. Here, the imaging device 13 periodically or continuously captures images of the imaging target 20 in its state before operation (or before startup), in its state during operation, and within the imaging range after the imaging target 20 has finished operation. As a result, the imaging device 13 generates a series of images, either a moving image or multiple still images representing the imaging range. The colors of the captured images are expressed in RGB (Red Green Blue) format. The image acquisition unit 121 acquires such captured images from the imaging device 13.

[0030] When the image acquisition unit 121 acquires an image, it extracts a portion of the image and analyzes the partial image (hereinafter simply referred to as the partial image), which is a part of the image. If the image acquisition unit 121 determines, as a result of analyzing the partial image, that there is no liquid leakage, it stores the liquid-free partial image as the first reference image in the first image storage unit 111. As a result, as shown in Figure 4(a), the first image storage unit 111 stores the first reference image 31 that does not contain liquid leakage. The first reference image 31 may represent any of the following states, as long as it does not contain liquid leakage: the state before operation starts, the state during operation, or the state after operation has ended. The first image storage unit 111 may store one or more first reference images 31.

[0031] On the other hand, regardless of the results of the partial image analysis, the image acquisition unit 121 stores one of the partial images (for example, the most recently captured image) as the first detection target image in the second image storage unit 112. As a result, if the image acquisition unit 121 determines, based on the partial image analysis, that there is a liquid leak on the surface of the object to be imaged 20, the second image storage unit 112 stores the first detection target image 32, which includes the liquid leak 28, as shown in Figure 4(b).

[0032] The image processing unit 122 acquires the first reference image 31 from the first image storage unit 111. Based on a known conversion method for converting from RGB format to HSV (Hue Saturation Value) format, the image processing unit 122 converts the first reference image 31 to HSV format. Then, the image processing unit 122 extracts or separates a second reference image (not shown) consisting solely of the S component (saturation component) from the converted first reference image 31. That is, the H component (hue component) and V component (brightness component) are excluded. The image processing unit 122 also acquires the first detection target image 32 from the second image storage unit 112 and converts the first detection target image 32 to HSV format. Then, the image processing unit 122 extracts or separates a second detection target image (not shown) consisting solely of the S component from the converted first detection target image 32.

[0033] Once the separation into the second reference image and the second detection target image is complete, the image processing unit 122 generates a third reference image and a third detection target image by smoothing the second reference image and the second detection target image, respectively. After generating the third reference image and the third detection target image, the image processing unit 122 detects liquid leakage based on the comparison result between the third reference image and the third detection target image.

[0034] More specifically, the image processing unit 122 detects a specific area where liquid leakage has occurred based on a first background difference image, which is the result of processing the third reference image and the third detection target image using the background difference method. The image processing unit 122 may also generate a second background difference image by removing noise from the first background difference image based on a predetermined noise reduction method for removing noise from the image. The image processing unit 122 may then detect a specific area where liquid leakage has occurred based on the second background difference image. The predetermined noise reduction method can be, for example, either or both of the following: expansion and contraction processing and GLCM (Gray Level Co-occurrence Matrix) processing.

[0035] The dilation and condensation process is a well-known technique, and details will be omitted here, but it includes a dilation process that replaces the pixel of interest with white if even one white pixel exists around it (a kernel size such as 3 pixels x 3 pixels). Furthermore, the dilation and condensation process also includes a condensation process that replaces the pixel of interest with black if even one black pixel exists around it. If a specific area represents an engine oil leak, that area appears relatively large in the image. Therefore, it is assumed that the specific area is unlikely to disappear even if the dilation and condensation process is applied to the first background subtraction image. On the other hand, noise appears as a relatively very small area in the image. Therefore, it is assumed that the noise is likely to be removed if the dilation and condensation process is applied to the first background subtraction image.

[0036] On the other hand, GLCM processing is also a well-known technique, and details will be omitted here, but for example, a GLCM is generated from surrounding pixels arranged around the pixel of interest, and multiple texture features are calculated from the GLCM as characteristic values. These characteristic values ​​include, for example, entropy, dissimilarity, contrast, and energy. Here, if the liquid leak represented by a specific area represents an engine oil leak, unlike noise, there are uniform properties or states unique to oil. For this reason, the entropy tends to be small and the energy tends to be large. Therefore, GLCM processing compares these characteristic values ​​with a predetermined threshold for extracting engine oil leaks individually, removes noise, and detects engine oil leaks. Thus, GLCM processing has the characteristic that pixels with low frequency (e.g., noise areas) are difficult to detect.

[0037] The discrimination model storage unit 113 stores a trained model as the first discrimination model, which is created by machine learning using images of oil colors, such as engine oil and ATF, expressed in RGB format, and images of non-oil colors, such as the colors of liquids other than oil (e.g., LLC and coolant), expressed in RGB format.

[0038] The discrimination model can be represented by a predetermined distribution in a three-dimensional space of the R, G, and B axes, as shown in Figure 5. The predetermined distribution is divided into two areas by the discrimination boundary BD. For example, distribution area AR1 represents the set of R, G, and B values ​​corresponding to oil, and distribution area AR2 represents the set of R, G, and B values ​​corresponding to liquids excluding oil. The R, G, and B values ​​all have 256 levels. Such a discrimination model is generated based on OCSVM (One-Class Support Vector Machine) and stored in advance in the discrimination model storage unit 113.

[0039] Although not shown in the diagram, the discrimination model storage unit 113 may also store a trained model as a second discrimination model, which is created by machine learning using refrigerant colors expressed in RGB format as the color of liquid refrigerant (e.g., LLC or cooling water) and non-refrigerant colors expressed in RGB format as the color of liquids excluding liquid refrigerant (e.g., rainwater or muddy water).

[0040] The discrimination unit 123 acquires a first discrimination model from the discrimination model storage unit 113. Based on the acquired first discrimination model, the specific area detected by the image processing unit 122, and the first detection target image, the discrimination unit 123 determines the type of liquid leaking in the specific area. Processing using the background subtraction method or noise reduction method by the image processing unit 122 may not be able to accurately determine the type of liquid. Therefore, the discrimination unit 123 determines whether the liquid leaking in the specific area is an oil leak or not, based on the first discrimination model, the specific area, and the first detection target image. In this way, the liquid type can be accurately determined by using the first discrimination model. The discrimination unit 123 may also acquire a second discrimination model to determine whether the liquid leaking in the specific area is a liquid refrigerant leak or not.

[0041] Next, the processes executed by the control unit 120 will be described with reference to Figures 6 and 7.

[0042] First, as shown in Figure 6, the image acquisition unit 121 extracts an image from the captured image (step S1). More specifically, the image acquisition unit 121 acquires an image from the imaging device 13, extracts a portion of the captured image, and analyzes the partial image. If the analysis of the partial image determines that there is no liquid leakage, the image acquisition unit 121 stores the partial image without liquid leakage as the first reference image in the first image storage unit 111. In addition, regardless of the result of the partial image analysis, the image acquisition unit 121 stores one of the partial images as the first detection target image in the second image storage unit 112.

[0043] When the processing in step S1 is completed, the image processing unit 122 then acquires a first reference image from the first image storage unit 111, converts the first reference image to HSV format, and extracts a second reference image of the S component alone (step S2). When the processing in step S2 is completed, the image processing unit 122 acquires a first detection target image from the second image storage unit 112, converts the first detection target image to HSV format, and extracts a second detection target image of the S component alone (step S3).

[0044] When the processing in step S3 is completed, the image processing unit 122 performs a smoothing process on both the second reference image and the second detection target image (step S4). By performing the smoothing process, the image processing unit 122 generates a third reference image obtained by smoothing the second reference image, and a third detection target image obtained by smoothing the second detection target image. The smoothing process is a well-known technique and its details will be omitted, but it is a process that removes noise components by replacing each pixel with the average value of the surrounding pixels. The smoothing process reduces abrupt changes in saturation within the image. Therefore, if the liquid leak represents an engine oil leak, the noise portion will be reduced, but the oil portion, which has uniform properties or a uniform state, is expected to remain almost unchanged. Uniform properties or a uniform state refer to, for example, a smooth or flat appearance.

[0045] When the processing in step S4 is completed, the image processing unit 122 performs background subtraction processing based on the third reference image and the third detection target image (step S5). In this embodiment, the background subtraction processing is performed with respect to the first background subtraction image of the third reference image and the third detection target image, outputting white pixels when the saturation difference with surrounding pixels is greater than or equal to a threshold, and outputting black pixels when it is less than the threshold.

[0046] For example, the image processing unit 122 performs background subtraction processing, outputting a white pixel when the brightness difference with surrounding pixels is above a threshold, and outputting a black pixel when it is below the threshold. As a result, as shown in Figure 7(a), the detection accuracy of the specific area 42 that appears in the background subtraction image 41 decreases. Consequently, in subsequent processing, it may not be possible to accurately determine the type of liquid. However, according to the background subtraction processing of this embodiment, as shown in Figure 7(b), the detection accuracy of the specific area 52 that appears in the background subtraction image 51 improves. As a result, in subsequent processing, when determining the type of liquid, it is possible to accurately determine the type of liquid.

[0047] Once the processing in step S5 is complete, the image processing unit 122 removes noise from the background difference image based on a predetermined noise reduction method (step S6). More specifically, the image processing unit 122 generates a noise-reduced background difference image (not shown) as a second background difference image by removing noise from the background difference image 51 as the first background difference image.

[0048] Once the processing in step S6 is complete, the image processing unit 122 detects a specific area based on the denoised background difference image (step S7). If liquid leakage has occurred in the object being imaged 20, the specific area 52 will clearly appear in the denoised background difference image, as described above. Therefore, the image processing unit 122 can detect the specific area 52 with high accuracy. By detecting the specific area 52, the area (region) where liquid leakage has occurred can be identified.

[0049] Once the processing in step S7 is complete, the discrimination unit 123 determines the type of liquid (step S8) and terminates the process. For example, the discrimination unit 123 obtains a first discrimination model from the discrimination model storage unit 113. Then, based on the first discrimination model, the specific area detected by the image processing unit 122, and the first detection target image, the discrimination unit 123 determines whether the liquid leak represented by the specific area 52 is an oil leak or not. The specific area 52 is detected with high accuracy by the various image processing operations performed by the image processing unit 122 as described above. Therefore, it is possible to determine with high accuracy whether the liquid leak represented by the specific area 52 is an oil leak or not.

[0050] As described above, in step S2, the image processing unit 122 of the control unit 120 of the image processing apparatus 100 according to this embodiment performs a process to convert a first reference image, in which the color is expressed in RGB format and does not contain any liquid leakage, into HSV format and extract a second reference image consisting only of the S component. Next, in step S3, the image processing unit 122 performs a process to convert a first detection target image, in which the color is expressed in RGB format, into HSV format and extract a second detection target image consisting only of the S component. After that, in step S4, the image processing unit 122 performs a process to generate a third reference image and a third detection target image by smoothing the second reference image and the second detection target image, respectively. Then, in steps S5 and S7, the image processing unit 122 performs a process to detect liquid leakage 28 based on the comparison result between the third reference image and the third detection target image. These processes can suppress a decrease in the accuracy of liquid leakage detection.

[0051] In this embodiment, an example of detecting a liquid leak on the side of an object to be imaged 20 (for example, an engine 21 or an automatic transmission 23) has been described. Alternatively, a flat-bottomed tray (for example, an oil pan) may be placed under the object to be imaged 20 to receive the liquid, and the imaging device 13 may image the liquid that accumulates in the tray. For example, when colorless and transparent kerosene drips from the object to be imaged 20 into the tray, the kerosene usually drips onto a single point in the tray first, and then gradually spreads across the tray. As a result, there is a possibility that the color difference between the color of the tray and the color of the kerosene will be difficult to discern. Even in such a case, this embodiment can suppress a decrease in the accuracy of kerosene leak detection.

[0052] Although preferred embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims. [Explanation of symbols]

[0053] 10 Terminal devices 13 Imaging device 20 Objects to be imaged 21 Engine 22 Torque Converter 23 Automatic transmission 28. Leakage 100 Image Processing Devices 120 Control Unit

Claims

1. A process to extract a second reference image of the S component alone by converting a first reference image, which is represented in RGB (Red Green Blue) format and does not contain any liquid leakage, into HSV (Hue Saturation Value) format, and A process to convert the first target image for detection, whose color is represented in RGB format, into the HSV format and extract a second target image for detection consisting only of the S component, A process to generate a third reference image and a third detection target image by smoothing the second reference image and the second detection target image, respectively, A process to detect the liquid leak based on the comparison result between the third reference image and the third target image, An image processing apparatus having a control unit that performs the following actions.

2. The control unit detects the specific area where the liquid leak occurred based on a first background difference image, which is the result of processing the third reference image and the third detection target image using the background difference method. An image processing apparatus according to claim 1, which performs processing.

3. The control unit generates a second background difference image by removing noise from the first background difference image based on a predetermined noise reduction method for removing noise from the image, and detects the specific area where the liquid leak occurred based on the second background difference image. The image processing apparatus according to claim 2, characterized by performing processing.

4. The control unit determines the type of liquid leaking based on a trained model that has been machine-learned to represent the oil color expressed in RGB format and the non-oil color expressed in RGB format, the specific area, and the first detection target image. The image processing apparatus according to claim 2 or 3, characterized by performing processing.

5. The control unit acquires the first reference image and the first detection target image from an imaging device that images an object in which liquid circulates, and displays the liquid type determination result on a display device. The image processing apparatus according to claim 4, characterized by performing processing.

Images