Image processing device and program

Abstract

To provide an image processing device and a program capable of mitigating or removing a reflection image on an object of photographing.SOLUTION: An image processing device 1 accepts image data being a processing object, and detects a prescribed object from the accepted image data. Then the image processing device executes detection processing of a three dimensional object in a photographing range of the detected object, and executes prescribed processing to the image data being the processing object by using information regarding the detected three dimensional object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and a program.

Background Art

[0002] When photographing a scene including an object with a relatively high reflectivity such as metal, an intended image may not be obtained due to a reflected image on the object. For example, in a scene including a mirror surface, an image of the photographer himself / herself may be captured, which may not be the intended image of the photographer.

[0003] Also, in an image captured for industrial purposes, if reflection is captured, the purpose may not be achieved. As an example, when imaging a dent (depression) generated on a vehicle, if an image of the surroundings is captured on the vehicle surface, the dent may become less noticeable, or an image that may be mistaken for the dent may be captured.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique disclosed in Non-Patent Document 1, it is considered to estimate and remove the reflection on the glass surface in consideration of the case where there is a translucent glass or the like between the object to be imaged and the photographer by machine deep learning.

[0006] However, the technology described in Non-Patent Document 1 does not take into account the removal of reflections from the object. As mentioned above, the technology to remove reflections from an object is useful not only for general photography purposes but also when taking images for industrial applications such as vehicle dent analysis, and there is a demand for such technology.

[0007] This invention has been made in view of the above circumstances, and one of its objectives is to provide an image processing device and program that can reduce or remove reflected images in an object being imaged. [Means for solving the problem]

[0008] One aspect of the present invention, which solves the problems of the above-mentioned conventional example, is an image processing apparatus comprising: receiving means for receiving image data to be processed; primary detection means for detecting a predetermined object from the received image data; secondary detection means for performing a three-dimensional object detection process within the imaging range of the detected object; and image processing means for performing a predetermined process on the image data to be processed using information relating to the three-dimensional object detected by the secondary detection means. [Effects of the Invention]

[0009] According to the present invention, reflected images in the object being imaged can be reduced or removed. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 2] This is a functional block diagram showing an example of an image processing apparatus according to an embodiment of the present invention. [Figure 3] This is an explanatory diagram illustrating an example of operation of an image processing apparatus according to an embodiment of the present invention. [Figure 4] This is a flowchart illustrating an example of the operation of an image processing apparatus according to an embodiment of the present invention. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described with reference to the drawings. The image processing apparatus 1 according to an embodiment of the present invention is configured to include a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and an input / output unit 15, as illustrated in Figure 1.

[0012] Here, the control unit 11 is a program control device such as a CPU, and operates according to a program stored in the memory unit 12. In this embodiment, the control unit 11 receives image data to be processed and detects a predetermined object from the received image data. The control unit 11 also performs a three-dimensional object detection process within the range in which the detected object is imaged, and uses the information relating to the three-dimensional object detected in this detection process to perform a predetermined process on the image data that was the subject of processing. The detailed operation of this control unit 11 will be described later.

[0013] The storage unit 12 is a memory device or disk device, etc., and holds the program executed by the control unit 11. This storage unit 12 also operates as the work memory for the control unit 11.

[0014] The operation unit 13 includes a mouse, keyboard, etc., and accepts user input and outputs the content of the accepted operation to the control unit 11. The display unit 14 is a display, etc., and displays and outputs information according to instructions input from the control unit 11. The input / output unit 15 is a USB interface, network interface, etc., and communicates with external devices, server equipment, etc. In one example of this embodiment, the input / output unit 15 is connected to the camera C and receives input of multiple image data (for example, a moving image including a series of image data captured in succession) taken by the camera C while changing the line of sight to the subject, and outputs the image data to the control unit 11 as the target of processing.

[0015] Next, an example of the operation of the control unit 11 will be described. In this embodiment, the control unit 11 executes the program stored in the storage unit 12, thereby functionally realizing a configuration that includes a receiving unit 21, a primary detection unit 22, a secondary detection unit 23, and a processing execution unit 24, as illustrated in Figure 2.

[0016] The receiving unit 21 receives multiple image data to be processed via the input / output unit 15. In this example, each of these image data is assumed to be monocular image data without depth information (distance information), captured by a single camera in a smartphone or the like.

[0017] The primary detection unit 22 detects the range (imaging range) in which a predetermined object is imaged from each of the image data received by the receiving unit 21. Here, the primary detection unit 22 performs general object detection processing from image data. The objects detected in this way include, for example, if the image data to be processed is an image of a vehicle, the entire vehicle (hereinafter referred to as the overall object) as well as its various parts such as the body, tires, wheels, glass surface, and mirror parts (hereinafter referred to as component objects). Similarly, if the image data to be processed is an image of a landscape, the objects detected include not only the overall object such as the imaged building, but also its component objects such as its walls and glass surface.

[0018] For example, when performing this object detection process using a machine learning model such as YOLO (https: / / github.com / AlexeyAB / darknet), when training the model, the machine learning should include not only the overall image of a vehicle, but also images of parts of the vehicle that have substantially uniform reflectivity, such as door panels. This allows the temporary detection unit 22 to detect not only the overall object, such as the entire vehicle, but also the imaged portions of its components that have substantially uniform reflectivity (tires, wheels, glass surfaces, etc.) as imaged portions of the constituent object.

[0019] The primary detection unit 22 generates information on the area representing the imaging range of the detected object (each of the overall object and the component objects) (such as a mask image specifying a group of pixels included in the area), and associates information identifying the detected object (such as "entire vehicle", "tire", etc.) with each of the generated area information and stores it in the storage unit 12.

[0020] Also, the primary detection unit 22 acquires information on the distances from the position (camera position) where the image data to be processed was captured to each part on the above-mentioned detected overall object. As an example, the primary detection unit 22 determines the distance to a part (hereinafter referred to as the primary detection part) that is predetermined as a part with a relatively low reflectance among the points on the object imaged in the image data to be processed.

[0021] For example, the primary detection unit 22 refers to a reflectance database storing information on the reflectance for each detectable component object, and finds a component object whose reflectance is below a predetermined threshold among the component objects detected from the image to be processed. Then, the primary detection unit 22 uses the range in which the component object found here with a reflectance below the predetermined threshold is imaged as the primary detection part, and determines the distance to the primary detection part. However, this is just an example, and the part used as the primary detection part may include a predetermined part such as the boundary line part of a component object that is considered to have less reflection, even if it is not inside the component object, as long as it is a part that is set in advance as a part with a relatively low reflectance and where it is difficult for other objects to be reflected. Furthermore, the primary detection part may be set or modified by a user operation.

[0022] The calculation of this distance can adopt well-known methods, such as the methods used in technologies like SLAM (Simultaneous Localization and Mapping). As an example, the primary detection unit 22 extracts predetermined feature points in the above-mentioned primary detection part. Then, for the extracted feature points, the primary detection unit 22 uses well-known methods, such as a method for detecting the position change of the feature points between a plurality of input image data, to obtain distance information (point cloud information) from the camera position where the image data was captured to each feature point.

[0023] The point cloud information obtained by the processing of the primary detection unit 22 represents the distance between each point on the object (not the object reflected in it, but the object itself, for example, each of the above-mentioned feature points) and the camera position where one of the plurality of input image data (the image data for which the distance is calculated, hereinafter referred to as the distance calculation target image when distinction is necessary) was captured.

[0024] Based on the obtained point cloud information, the primary detection unit 22 estimates the region in the three-dimensional coordinates occupied by the entire object (hereinafter referred to as the three-dimensional region). This estimation can adopt well-known methods, such as the methods introduced in Shoichi Tsuchie, "Point Cloud Surface Tension Allowing Measurement Noise", Unisys Technology Review, Vol 87, Nov. 2005, p.88, etc. Also, the three-dimensional coordinate system here can be a camera coordinate system with the camera position where the processed image data was captured as the origin, the line-of-sight direction of the camera as the Z-axis, and within the plane with this Z-axis as the normal, the width direction of the processed image data as the X-axis (for example, the right direction is defined as the positive direction), and the height direction as the Y-axis (for example, the upward direction is defined as the positive direction).

[0025] As an example, when processing the image data of the vehicle illustrated in FIG. 3(a) as the processing target, the primary detection unit 22 uses the entire vehicle shown in FIG. 3(a) as the entire object T, and detects the tire part (Y), door panel (D), mirror (M), glass surface (G),... etc. that make up the vehicle as the constituent objects.

[0026] The primary detection unit 22 then uses feature points detected from areas such as the tire portion (Y), which is predetermined as having relatively low reflectivity, and the area between a pair of door panels (D), as well as the outer edge portion (B) of the door panel, to obtain point cloud information representing the distance between the camera position that captured the distance calculation target image, which is one of the image data, and the feature point, using the position where the feature point was captured in each of the continuously captured image data.

[0027] The primary detection unit 22 further estimates the three-dimensional shape of the entire vehicle (only the part visible from the camera is required) using the obtained point cloud information.

[0028] This allows the distance from the camera position to each part of the entire object to be estimated, and for each pixel constituting the distance calculation image, the distance (primary distance) to the object itself (not other objects reflected in the object) is estimated for at least the portion in which the object is captured.

[0029] The secondary detection unit 23 obtains distance information from the camera position where the distance calculation target image was captured to the feature point, using a method widely known in SLAM technology, such as a method for detecting changes in the position of the feature point among multiple input image data, for feature points located within the region of each component detected by the primary detection unit 22, that is, in the part of the overall object other than the primary detection portion.

[0030] The feature points detected by this secondary detection unit 23 may be feature points on the object itself, or feature points on other objects reflected in the object. When a distinction is necessary, the object whose actual form is captured in the image data to be processed will be called the primary object, and the other objects reflected as reflections on the surface of the primary object will be called secondary objects.

[0031] In the example of the vehicle illustrated in Fig. 3(a), since the reflectance of the area inside the door panel of the vehicle (the part excluding the primary detection part), the mirror (M), the glass surface (G), etc. is relatively high, there is a possibility that the secondary object is reflected. Therefore, the secondary detection unit 23 obtains information on the distance from the camera position where the distance calculation target image is captured to the feature point found from these parts.

[0032] Here, if a certain feature point P1 is found from the primary object T, the distance between the feature point P1 and the camera position where the distance calculation target image is captured is the distance r1 to the primary object T. On the other hand, if the feature point P2 is on the image of the secondary object reflected on the surface of the mirror M, this feature point P2 is the distance obtained by adding the distance d to the secondary object T2 imaged with the normal line of the surface of the mirror M as the symmetry axis to the distance r2 to the primary object (Fig. 3(b)).

[0033] The secondary detection unit 23 classifies each feature point into a feature point (referred to as a primary feature point) where the absolute value of the difference between the primary distance r obtained by the primary detection unit 22 for the pixel where the feature point is imaged in the distance calculation target image and the distance L from the secondary detection unit 23 to the feature point is less than a predetermined threshold value, a feature point where L > r and the absolute value of the above difference exceeds the above threshold value (referred to as a secondary feature point), and other feature points (feature points where L < r and the absolute value of the above difference exceeds the above threshold value, hereinafter referred to as foreground feature points). Here, the threshold value may be determined for each pixel, for example, as 0.8 times the corresponding primary distance r, or as a value common to the pixels, such as 0.8 times the average value of the primary distances r. Furthermore, it may be a predetermined value regardless of the primary distance.

[0034] The processing execution unit 24 performs predetermined processing based on the region information and distance information for each pixel of the distance calculation target image obtained by the operation of the primary detection unit 22 and the secondary detection unit 23. As an example, the processing execution unit 24 replaces the value of the pixel in which a feature point classified as a secondary feature point by the secondary detection unit 23 is captured, among the pixels of the distance calculation target image data to be processed, with the value of a surrounding pixel in which a feature point classified as a primary feature point is captured. If there are no feature points classified as primary feature points in the vicinity, the value of the pixel in which a feature point classified as a secondary feature point by the secondary detection unit 23 is captured may be replaced with a predetermined pixel value (for example, a background pixel value).

[0035] In this example of the embodiment, the processing execution unit 24 performs a removal process to remove the image of the secondary object, which is a three-dimensional object represented by the secondary feature points detected by the secondary detection unit 23, from the image data to be processed. As a result, the image of the secondary object that was reflected on the surface of the primary object is erased.

[0036] [Operation] One example of this embodiment has the above configuration and operates as follows. In one example of this embodiment, the image processing device 1 is a smartphone or the like, and the user captures image data (image data without depth information) when the user launches a camera application. The image processing device 1 holds the image data captured by the camera before and after the user operates the shutter as the object to be processed. The image processing device 1 also uses the image data at the time the user operates the shutter as the image to be used for distance calculation.

[0037] In this example, the image processing device 1 performs the operations illustrated in Figure 4, receiving the image data to be processed, including the image to be used for distance calculation (S1), and then performs object detection processing on each of the received image data (S2). In the following example, the operation of the image processing device 1 will be explained assuming that a user has used the image processing device 1 to take a photograph of a car outdoors.

[0038] As already mentioned in this example, the image processing device 1 detects not only the overall image of the vehicle, which is the overall object, but also individual parts of the vehicle, such as the tire section (Y), door panel (D), mirror (M), glass surface (G), etc., as constituent objects, as illustrated in Figure 3(a).

[0039] The image processing device 1 generates region information representing the imaging range in which each of the objects (including the whole object and its constituent objects) detected in step S2 is captured from the image data to be processed (S3). The image processing device 1 then associates specific information (such as "entire vehicle" or "tire") that identifies the detected object with each of the generated region information and stores it in the storage unit 12 (S4). At this point, the image processing device 1 identifies the whole object and the constituent objects that make up the whole object based on the inclusion relationship of the region information of each object.

[0040] For example, in the case of Figure 3(a), the imaging range of components such as the tire (Y), door panel (D), mirror (M), and glass surface (G) is included in the imaging range of the vehicle (T). Therefore, the image processing device 1 records the vehicle (T) as the overall object, and associates the tire (Y), door panel (D), mirror (M), and glass surface (G) with its components.

[0041] The image processing device 1 also refers to a reflectance database that stores reflectance information for each detectable object, obtains the reflectance corresponding to the specific information stored in the storage unit 12 in step S4, and finds an object among the objects detected in step S2 whose reflectance corresponding to the specific information identifying that object is below a predetermined threshold (S5: Identifying an object with relatively low reflectance).

[0042] The image processing device 1 determines the distance to the primary detection portion (S6) of the image data to be processed, which is the region in which the object identified in step S5 is captured.

[0043] In this example, the image processing device 1 extracts predetermined feature points in the primary detection section and detects the change in the position of the extracted feature points among multiple image data being processed, thereby obtaining distance information (point cloud information) from the camera position where the image data was captured to each feature point.

[0044] The image processing device 1 then estimates the area in three-dimensional coordinates (hereinafter referred to as the three-dimensional area) occupied by the entire object detected in step S2 from the obtained point cloud information (S7). Using the information of the three-dimensional area of ​​the entire object estimated in step S7, the image processing device 1 estimates the distance from the camera position to each part of the entire object, and calculates the primary distance to the object (not other objects reflected in the object, but the object itself) for at least the portion of each pixel in the distance calculation target image in which the object is captured (S8).

[0045] The image processing device 1 also obtains distance information from the camera position that captured the distance calculation target image to the feature point, using the feature point detected from the portion of the overall object imaging range other than the primary detection portion (S9).

[0046] As illustrated in Figure 3(b), if a feature point P1 is found on a primary object, the distance between that feature point P1 and the camera position that captured the image subject to distance calculation will be the distance r to the primary object. On the other hand, if a feature point P2 is on the image of a secondary object, which is an object reflected on the surface of the mirror M, the distance to this feature point P2 will be the distance r to the primary object plus the distance d to another object (secondary object) T that was captured with the normal to the surface of the mirror M as the axis of symmetry.

[0047] The image processing apparatus 1 classifies each of the feature points detected in step S9 into a primary feature point for which the absolute value of the difference between the primary distance r obtained in step S8 for the pixel in the distance calculation target image where the feature point is imaged and the distance L to the feature point obtained in step S9 is less than a predetermined threshold value, a secondary feature point for which L>r and the absolute value of the difference exceeds the threshold value, and other foreground feature points (feature points for which L<r and the absolute value of the difference exceeds the threshold value) (S10). Here, the threshold value is set in advance as a predetermined value.

[0048] As a result, for pixels corresponding to portions other than the primary detection portion in the imaging range of the entire object, among the pixels that have become feature points, the image processing apparatus 1 can determine whether the image captured in the pixel is an image based on the color of the primary object, an image in a state where the secondary object is reflected, or an image of a foreground on the camera position side of the primary object.

[0049] The image processing apparatus 1 sequentially selects pixels in the distance calculation target image as the target pixels and determines whether the target pixel is a pixel in which a feature point is imaged (S11). In step S11, if the target pixel is a pixel in which a feature point is imaged (S11: Yes), the image processing apparatus 1 checks whether the feature point imaged in the target pixel (referred to as the target feature point) is classified as a secondary feature point in the classification in step S10 (S12). In the determination in step S12, if the target feature point is a feature point classified as a secondary feature point (S12: Yes), the image processing apparatus 1 replaces the value of the target pixel in which the target feature point is imaged with the value of a pixel in the periphery thereof in which a feature point classified as a primary feature point is imaged to correct the pixel value. When there is no feature point classified as a primary feature point in the vicinity, the image processing apparatus 1 may replace the pixel value of the target pixel with a pixel value defined in advance as a background pixel value.

[0050] Furthermore, if the image processing device 1 determines in step S11 that the pixel of interest is not the pixel on which a feature point was captured (S11: No), and if it determines in step S12 that the feature point of interest is not classified as a secondary feature point (S12: No), the pixel value of the pixel of interest remains unchanged.

[0051] The image processing device 1 repeatedly executes steps S11 to S13 for each pixel of the image to be used for distance calculation.

[0052] As a result, the image processing device 1 removes the image of a secondary object that is reflected in the primary object captured in the distance calculation image. The image processing device 1 then stores the distance calculation image, after the image of the secondary object has been removed, in the storage unit 12 as captured image data.

[0053] Thus, according to the image processing device 1 of this embodiment, reflected images in the object being imaged can be reduced or removed. [Explanation of Symbols]

[0054] 1 Image processing device, 11 Control unit, 12 Storage unit, 13 Operation unit, 14 Display unit, 15 Input / Output unit, 21 Receiving unit, 22 Primary detection unit, 23 Secondary detection unit, 24 Processing execution unit.

Claims

1. A means of receiving image data to be processed. A primary detection means for detecting a predetermined object from the received image data, A secondary detection means that performs a three-dimensional object detection process within the imaging range of the detected object. And, Image processing means that uses the information relating to the three-dimensional object detected by the secondary detection means to perform predetermined processing on the image data that is the target of processing. Includes, The secondary detection means refers to a reflectance database storing reflectance information for each detectable object to obtain the surface reflectance of the object detected by the primary detection means, and uses the obtained surface reflectance to perform a three-dimensional object detection process within the imaging range of the object.

2. An image processing apparatus according to claim 1, The primary detection means is an image processing device that performs a three-dimensional object detection process to detect a predetermined three-dimensional object from the received image data.

3. An image processing apparatus according to claim 1 or 2, The secondary detection means is an image processing device that performs a three-dimensional object detection process to detect a predetermined three-dimensional object within the imaging range of the object detected by the primary detection means.

4. An image processing apparatus according to any one of claims 1 to 3, The image processing means is an image processing device that performs a removal process as a predetermined process, which removes the image of the three-dimensional object detected by the secondary detection means from the image data subject to the processing.

5. Computer A means of receiving image data to be processed. A primary detection means for detecting a predetermined object from the received image data, A secondary detection means that performs a three-dimensional object detection process within the imaging range of the detected object. And, Image processing means that uses the information relating to the three-dimensional object detected by the secondary detection means to perform predetermined processing on the image data that is the target of processing. To make it function as, A program that, when used as the secondary detection means, causes the computer to refer to a reflectance database storing reflectance information for each detectable object, to obtain the surface reflectance of the object detected by the primary detection means, and to use the obtained surface reflectance to perform a three-dimensional object detection process within the imaging range of the object.

Images