Image processing device, image processing method, and program

Abstract

The present invention recognizes which one of a plurality of designated region classes, which differ from each other and are related to an object shown in an acquired captured image, each pixel of the captured image belongs to. The present invention specifies, as a candidate region of interest, an unrecognized-class region that does not belong to any one of the plurality of different region classes in a captured image region indicating a predetermined region class among the plurality of region classes. The present invention determines whether or not the candidate region of interest is an intended region set in advance.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In order to recognize a specific object in an image, a recognition model generated by machine learning information such as the shape and color of the object is used to recognize the object. However, generating a recognition model by machine learning requires preparing many images that contain the object as correct answer data and subjecting the images to machine learning, which requires time and effort. Patent Documents 1 and 2 disclose a related technology for detecting obstacles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-224797 A [Patent Document 2] JP 2008-529183 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the image processing techniques described above, there is a demand for a technique for easily recognizing a desired object contained in an image.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an image processing device, an image processing method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0006] According to a first aspect of the present invention, an image processing device comprises an area recognition means for recognizing to which of a plurality of different area classes specified for an object appearing in an acquired captured image each pixel of a depth map image generated based on an acquired captured image belongs, a candidate area of ​​interest identification means for identifying a class-unrecognized area in an area of ​​the captured image that indicates a predetermined area class among the plurality of different area classes and that does not belong to any of the plurality of area classes as a candidate area of ​​interest, and a desired area determination means for determining whether the candidate area of ​​interest is a predetermined desired area.

[0007] According to a second aspect of the present invention, an image processing method recognizes which of a plurality of different area classes specified for an object appearing in an acquired captured image each pixel of a depth map image generated based on an acquired captured image belongs to, identifies a class-unrecognized area in an area of ​​the captured image that indicates a predetermined area class among the plurality of different area classes and does not belong to any of the plurality of area classes as a candidate area of ​​interest, and determines whether the candidate area of ​​interest is a predetermined desired area.

[0008] According to a third aspect of the present invention, the program causes a computer of an image processing device to function as an area recognition means for recognizing to which of a plurality of different area classes specified for an object appearing in the acquired captured image each pixel of a depth map image generated based on an acquired captured image belongs, a candidate area of ​​interest identification means for identifying a class-unrecognized area in the captured image that indicates a predetermined area class among the plurality of different area classes and does not belong to any of the plurality of area classes as a candidate area of ​​interest, and a desired area determination means for determining whether the candidate area of ​​interest is a predetermined desired area. Effect of the Invention

[0009] According to the present invention, it is possible to provide a technique for easily recognizing a desired object contained in an image. [Brief description of the drawings]

[0010] [Figure 1]1 is a diagram showing an overview of an image processing system according to an embodiment of the present invention. [Diagram 2] 1 is a hardware configuration diagram of an image processing apparatus according to an embodiment of the present invention. [Diagram 3] 1 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram showing an outline of processing performed by an image processing device according to an embodiment of the present invention. [Diagram 5] FIG. 4 is a diagram showing a processing flow of the image processing device according to the present embodiment. [Figure 6] FIG. 1 is a diagram showing a minimum configuration of an image processing device according to an embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing a processing flow of an image processing device with a minimum configuration according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] An image processing apparatus according to an embodiment of the present invention will now be described with reference to the drawings. FIG. 1 is a diagram showing an outline of an image processing system including an image processing device according to this embodiment. As shown in FIG. 1, the image processing system 100 is configured by connecting an image processing device 1 and a camera 2 mounted on a vehicle 20 via a wireless communication network or a wired communication network. The image processing system 100 may include a server device 3. The server device 3 may be communicatively connected to the image processing device 1 and the camera 2. In this embodiment, the camera 2 captures an image including a road and a vehicle traveling on the road. The camera 2 outputs the image to the image processing device 1. The image processing device 1 uses the image acquired from the camera 2 to identify a candidate region of interest that may include a specific object, and identifies a region of interest that is likely to show a specific object or the like from the candidate region of interest. In this embodiment, the image processing device 1 identifies an area in the image in which an obstacle lying on the road is captured as a region of interest. Identifying an obstacle as a region of interest from an image can be used for processing to avoid obstacles in autonomous driving or the like.

[0012] FIG. 2 is a diagram showing the hardware configuration of the image processing device. As shown in this figure, the image processing device 1 is a computer equipped with various hardware components such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a HDD (Hard Disk Drive) 104, a communication module 105, and a database 106. The server device 3 also has a similar configuration.

[0013] FIG. 3 is a functional block diagram of the image processing device. The image processing device 1 starts up when the power is turned on based on the start of the vehicle 20, and executes a pre-stored image processing program. As a result, the image processing device 1 exerts the functions of an image acquisition unit 11, a depth map generation unit 12, a region recognition unit 13, a boundary detection unit 14, a candidate region of interest extraction unit 15, a region of interest determination unit 16, an output unit 17, and a shape recognition unit 18.

[0014] The image acquisition unit 11 acquires an image from the camera 2 . The depth map generator 12 generates depth map information using the images acquired from the camera 2 . The area recognition unit 13 recognizes to which of a plurality of different area classes designated for the object appearing in the captured image each pixel of the acquired captured image belongs. The boundary detection unit 14 detects the boundaries of each area recognized during shooting. The candidate region of interest extraction unit 15 identifies, in a region in a captured image that indicates a predetermined region class among a plurality of different region classes, a class-unrecognized region that does not belong to any of the plurality of region classes as a candidate region of interest. The region of interest determination unit 16 determines whether the candidate region of interest is a desired region of interest. The output unit 17 outputs the region of interest. The shape recognition unit 18 detects the shape of the road based on the steering angle and other information obtained from the CAN information and the like.

[0015] Specifically, the area of ​​interest determination unit 16 determines whether the candidate area of ​​interest indicates a three-dimensional object based on the depth information indicated by each pixel in the candidate area of ​​interest, and when the candidate area of ​​interest indicates a three-dimensional object, it determines that the candidate area of ​​interest is an area of ​​interest.

[0016] The area of ​​interest determination unit 16 may determine whether the candidate area of ​​interest indicates a stationary object based on changes in depth information indicated by each pixel of the candidate area of ​​interest, and when the candidate area of ​​interest indicates a stationary object, may determine that the candidate area of ​​interest is an area of ​​interest.

[0017] The area recognition unit 13 may recognize to which of a plurality of different area classes designated for objects appearing in the captured image each pixel of a captured image including a road and a moving object such as a vehicle 20 traveling on the road belongs. At this time, the candidate area of ​​interest extraction unit 15 may specify, as a candidate area of ​​interest, an unclassified area at least part of which is adjacent to an area in the captured image belonging to a road class indicating a road area among the plurality of area classes.

[0018] The region of interest determination section 16 may determine the region of interest from among candidate regions of interest in regions in the captured image excluding regions belonging to moving objects among a plurality of region classes.

[0019] FIG. 4 is a diagram showing an outline of processing performed by the image processing device. 4 shows a captured image (40), a processing result (41) of area recognition unit 13 using the captured image, a processing result (42) of boundary detection unit 14 and candidate area of ​​interest extraction unit 15, and a processing result (43) of area of ​​interest determination unit 16. The processing result (42) of boundary detection unit 14 and candidate area of ​​interest extraction unit 15 includes information B indicating the boundary and pixel information indicating candidate area of ​​interest A1. Furthermore, the processing result (43) of area of ​​interest determination unit 16 includes pixel information indicating area of ​​interest A2.

[0020] FIG. 5 is a diagram showing a processing flow of the image processing device. The processing flow of the image processing device 1 will be explained step by step below. While the vehicle 20 is traveling, the camera 2 outputs the captured image generated by shooting to the image processing device 1. The image acquisition unit 11 of the image processing device 1 acquires the captured image and outputs it to the depth map generation unit 12. The depth map generation unit 12 generates depth map information based on the acquired captured image (step S101). Note that the image acquisition unit 11 may acquire the depth map information generated by the camera 2 in advance. In this case, the depth map generation unit 12 does not need to be provided in the image processing device 1. The depth map information is array information of the distance to the subject contained in each pixel of each captured image. The depth map information may be generated by a known technique. The depth map generation unit 12 outputs the depth map information to the boundary detection unit 14 and the area of ​​interest determination unit 16.

[0021] The area recognition unit 13 acquires a photographed image (40). The area recognition unit 13 recognizes the subject in the photographed image for each area class, such as sky, wall, road, moving object (traffic participant), person, etc. (step S102). The area recognition unit 13 may use a known technology for recognizing each pixel of the photographed image for each area class indicating a plurality of different objects. In this process, the area recognition unit 13 calculates the probability that each pixel of the photographed image belongs to each area class. The area recognition unit 13 generates area recognition information that holds information on the probability that each pixel belongs to each area class. The area recognition information is array information of probability information for each area class for each pixel of the photographed image. The area recognition unit 13 outputs the area recognition information, which is the processing result (41), to the boundary detection unit 14. The area recognition unit 13 may calculate the probability of each pixel of the photographed image for each area class using an area class calculation model that receives the photographed image as input and outputs the probability of belonging to a plurality of predetermined area classes. The area class calculation model may be, for example, a model in which a large number of images are input, information indicating the area class of each pixel of the images is used as correct answer data, and the relationship between them is machine-learned. The area recognition unit 13 may obtain the steering angle from the CAN information obtained from the shape recognition unit 18, estimate the road curvature according to the steering angle based on the information, and recognize the shape of the road from the estimation result or the like. For example, if the steering angle is 15 degrees to the left, the road is curving to the left. The area recognition unit 13 may recognize that the left rear corner is curving to the left in the depth map information, and may use the information indicating the curvature to the left along with the color and brightness in the image to multiply the probability of the pixels of the road class by a correction coefficient so that the center left side of the image is higher, thereby increasing the probability of the pixels of the road class.

[0022] The boundary detection unit 14 acquires area recognition information. The boundary detection unit 14 identifies pixels whose area classes are different from adjacent pixels based on the probability of the area class corresponding to each pixel of the captured image included in the area recognition information, and generates boundary information that holds a boundary flag for the pixel (step S103). The boundary information is an array of information indicating whether the area class of each pixel is a boundary with another area class. The boundary detection unit 14 outputs the depth map information, area recognition information, and boundary information to the candidate area of ​​interest extraction unit 15.

[0023] The candidate area of ​​interest extraction unit 15 acquires a photographed image, depth map information, area recognition information, and boundary information. Using the area recognition information, the candidate area of ​​interest extraction unit 15 identifies a class unrecognized area that does not belong to any of a plurality of area classes in an area in the photographed image that indicates a specified area class among a plurality of different area classes as a candidate area of ​​interest (step S104). For example, when the plurality of area classes are area classes such as sky, wall, road, moving body (traffic participant), and person, and the specified predetermined area class is a road class, the candidate area of ​​interest extraction unit 15 identifies an area of ​​pixels in which all the probabilities of the plurality of area classes are less than a predetermined probability value from the road class area, and identifies the area as a class unrecognized area. The candidate area of ​​interest extraction unit 15 identifies the class unrecognized area as a candidate area of ​​interest.

[0024] The candidate area of ​​interest extraction unit 15 may determine, as a candidate area of ​​interest, a class unrecognized area that is at least partially adjacent to an area in a captured image that belongs to a road class indicating a road area among a plurality of designated area classes, and identify the candidate area of ​​interest as a candidate area of ​​interest. In other words, a class unrecognized area that is at least partially adjacent to an area in a captured image that belongs to a road class indicating a road area among a plurality of designated area classes becomes an area that straddles the boundary between the road class and another class, and the candidate area of ​​interest extraction unit 15 may determine such an area as a class unrecognized area. After identifying the class unrecognized area as a candidate area of ​​interest, the candidate area of ​​interest extraction unit 15 generates candidate area of ​​interest information that indicates the arrangement information of pixels indicating the candidate area of ​​interest. The candidate area of ​​interest extraction unit 15 outputs the captured image, depth map information, area recognition information, boundary information, and candidate area of ​​interest information to the area of ​​interest determination unit 16.

[0025] The region of interest determination unit 16 acquires the captured image, depth map information, region recognition information, boundary information, and candidate region of interest information. The region of interest determination unit 16 identifies a region of interest from among the candidate regions of interest indicated by the candidate region of interest information (step S105). Specifically, the region of interest determination unit 16 acquires depth information indicated by each pixel of the candidate region of interest from the depth map information, and determines whether the candidate region of interest indicates a three-dimensional object based on the depth information. When the region of interest determination unit 16 determines that the candidate region of interest indicates a three-dimensional object, the region of interest determination unit 16 determines that the candidate region of interest is a region of interest. As an example, when the depth information of the pixels indicating the candidate region of interest is uniform in the vertical and horizontal directions, the region of interest determination unit 16 determines that the region of interest is a three-dimensional object because the regions form a surface, and determines that the candidate region of interest is a region of interest. The depth information is uniform in the vertical and horizontal directions when, for example, the error in the depth information of each of the other adjacent pixels based on a certain pixel is less than a predetermined value compared with the depth information of the reference pixel. Even if the depth information of a reference pixel and its adjacent pixels does not indicate a uniform surface based on the depth information of the reference pixel and its adjacent pixels, the attention area determination unit 16 may determine the area as the attention area if it determines that the area including those pixels is not a horizontal surface but an area indicating a three-dimensional object based on the depth information of the reference pixel and its adjacent pixels. For example, if the difference in the depth information between a certain reference pixel and the adjacent pixel is such that the difference indicates a continuous surface, the attention area determination unit 16 may determine that the area is the attention area as containing a three-dimensional object.

[0026] The region of interest determination unit 16 may determine whether the candidate region of interest indicates a stationary object based on the change in depth information indicated by each pixel of the candidate region of interest, and may determine the candidate region of interest as a region of interest when the candidate region of interest indicates a stationary object. For example, the region of interest determination unit 16 calculates the change in distance of the candidate region of interest during a predetermined period from the change in depth information of the region. The region of interest determination unit 16 also calculates the change in distance of the pixels of the region determined to be a moving object during the same period from the change in depth information of the region. The region of interest determination unit 16 determines that the candidate region of interest is a stationary object when the difference between the change in absolute value of the change in distance of the candidate region of interest during a predetermined period and the change in absolute value of the change in distance of the pixels of the region determined to be a moving object during the same period is equal to or greater than a predetermined change threshold. When the camera 2 of the moving object is used as a reference, the distance of other moving objects running at a similar speed to the moving object equipped with the camera 2 does not change significantly, and the distance of stationary objects changes significantly, so that the candidate region of interest can be determined to be a stationary object by such processing. When the attention candidate region is determined to be a three-dimensional object and a stationary object, the attention candidate region determination unit 16 may determine the attention candidate region as the attention region. Even when the attention candidate region is not determined to be a three-dimensional object, the attention candidate region determination unit 16 may determine the attention candidate region as the attention region when it is determined to be a stationary object.

[0027] The region of interest determination unit 16 may determine, as a region of interest, a candidate region of interest in a region in the captured image excluding regions that belong to a class including a moving object. A region that belongs to a class including a moving object is a region in which the probability of the class being included in the value in the region recognition information corresponding to each pixel of the captured image is equal to or higher than a predetermined probability. The region of interest determination unit 16 may determine, as a region of interest, from among the candidate regions of interest excluding regions that are recognized as regions of a class including such a moving object.

[0028] The region of interest determination unit 16 may obtain the steering angle from the CAN information obtained from the shape recognition unit 18, estimate the road curvature according to the steering angle based on the information, recognize the road shape based on the estimation result, etc., and correct the recognition result of the region recognition unit 13. For example, if the steering angle is 15 degrees to the left, the road is curving to the left. The region recognition unit 13 may recognize that the left rear part is curving to the left in the depth map information, and use the information indicating the curvature to the left along with the color and brightness in the image to multiply the probability of the pixels of the road class by a correction coefficient so that the center left side of the image is higher, thereby increasing the probability of the pixels of the road class. The region of interest determination unit 16 may determine the region of interest from the candidate regions of interest using the region recognition information obtained by correcting the recognition result of the region recognition unit 13.

[0029] Even if the region of interest determination unit 16 determines a region of interest from among the candidate regions of interest, it is not necessary to immediately output the region as a region of interest. For example, the region of interest determination unit 16 may use a plurality of captured images to determine that a region of interest with little deviation, such as a position deviation of less than a threshold value, is the same region of interest, and if the same region of interest is included in all captured images, the region of interest may be determined to be a region of interest. For example, if the same region of interest is included in a predetermined number of consecutive captured images, it may be determined to output the region of interest. The region of interest determination unit 16 may determine that a candidate region of interest is a region of interest if its size is equal to or larger than a predetermined size. After identifying the region of interest, the region of interest determination unit 16 generates region of interest information indicating array information of pixels indicating the region of interest. The region of interest determination unit 16 outputs the region of interest information to the output unit 17.

[0030] The output unit 17 outputs the captured image and the area of ​​interest information to a predetermined device (step S106). For example, if the area of ​​interest does not belong to any of a plurality of area classes designated in advance, it is highly likely to be an obstacle. Therefore, if the output destination is, for example, an automatic driving processing device, it can recognize the obstacle and control the stopping operation of the vehicle 20.

[0031] According to the above process, it is possible to determine a region of interest that indicates an object of interest, such as an obstacle, captured in an image generated by the camera 2 without learning the object of interest. This provides a technology for easily recognizing a desired object included in an image.

[0032] According to the above-mentioned process, the image processing device 1 determines the region of interest. However, the image processing device 1 may transmit the image acquired from the camera 2 to the server device 3, and the server device 3 may determine the region of interest from the acquired image in the same manner as the above-mentioned process. Alternatively, the server device 3 may perform only a part of the above-mentioned process, transmit the result to the image processing device 1, and the image processing device 1 may perform the subsequent process in the same manner as the above-mentioned process to determine the region of interest.

[0033] In the above process, a process for determining an image area such as an obstacle in a captured image of the area in front of the traveling vehicle 20 captured by the camera 2 provided on the vehicle 20 has been specifically described. However, the image processing device 1 may be a device that determines an image area such as an obstacle in a captured image of the area in front of a moving object flying in the air such as an aircraft captured by the camera 2 provided on the moving object. The image processing device 1 may also be a device that similarly determines a desired area of ​​interest in a desired image.

[0034] FIG. 6 is a diagram showing the minimum configuration of an image processing device. FIG. 7 is a diagram showing a processing flow of an image processing device with a minimum configuration. The image processing device 1 includes at least a candidate region of interest specification means 61 and a region of interest determination means 62 . The target candidate region identifying means 61 identifies a class unrecognized region that does not belong to any of a plurality of region classes in a region in a captured image that indicates a predetermined region class among a plurality of different region classes as a target candidate region (step S701). The region of interest determination means 62 determines whether the candidate region of interest is a desired region of interest (step S702).

[0035] Each of the above-mentioned devices has a computer system inside. The steps of each of the above-mentioned processes are stored in a computer-readable recording medium in the form of a program, and the above processes are performed by the computer reading and executing this program. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Also, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0036] The program may be a program for implementing some of the above-mentioned functions, or may be a so-called differential file (differential program) that can implement the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0037] 1. Image processing device 2. Camera 3. Server equipment 11. Image acquisition section 12. Depth map generator 13...Area recognition section 14. Boundary detection section 15: Attention candidate region extraction unit (attention candidate region identification means) 16: Attention area determination unit (attention area determination means) 17 Output section 18...Shape grasping part

Claims

1. an area recognition means for recognizing to which of a plurality of different area classes related to an object appearing in the captured image each pixel of the acquired captured image belongs; a candidate region of interest specification means for specifying a class unrecognized region that does not belong to any of the plurality of different region classes in a region in the captured image that indicates a predetermined region class among the plurality of different region classes as a candidate region of interest; an area of ​​interest determination means for determining whether the candidate area of ​​interest indicates a three-dimensional object based on depth information indicated by each pixel of the candidate area of ​​interest, determining whether the candidate area of ​​interest indicates a stationary object based on a change in the depth information indicated by each pixel of the candidate area of ​​interest, and determining that the candidate area of ​​interest indicates a three-dimensional object and a stationary object when the candidate area of ​​interest indicates a three-dimensional object and a stationary object; An image processing device comprising:

2. The area recognition means estimates a curvature of a road according to a steering angle, and recognizes a shape of the road based on the estimation result. The image processing device according to claim 1 .

3. the area recognition means recognizes which of a plurality of different area classes related to objects shown in the photographed image each pixel of the photographed image, which includes a road and a moving object traveling on the road, belongs to; The target candidate area specifying means specifies, as the target candidate area, the class unrecognized area at least partly adjacent to an area in the captured image that belongs to a road class indicating a road area among the plurality of area classes.

3. The image processing device according to claim 1 or 2.

4. The region of interest determination means determines the region of interest from among the candidate regions of interest in a region in the captured image excluding regions that belong to moving bodies among the plurality of region classes. The image processing device according to any one of claims 1 to 3.

5. Recognizing which of a plurality of different area classes related to the object shown in the captured image each pixel of the captured image belongs to; Identifying a class unrecognized area that does not belong to any of the plurality of area classes in an area in the captured image that indicates a predetermined area class among the plurality of different area classes as a candidate area of ​​interest; A determination is made as to whether the candidate region of interest indicates a three-dimensional object based on depth information indicated by each pixel of the candidate region of interest, and a determination is made as to whether the candidate region of interest indicates a stationary object based on a change in the depth information indicated by each pixel of the candidate region of interest, and when the candidate region of interest indicates both a three-dimensional object and a stationary object, the candidate region of interest is determined to be a predetermined region of interest. Image processing methods.

6. The computer of the image processing device an area recognition means for recognizing to which of a plurality of different area classes related to an object appearing in the captured image each pixel of the acquired captured image belongs; a candidate region of interest specification means for specifying, in a region in the captured image that indicates a predetermined region class among the plurality of different region classes, a class unrecognized region that does not belong to any of the plurality of region classes as a candidate region of interest; a region of interest determination means for determining whether the candidate region of interest indicates a three-dimensional object based on depth information indicated by each pixel of the candidate region of interest, determining whether the candidate region of interest indicates a stationary object based on a change in the depth information indicated by each pixel of the candidate region of interest, and determining that the candidate region of interest indicates a three-dimensional object and a stationary object when the candidate region of interest indicates a three-dimensional object and a stationary object, as a predetermined region of interest; A program that functions as a

Images