Image processing apparatus, program and method thereof

Abstract

An image processing device according to an embodiment comprises: a two-dimensional data acquisition unit (13) that acquires an image that is two-dimensional data; a three-dimensional data acquisition unit (11) that acquires three-dimensional data of at least a portion of the area captured as the image; a to-be-extracted area setting unit (12) that outputs, as to-be-extracted area coordinates, two-dimensional coordinates of an area including point cloud data for which distance information of the three-dimensional data is within a preset recognition interval; and an object image extraction unit (14) that extracts, from the image, the image of the area corresponding to the to-be-extracted area coordinates, as an object image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, its program Mu and and a method, and particularly relates to an image processing apparatus, its program and a method for cutting out an object image including a predetermined object from a photographed image.

Background Art

[0002] In recent years, technologies for recognizing specific objects captured in images obtained by cameras have been required in many fields. Therefore, as an example of object recognition in images, Patent Document 1 discloses one case.

[0003] The image analysis apparatus described in Patent Document 1 includes a distance information analysis unit that detects object information including the position and size of an object from measurement data obtained from a lidar, an image analysis unit that detects object information from image data obtained from a camera, a shooting condition acquisition unit that acquires shooting conditions in the first and second shooting areas of the lidar and the camera respectively, and an information integration unit that performs an integration process of detecting results of the distance information analysis unit and the image analysis unit in a common area where the first and second shooting areas overlap based on the acquired shooting conditions to generate new object information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when a distant object and a nearby object are captured in an image, there is an overlap between the nearby object and the distant object, so there is a problem that it is difficult to detect a distant object based on the two-dimensional data image.

Means for Solving the Problems

[0006] An image processing apparatus according to one embodiment includes: a two-dimensional data acquisition unit that acquires an image which is two-dimensional data; a three-dimensional data acquisition unit that acquires three-dimensional data for at least a portion of the area to be captured as the image; an extraction target area setting unit that outputs the two-dimensional coordinates of an area containing point cloud data whose distance information of the three-dimensional data falls within a preset recognition interval as extraction target area coordinates; and an object image extraction unit that extracts an image of an area corresponding to the extraction target area coordinates from the image as an object image.

[0007] Image processing program according to one embodiment Mu is The calculation unit is made to execute the following: a two-dimensional data acquisition process that acquires an image which is two-dimensional data acquired by a two-dimensional data acquisition unit; a three-dimensional data acquisition process that acquires three-dimensional data output by a three-dimensional data acquisition unit for at least a portion of the area to be captured as the image; an extraction target area setting process that outputs the two-dimensional coordinates of the area containing point cloud data whose distance information of the three-dimensional data falls within a pre-set recognition interval as extraction target area coordinates; and an object image extraction process that extracts an image of the area corresponding to the extraction target area coordinates from the image as an object image.

[0008] An image processing method according to one embodiment involves causing a calculation unit to execute the following: a two-dimensional data acquisition process for acquiring an image which is two-dimensional data acquired by a two-dimensional data acquisition unit; a three-dimensional data acquisition process for acquiring three-dimensional data output by a three-dimensional data acquisition unit for at least a portion of the area to be captured as the image; an extraction target area setting process for outputting the two-dimensional coordinates of an area containing point cloud data whose distance information of the three-dimensional data falls within a preset recognition interval as extraction target area coordinates; and an object image extraction process for extracting an image of an area corresponding to the extraction target area coordinates from the image as an object image. [Effects of the Invention]

[0009] Image processing apparatus according to one embodiment, and its program Mu andAccording to this method, distant objects can be detected based on images, which are two-dimensional data. [Brief explanation of the drawing]

[0010] [Figure 1] This figure illustrates an object detected by the image processing device according to Embodiment 1. [Figure 2] This is a block diagram of the image processing device according to Embodiment 1. [Figure 3] This is a hardware configuration diagram of the image processing device according to Embodiment 1. [Figure 4] This is a flowchart illustrating the operation of the image processing apparatus according to Embodiment 1. [Figure 5] This is a block diagram of an image processing device according to Embodiment 2. [Figure 6] This is a flowchart illustrating the operation of the image processing apparatus according to Embodiment 2. [Figure 7] This is a block diagram of the image processing device according to Embodiment 3. [Figure 8] This is a flowchart illustrating the operation of the image processing apparatus according to Embodiment 3. [Modes for carrying out the invention]

[0011] For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. Furthermore, each element shown in the drawings as a functional block performing various processes can be composed of a CPU (Central Processing Unit), memory, and other circuits in hardware terms, and implemented in software terms by programs loaded into memory. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof, and are not limited to any one of these. In each drawing, the same elements are denoted by the same reference numeral, and redundant explanations have been omitted where necessary.

[0012] Furthermore, the programs described above can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0013] Embodiment 1 First, we will describe images in which object detection accuracy is improved by using the image processing device described in the embodiment. Therefore, Figure 1 shows a diagram illustrating an object detected by the image processing device according to Embodiment 1. The example shown in Figure 1 is an image taken of the scenery visible in the direction of travel of a train. In train operation, special signal lights (special lights in Figure 1) that can be seen from more than 800m away from the train driver are used. These special signal lights are installed at locations where caution is required regarding falling rocks, avalanches, strong winds, level crossings, etc. Such special signal lights are to be visually confirmed by the driver, but they are normally off when there is no abnormality, and it is difficult to react immediately due to their special nature of emitting a stop signal at an unexpected time. For this reason, there is a need to detect signals and obstacles installed at a distance and warn the driver, but as shown in Figure 1, the special signal lights to be detected are hidden by support poles for suspending overhead wires, making it difficult to detect them from a single image.

[0014] Therefore, in the image processing apparatus described below, three-dimensional data for measuring distance information such as LiDAR (Light Detection and Ranging) is utilized to generate an object image in which only the imaging range at a distance within a preset recognition interval is cut out. By using such an object image, it is possible to exclude objects that hinder the recognition of the detection target in the perspective direction. Therefore, the image processing apparatus described below can improve the object recognition accuracy. Object recognition using such an object image is a process required not only for railways but also for all movable bodies such as automobiles and drones that can move.

[0015] Fig. 2 shows a block diagram of the image processing apparatus 1 according to Embodiment 1. As shown in Fig. 1, the image processing apparatus 1 according to Embodiment 1 includes a three-dimensional data acquisition unit 11, an extraction target area setting unit 12, a two-dimensional data acquisition unit 13, and an object image extraction unit 14.

[0016] The three-dimensional data acquisition unit 11 acquires three-dimensional data for at least a partial area of the range imaged as the image acquired by the two-dimensional data acquisition unit 13. The three-dimensional data acquisition unit 11 outputs, for example, point cloud data that is a set of measurement points whose values change according to the magnitude of the distance such as LiDAR.

[0017] The extraction target area setting unit 12 outputs the two-dimensional coordinates of the area including the point cloud data whose distance information in the three-dimensional data is within a preset recognition interval as the extraction target area coordinates. That is, these extraction target area coordinates are the two-dimensional coordinates of the part where the object exists within the range of the recognition interval.

[0018] The two-dimensional data acquisition unit 13 acquires an image that is two-dimensional data. Here, the two-dimensional data acquisition unit 13 is, for example, a device that outputs the imaging range of an optical camera, an infrared camera, etc. as two-dimensional image information.

[0019] The object image extraction unit 14 extracts an image of the region corresponding to the extraction target region coordinates from the image as an object image. Here, in the image processing device 1, the two-dimensional coordinates of the shooting range of the extraction target region setting unit 12 and the two-dimensional coordinates of the point cloud data of the three-dimensional data acquisition unit 11 are assumed to be calibrated in advance to match. Furthermore, since the number of pixels in the point cloud data is less than the number of pixels in the image output by the two-dimensional data acquisition unit 13, it is preferable for the object image extraction unit 14 to take this difference in pixel count into account when extracting an object image from the image and extract an image of a slightly wider range than the range specified by the extraction target region coordinates as the object image.

[0020] The image processing device 1 according to Embodiment 1 can be configured as dedicated hardware, but it can also be realized by running an image processing program on a computer. Therefore, Figure 3 shows a hardware configuration diagram of the image processing device according to Embodiment 1. In Figure 3, a computer 100 is shown as part of the hardware configuration of the image processing device 1.

[0021] Computer 100 includes an arithmetic unit 101, a memory 102, a three-dimensional data acquisition unit 103, Two-dimensional data acquisition unit 104 It has the following features. In the computer 100, the arithmetic unit 101, memory 102, three-dimensional data acquisition unit 103, and two-dimensional data acquisition unit 104 are configured to communicate with each other via a bus.

[0022] In Figure 3, the three-dimensional data acquisition unit 103 and the two-dimensional data acquisition unit 104 are physical hardware such as sensors, and the image data and three-dimensional data are stored in the memory 102 via a bus. The calculation unit 101 executes an image processing program and outputs the generated object image to the memory 102. The memory 102 is a storage device that stores data handled by the computer, such as volatile memory like DRAM or non-volatile memory like flash memory.

[0023] The image processing program has the following processes performed by the calculation unit 101: two-dimensional data acquisition processing, three-dimensional data acquisition processing, extraction target area setting processing, and object image extraction processing. Two-dimensional data acquisition processing is performed by the two-dimensional data acquisition unit 13, and stores the two-dimensional image data acquired by the two-dimensional data acquisition unit 104 into memory 102. Three-dimensional data acquisition processing is performed by the three-dimensional data acquisition unit 11, and stores the three-dimensional data output by the three-dimensional data acquisition unit 103 for at least a portion of the area captured as an image acquired by the two-dimensional data acquisition unit 104 into memory 102. Extraction target area setting processing is performed by the extraction target area setting unit 12, and outputs the two-dimensional coordinates of the area containing point cloud data where the distance information of the three-dimensional data falls within a pre-set recognition interval as the extraction target area coordinates. Object image extraction processing is performed by the object image extraction unit 14, and extracts the image of the area corresponding to the extraction target area coordinates from the image as an object image. The calculation unit 101 may also acquire data directly from the three-dimensional data acquisition unit 103 and the two-dimensional data acquisition unit 104 without going through the memory 102.

[0024] Here, we will explain the operation of the image processing apparatus 1 according to Embodiment 1. Figure 4 shows a flowchart illustrating the operation of the image processing apparatus according to Embodiment 1.

[0025] As shown in Figure 4, when the image processing device 1 starts operation, the three-dimensional data acquisition unit 11 acquires three-dimensional data (step S11). The image processing device 1 then uses the extraction target area setting unit 12 to set the two-dimensional coordinates of point cloud data obtained from objects at a predetermined distance from the shooting position as the extraction target area coordinates (step S12). In parallel with the processing in steps S11 and S12, the image processing device 1 acquires two-dimensional data (for example, an image) using the two-dimensional data acquisition unit 13 (step S13). Subsequently, the image processing device 1 determines the position of the object to be detected in the two-dimensional data based on the extraction target area coordinates and extracts an object image containing the object to be detected from the two-dimensional data (step S14). The object image extraction unit 14 then outputs the object image, completing the output operation of one object image (step S15). The image processing device 1 repeatedly executes the operations of steps S11 to S15 at a predetermined cycle.

[0026] As described above, the image processing device 1 according to Embodiment 1 outputs an object image that includes only images in which objects within the recognition interval are captured. This reduces the amount of information processing required when recognizing the target object from the object image output by the image processing device 1. Furthermore, detection from distance information subject By performing object recognition using an image containing only objects that are within the expected distance, it becomes possible to improve recognition accuracy.

[0027] Embodiment 2 Embodiment 2 describes an image processing device 2, which is another form of the image processing device 1 according to Embodiment 1. Note that components that are the same as those described in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1, and their descriptions are omitted.

[0028] Figure 5 shows a block diagram of the image processing device 2 according to Embodiment 2. As shown in Figure 5, the image processing device 2 according to Embodiment 2 is an image processing device 1 according to Embodiment 1 with the addition of an object recognition unit 21 and a notification unit 22. The object recognition unit 21 recognizes objects included in an object image. The object recognition unit 21 may also recognize objects by taking into account the coordinates of the extraction target area when recognizing an object. The object recognition unit 21 can, for example, recognize an object by comparing it with a pre-registered list of detection target candidates, or it can perform recognition using artificial intelligence. The notification unit 22 notifies information about the object recognized by the object recognition unit 21.

[0029] The object recognition unit 21 recognizes, for example, at least one of the following as an object: a signal placed within the shooting range of the two-dimensional data acquisition unit (e.g., a special signal light), a stone, a fallen tree, and a person located in a restricted area set within the shooting range of the two-dimensional data acquisition unit. When a special signal light is the object to be recognized, the unit also recognizes the light pattern from the image. The notification unit 22 notifies the person or equipment that will receive the notification, such as a train driver, of the recognition results based on the object or light pattern recognized by the object recognition unit 21. The equipment that will receive the notification could be various things, such as a railway operation control system or vehicle brakes.

[0030] Furthermore, the operation of the object recognition unit 21 and the notification unit 22 can also be realized by the image processing program executed by the calculation unit 101 shown in Figure 3.

[0031] Next, the operation of the image processing device 2 according to Embodiment 2 will be described. Figure 6 shows a flowchart illustrating the operation of the image processing device 2 according to Embodiment 2. As shown in Figure 6, in the operation of the image processing device 2 according to Embodiment 2, the object image output in step S15 of the operation of the image processing device 1 according to Embodiment 1 shown in Figure 4 is used to perform the processing in steps S21 to S26.

[0032] In step S21, the object recognition unit 21 recognizes an object using the object image output in step S15. If the recognized object is a traffic light (YES branch of step S22), the object recognition unit 21 recognizes the light emission pattern from the image corresponding to the traffic light and notifies the driver of the recognition result of the light emission pattern using the notification unit 22 (steps S22-S24). On the other hand, if the recognized object is not a traffic light (NO branch of step S22), the object recognition unit 21 issues a warning to the driver if it is a foreign object that should warrant a warning (YES branch of step S25, step S26), and terminates the operation without issuing a warning if it does not require a warning (NO branch of step S25).

[0033] As described above, according to the image processing device 2 of Embodiment 2, the object image extraction unit 14 outputs an object image that can be used to notify the driver of specific recognition results or to issue warnings. Furthermore, in the image processing device 2 of Embodiment 2, the object recognition unit 21 can perform recognition processing with less computation by utilizing the object image output by the object image extraction unit 14.

[0034] Embodiment 3 Embodiment 3 describes an image processing apparatus 3 that is another form of the image processing apparatus 2 according to Embodiment 2. Note that components that are the same as those described in Embodiments 1 and 2 are denoted by the same reference numerals as in Embodiments 1 and 2, and their descriptions are omitted.

[0035] Figure 7 shows a block diagram of the image processing device 3 according to Embodiment 3. As shown in Figure 7, the image processing device 3 according to Embodiment 3 is an image processing device 2 according to Embodiment 2 with the addition of a self-position estimation unit 31 and a scan direction specification unit 32, and the object recognition unit 21 replaced with an object recognition unit 33.

[0036] The self-position estimation unit 31 estimates the current position of the device and outputs self-position estimation information. This self-position estimation unit 31 outputs the current position of the vehicle equipped with the image processing device 3 as self-position estimation information, for example, by using position information acquired using a device such as GPS.

[0037] The scan direction specification unit 32 specifies the direction for acquiring two-dimensional data and the direction for acquiring three-dimensional data. More specifically, the scan direction specification unit 32 grasps the current geographical position of the device from the self-position estimation information and gives direction control instructions to the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13 so that they face the shooting direction associated with the grasped position. For this reason, the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13 are equipped with a mechanism that allows them to change direction. In addition, if the direction of the object to be detected can be estimated from the recognition result of the object recognition unit 33, the scan direction specification unit 32 uses that result to specify the direction for acquiring two-dimensional data and the direction for acquiring three-dimensional data. In other words, in Embodiment 3, the efficiency of scanning the object to be detected is increased by feeding back the recognition result of the object recognition unit 33 to the scan direction specification unit 32.

[0038] The object recognition unit 33 switches between candidate lists of objects to be recognized according to the self-localization information. The image processing device 3 is mounted on a moving object, and it is conceivable that the type of object to be recognized will differ depending on the geographical location of the moving object. Therefore, the object recognition unit 33 can shorten the processing time by switching between candidate lists of objects to be recognized based on the self-localization information.

[0039] Here, a flowchart illustrating the operation of the image processing device 3 according to Embodiment 3 is shown in Figure 8. As shown in Figure 8, the image processing device 3 according to Embodiment 3 performs the processes of steps S31 and S32 before steps S11 and S13 of the image processing device 2 according to Embodiment 2 shown in Figure 6, and performs the process of step S33 instead of step S14. In addition, it performs the process of step S34 after the process of step S25.

[0040] In the third embodiment, when the image processing device 3 starts operation, it first performs a self-position estimation process in which the self-position estimation unit 31 outputs self-position estimation information (step S31). Then, using the self-position estimation information, the image processing device 3 has the scan direction specification unit 32 specify the scan direction of the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13 (step S32). After that, the image processing device 3 performs the processing from step S11 onwards.

[0041] Furthermore, in the image processing apparatus 3 according to Embodiment 3, the processing in step S14 is replaced with step S33. In step S33, the object recognition unit 33 determines the position of the object to be detected in the two-dimensional data based on the self-position estimation information and the coordinates of the extraction target area, and extracts an object image including the object to be detected from the two-dimensional data.

[0042] Furthermore, in the image processing apparatus 3 according to Embodiment 3, after it is determined in step S25 that no foreign matter is present, it is further determined whether or not there are any non-foreign objects that could be candidates for scanning (step S34). If it is determined in step S34 that an object to be scanned exists, its position in the image is fed back to the scan direction specification unit 32 (the NO branch of step S24). On the other hand, if it is determined in step S34 that no object to be scanned exists, the process is terminated.

[0043] As described above, in the image processing apparatus 3 according to Embodiment 3, by changing the scanning direction of the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13 according to the position, the detection accuracy of the object to be detected can be improved even if the shooting angle of the three-dimensional data acquisition unit 11 and the extraction target area setting unit 12 is narrow. Furthermore, in the image processing apparatus 3 according to Embodiment 3, by changing the scanning direction of the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13, the probability of the object to be detected being hidden by other objects can be reduced.

[0044] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. [Explanation of Symbols]

[0045] 1-3 Image Processing Devices 11. Three-dimensional data acquisition unit 12 Extraction Target Area Setting Unit 13 Two-dimensional data acquisition unit 14. Object Image Extraction Unit 21 Object recognition section 22 Notification Department 31 Self-position estimation part 32 Scan direction specification section 33 Object recognition section 100 Computers 101 Arithmetic section 102 memory 103 Three-dimensional data acquisition unit 104 Two-dimensional data acquisition unit

Claims

1. A two-dimensional data acquisition unit that acquires images, which are two-dimensional data, A three-dimensional data acquisition unit acquires three-dimensional data for at least a portion of the area captured as the aforementioned image, An extraction target area setting unit outputs the two-dimensional coordinates of a region containing point cloud data whose distance information for the three-dimensional data falls within a pre-set recognition interval as the extraction target area coordinates. An object image extraction unit extracts an image of the region corresponding to the target region coordinates from the aforementioned image as an object image, It has an object recognition unit that recognizes objects included in the object image, The object recognition unit is an image processing device that recognizes an object by taking into account the coordinates of the extraction target area.

2. The image processing apparatus according to claim 1, further comprising a notification unit for notifying information relating to an object recognized by the object recognition unit.

3. It has a self-position estimation unit that estimates the current position of the device and outputs self-position estimation information. The image processing apparatus according to claim 1, wherein the object recognition unit switches a candidate list containing the object to be recognized in accordance with the self-position estimation information.

4. The image processing apparatus according to claim 1, wherein the object includes a signal placed within the shooting range of the two-dimensional data acquisition unit, and at least one of a stone, a fallen tree, and a person located in a restricted area set with respect to the shooting range of the two-dimensional data acquisition unit.

5. A self-position estimation unit that estimates the current position of the device and outputs self-position estimation information, It has a scan direction specification unit that specifies the acquisition direction of the two-dimensional data and the acquisition direction of the three-dimensional data, The image processing apparatus according to claim 1, wherein the two-dimensional data acquisition unit and the three-dimensional data acquisition unit acquire data in the direction specified by the scan direction specification unit.

6. The image processing apparatus according to claim 1, wherein the three-dimensional data acquisition unit outputs point cloud data, which is a set of measurement points whose value changes according to the magnitude of the distance.

7. A two-dimensional data acquisition process that acquires an image, which is two-dimensional data, obtained by the two-dimensional data acquisition unit, A three-dimensional data acquisition process that acquires three-dimensional data output by a three-dimensional data acquisition unit for at least a portion of the area captured as the aforementioned image, A process for setting the extraction target area, which outputs the two-dimensional coordinates of the region containing point cloud data whose distance information for the three-dimensional data falls within a pre-set recognition interval as the extraction target area coordinates, An object image extraction process that extracts an image of the region corresponding to the target region coordinates from the aforementioned image as an object image, The arithmetic unit is made to perform object recognition processing to recognize objects included in the aforementioned object image, An image processing program that recognizes an object in the object recognition process by taking into account the coordinates of the extraction target area.

8. A two-dimensional data acquisition process that acquires an image, which is two-dimensional data, obtained by the two-dimensional data acquisition unit, A three-dimensional data acquisition process that acquires three-dimensional data output by a three-dimensional data acquisition unit for at least a portion of the area captured as the aforementioned image, A process for setting the extraction target area, which outputs the two-dimensional coordinates of the region containing point cloud data whose distance information for the three-dimensional data falls within a pre-set recognition interval as the extraction target area coordinates, An object image extraction process that extracts an image of the region corresponding to the target region coordinates from the aforementioned image as an object image, The arithmetic unit is made to perform object recognition processing to recognize objects included in the aforementioned object image, An image processing method for recognizing an object in the object recognition process, taking into account the coordinates of the extraction target area.

Images