Image processing device, program and method thereof

Abstract

An image processing device according to an embodiment comprises: a two-dimensional data acquisition unit (13) that acquires an image that is two-dimensional data; a three-dimensional data acquisition unit (11) that acquires three-dimensional data of at least a portion of the area captured as the image; a to-be-extracted area setting unit (12) that outputs, as to-be-extracted area coordinates, two-dimensional coordinates of an area including point cloud data for which distance information of the three-dimensional data is within a preset recognition interval; and an object image extraction unit (14) that extracts, from the image, the image of the area corresponding to the to-be-extracted area coordinates, as an object image.

Description

Image processing device, non-transitory computer-readable medium having its program recorded thereon, and method

[0001] The present invention relates to an image processing device, a non-transitory computer-readable medium having a program recorded thereon, and a method, and more particularly to an image processing device, a program, and a method for extracting an object image including a predetermined object from a captured image.

[0002] In recent years, there has been a demand in many fields for technology that can recognize specific objects captured in images captured by a camera. Patent Document 1 discloses an example of object recognition in an image.

[0003] The image analysis device described in Patent Document 1 includes a distance information analysis unit that detects object information including the position and size of the object from measurement data acquired from the LIDAR; an image analysis unit that detects object information from image data acquired from the camera; an imaging condition acquisition unit that acquires imaging conditions in the first and second imaging areas of the LIDAR and the camera, respectively; and an information integration unit that performs an integration process to integrate the detection results of the distance information analysis unit and the detection results of the image analysis unit in the common area where the first and second imaging areas overlap based on the acquired imaging conditions, and generates new object information.

[0004] Japanese Patent Application Laid-Open No. 2022-17619

[0005] However, when a distant object and a nearby object are reflected in an image, there is an overlap between the object in the foreground and the distant object, which makes it difficult to detect the distant object based on the image, which is two-dimensional data.

[0006] An image processing device according to one embodiment includes a two-dimensional data acquisition unit that acquires an image as two-dimensional data, a three-dimensional data acquisition unit that acquires three-dimensional data for at least a portion of the area captured as the image, an extraction target area setting unit that outputs, as extraction target area coordinates, the two-dimensional coordinates of an area including point cloud data in which the distance information of the three-dimensional data falls within a predetermined recognition range, and an object image extraction unit that extracts, from the image, an image of an area corresponding to the extraction target area coordinates as an object image.

[0007] A non-transitory computer-readable medium having an image processing program recorded thereon according to one embodiment causes an arithmetic unit to execute the following: a two-dimensional data acquisition process for acquiring an image, which is two-dimensional data acquired by a two-dimensional data acquisition unit; a three-dimensional data acquisition process for acquiring three-dimensional data output by a three-dimensional data acquisition unit for at least a portion of the area captured as the image; an extraction target area setting process for outputting, as extraction target area coordinates, the two-dimensional coordinates of an area including point cloud data whose distance information of the three-dimensional data is within a predetermined recognition range; and an object image extraction process for extracting, from the image, an image of an area corresponding to the extraction target area coordinates as an object image.

[0008] An image processing method according to one embodiment has an arithmetic unit execute a two-dimensional data acquisition process for acquiring an image, which is two-dimensional data acquired by a two-dimensional data acquisition unit; a three-dimensional data acquisition process for acquiring three-dimensional data output by a three-dimensional data acquisition unit for at least a portion of the area captured as the image; an extraction target area setting process for outputting, as extraction target area coordinates, the two-dimensional coordinates of an area including point cloud data whose distance information of the three-dimensional data is within a predetermined recognition range; and an object image extraction process for extracting, from the image, an image of an area corresponding to the extraction target area coordinates as an object image.

[0009] According to an image processing device, a non-temporary recording medium on which a program for the image processing device is recorded, and a method thereof according to an embodiment, it is possible to detect a distant object based on an image that is two-dimensional data.

[0010] FIG. 1 is a diagram illustrating an object detected by an image processing device according to a first embodiment. FIG. 2 is a block diagram of the image processing device according to the first embodiment. FIG. 3 is a hardware configuration diagram of the image processing device according to the first embodiment. FIG. 4 is a flowchart illustrating the operation of the image processing device according to the first embodiment. FIG. 5 is a block diagram of the image processing device according to a second embodiment. FIG. 6 is a flowchart illustrating the operation of the image processing device according to the second embodiment.

[0011] For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. Furthermore, each element shown in the drawings as a functional block performing various processes can be configured in hardware with a CPU (Central Processing Unit), memory, and other circuits, and in software with a program loaded into memory. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof, and are not limited to any of these. In addition, the same elements are designated by the same reference numerals in each drawing, and redundant explanations are omitted as necessary.

[0012] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program can also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0013] First Embodiment First, we will explain images for which object detection accuracy is improved by using the image processing device described in the embodiment. FIG. 1 illustrates an example of an object detected by the image processing device according to the first embodiment. The example shown in FIG. 1 is an image captured of a landscape seen in the direction of train travel. Trains use special signal lights (the "special light" in FIG. 1 ) that are visible from 800 meters or more away from the driver. These special signal lights are installed at locations requiring vigilance against falling rocks, avalanches, strong winds, railroad crossings, and other hazards. While these special signal lights are visually inspected by the driver, they are normally off when there are no abnormalities, and their unique nature of issuing stop signals at unexpected times makes it difficult to respond immediately. Therefore, there is a demand for detecting signals and obstacles installed far away and issuing warnings to the driver. However, as shown in FIG. 1 , the special signal lights to be detected are hidden by the support masts used to hang overhead wires, making them difficult to detect from a single image.

[0014] Therefore, the image processing device described below utilizes three-dimensional data that measures distance information, such as LiDAR (Light Detection and Ranging), to generate an object image that cuts out only the shooting range within a predetermined recognition distance. Using such an object image makes it possible to eliminate objects that obstruct the recognition of the detection target in the near and far directions, allowing the image processing device described below to improve object recognition accuracy. Object recognition using such object images is a process required not only for railways but also for all mobile objects, such as automobiles and drones.

[0015] 2 shows a block diagram of the image processing device 1 according to the first embodiment. As shown in FIG. 1, the image processing device 1 according to the first embodiment includes a three-dimensional data acquisition unit 11, an extraction target region setting unit 12, a two-dimensional data acquisition unit 13, and an object image extraction unit 14.

[0016] The three-dimensional data acquisition unit 11 acquires three-dimensional data for at least a part of the area captured as an image acquired by the two-dimensional data acquisition unit 13. The three-dimensional data acquisition unit 11 outputs point cloud data, which is a collection of measurement points whose values ​​change depending on the magnitude of the distance, such as LiDAR.

[0017] The extraction target area setting unit 12 outputs, as extraction target area coordinates, the two-dimensional coordinates of an area including point cloud data whose distance information of the three-dimensional data falls within a predetermined recognition range. In other words, the extraction target area coordinates are the two-dimensional coordinates of a portion where an object exists within the range of the recognition range.

[0018] The two-dimensional data acquisition unit 13 acquires an image as two-dimensional data. Here, the two-dimensional data acquisition unit 13 is, for example, a device that outputs the imaging range of an optical camera, an infrared camera, or the like as two-dimensional image information.

[0019] The object image extraction unit 14 extracts an image of an area corresponding to the extraction target area coordinates from the image as an object image. Here, in the image processing device 1, the two-dimensional coordinates of the shooting range set by the extraction target area setting unit 12 and the two-dimensional coordinates of the point cloud data of the three-dimensional data set by the three-dimensional data acquisition unit 11 are pre-calibrated to match. Furthermore, since the number of pixels in the point cloud data is smaller than the number of pixels in the image output by the two-dimensional data acquisition unit 13, it is preferable that the object image extraction unit 14 extracts an image of an area slightly wider than the area specified by the extraction target area coordinates as the object image, taking into account this difference in pixel count when extracting the object image from the image.

[0020] The image processing device 1 according to the first embodiment can be configured as dedicated hardware, but can also be realized by executing an image processing program on a computer. Therefore, a hardware configuration diagram of the image processing device according to the first embodiment is shown in Fig. 3. Fig. 3 shows a computer 100 as the hardware configuration of the image processing device 1.

[0021] The computer 100 includes a calculation unit 101, a memory 102, a three-dimensional data acquisition unit 103, and an object image extraction unit 14. In the computer 100, the calculation unit 101, the memory 102, the three-dimensional data acquisition unit 103, and the two-dimensional data acquisition unit 104 are configured to be able to communicate with each other via a bus.

[0022] 3, the three-dimensional data acquisition unit 103 and the two-dimensional data acquisition unit 104 are physical hardware such as sensors, and image data and three-dimensional data are stored in the memory 102 via a bus. The calculation unit 101 executes an image processing program and outputs the generated object image to the memory 102. The memory 102 is a storage device that stores data handled by a computer, such as a volatile memory such as a DRAM or a non-volatile memory such as a flash memory.

[0023] The image processing program causes the calculation unit 101 to perform two-dimensional data acquisition processing, three-dimensional data acquisition processing, extraction target area setting processing, and object image extraction processing. The two-dimensional data acquisition processing is processing performed by the two-dimensional data acquisition unit 13, and stores images, which are two-dimensional data acquired by the two-dimensional data acquisition unit 104, in the memory 102. The three-dimensional data acquisition processing is processing performed by the three-dimensional data acquisition unit 11, and stores three-dimensional data output by the three-dimensional data acquisition unit 103 for at least a portion of the range captured as the image acquired by the two-dimensional data acquisition unit 104 in the memory 102. The extraction target area setting processing is processing performed by the extraction target area setting unit 12, and outputs, as extraction target area coordinates, two-dimensional coordinates of an area including point cloud data whose distance information of the three-dimensional data falls within a predetermined recognition range. The object image extraction processing is processing performed by the object image extraction unit 14, and extracts an image of an area corresponding to the extraction target area coordinates from the image as an object image. The calculation unit 101 may acquire data directly from the three-dimensional data acquisition unit 103 and the two-dimensional data acquisition unit 104 without going through the memory 102 .

[0024] Here, the operation of the image processing device 1 according to the first embodiment will be described. Fig. 4 shows a flowchart for explaining the operation of the image processing device according to the first embodiment.

[0025] As shown in FIG. 4, when the image processing device 1 starts operation, it acquires three-dimensional data using the three-dimensional data acquisition unit 11 (step S11). Then, the image processing device 1 uses the extraction target area setting unit 12 to set the two-dimensional coordinates of point cloud data obtained from an object located within a recognition zone that is a predetermined distance from the shooting position as extraction target area coordinates (step S12). In parallel with the processes of steps S11 and S12, the image processing device 1 also acquires two-dimensional data (e.g., an image) using the two-dimensional data acquisition unit 13 (step S13). Then, the image processing device 1 determines the position of the detection target object in the two-dimensional data based on the extraction target area coordinates and extracts an object image containing the detection target object from the two-dimensional data (step S14). The object image extraction unit 14 then outputs the object image, completing the output operation of one object image (step S15). The image processing device 1 repeatedly executes steps S11 to S15 at a predetermined interval.

[0026] As described above, the image processing device 1 according to the first embodiment outputs an object image including only an image in which an object within a recognition zone is reflected. This reduces the amount of information processing required when recognizing a detection target object from the object image output by the image processing device 1. Furthermore, it is possible to improve recognition accuracy by performing object recognition using an object image including only objects located at a distance where the detected object is assumed to be located based on distance information.

[0027] In the second embodiment, an image processing device 2 will be described, which is another embodiment of the image processing device 1 according to the first embodiment. Note that the same components as those described in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0028] FIG. 5 shows a block diagram of an image processing device 2 according to the second embodiment. As shown in FIG. 5, the image processing device 2 according to the second embodiment is obtained by adding an object recognition unit 21 and a notification unit 22 to the image processing device 1 according to the first embodiment. The object recognition unit 21 recognizes an object included in an object image. Furthermore, the object recognition unit 21 may recognize the object by taking into account the coordinates of an extraction target area when recognizing the object. The object recognition unit 21 can, for example, recognize the object by comparing it with pre-registered detection target candidates, or can perform recognition using artificial intelligence. The notification unit 22 notifies information about the object recognized by the object recognition unit 21.

[0029] The object recognition unit 21 recognizes, as an object, at least one of a signal (e.g., a special signal light) located within the imaging range of the two-dimensional data acquisition unit, a stone, a fallen tree, and a person present in a no-entry zone set within the imaging range of the two-dimensional data acquisition unit. Furthermore, when a special signal light is the recognition target, the light emission pattern is also recognized from the image. The notification unit 22 notifies a person or device that will receive the notification, such as a driver, of the recognition result based on the object or light emission pattern recognized by the object recognition unit 21. Possible devices that receive the notification here include, for example, a railway operation control system, a vehicle brake, and various other devices.

[0030] The operations of the object recognition unit 21 and the notification unit 22 can also be realized by the image processing program executed by the calculation unit 101 shown in FIG.

[0031] Next, the operation of the image processing device 2 according to the second embodiment will be described. Fig. 6 shows a flowchart illustrating the operation of the image processing device 2 according to the second embodiment. As shown in Fig. 6, the operation of the image processing device 2 according to the second embodiment involves processing of steps S21 to S26 using the object image output in step S15 of the operation of the image processing device 1 according to the first embodiment shown in Fig. 4.

[0032] In step S21, the object recognition unit 21 recognizes an object using the object image output in step S15. If the recognized object is a traffic light (YES branch in step S22), the object recognition unit 21 recognizes a light emission pattern from the image corresponding to the traffic light and notifies the driver of the recognition result of the light emission pattern using the notification unit 22 (steps S22 to S24). On the other hand, if the recognized object is not a traffic light (NO branch in step S22), the object recognition unit 21 issues a warning to the driver if it corresponds to a foreign object for which a warning should be issued (YES branch in step S25, step S26), or terminates operation without issuing a warning if the object does not require a warning (NO branch in step S25).

[0033] As described above, the image processing device 2 according to the second embodiment can notify the driver of specific recognition results or issue a warning using the object image output by the object image extraction unit 14. Furthermore, in the image processing device 2 according to the second embodiment, the object image output by the object image extraction unit 14 is used, allowing the recognition process in the object recognition unit 21 to be performed with a small amount of calculation.

[0034] In the third embodiment, an image processing device 3 will be described, which is another embodiment of the image processing device 2 according to the second embodiment. Note that the same components as those described in the first and second embodiments are denoted by the same reference numerals as those in the first and second embodiments, and the description thereof will be omitted.

[0035] Fig. 7 shows a block diagram of the image processing device 3 according to the third embodiment. As shown in Fig. 7, the image processing device 3 according to the third embodiment is obtained by adding a self-position estimation unit 31 and a scan direction designation unit 32 to the image processing device 2 according to the second embodiment and by replacing the object recognition unit 21 with an object recognition unit 33.

[0036] The self-position estimation unit 31 estimates the current position of the device itself and outputs self-position estimation information. For example, the self-position estimation unit 31 outputs the current position of the vehicle on which the image processing device 3 is mounted as self-position estimation information, based on position information acquired using a device such as a GPS.

[0037] The scan direction designation unit 32 designates the direction of acquisition of two-dimensional data and the direction of acquisition of three-dimensional data. More specifically, the scan direction designation unit 32 determines the current geographical position of the device itself from the self-position estimation information and issues orientation control instructions to the three-dimensional data designation unit 11 and the two-dimensional data designation unit 13 so that the three-dimensional data designation unit 11 and the two-dimensional data designation unit 13 are oriented in the shooting direction associated with the determined position. For this reason, the three-dimensional data designation unit 11 and the two-dimensional data designation unit 13 are equipped with a mechanism capable of changing their orientation. Furthermore, when the direction of the detection object can be estimated based on the recognition result of the object recognition unit 33, the scan direction designation unit 32 designates the direction of acquisition of two-dimensional data and the direction of acquisition of three-dimensional data using that result. In other words, in the third embodiment, the recognition result of the object recognition unit 33 is fed back to the scan direction designation unit 32 to improve the efficiency of scanning the detection object.

[0038] The object recognition unit 33 switches between candidate lists in which objects to be recognized are listed, based on the self-location estimation information. The image processing device 3 is mounted on a mobile object, and it is conceivable that the type of object to be recognized will differ depending on the geographical location of the mobile object. Therefore, the object recognition unit 33 can shorten the processing time by switching between candidate lists in which objects to be recognized are listed, based on the self-location estimation information.

[0039] Here, a flowchart illustrating the operation of the image processing device 3 according to the third embodiment is shown in Fig. 8. As shown in Fig. 8, the image processing device 3 according to the third embodiment performs the processes of steps S31 and S32 before steps S11 and S13 of the image processing device 2 according to the second embodiment shown in Fig. 6, and performs the process of step S33 instead of step S14. In addition, the process of step S34 is performed after the process of step S25.

[0040] When the image processing device 3 according to the third embodiment starts operation, first, the self-position estimation unit 31 performs a self-position estimation process to output self-position estimation information (step S31). Then, in the image processing device 3, the scan direction designation unit 32 designates the scan directions of the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13 using the self-position estimation information (step S32). Thereafter, the image processing device 3 performs the processes from step S11 onwards.

[0041] In the image processing device 3 according to the third embodiment, the process of step S14 is replaced with step S33. In step S33, the object recognition unit 33 determines the position of the detection target object in the two-dimensional data based on the self-position estimation information and the coordinates of the extraction target area, and cuts out an object image including the detection target object from the two-dimensional data.

[0042] Furthermore, in the image processing device 3 according to the third embodiment, after it is determined in step S25 that no foreign object is present, it is determined whether or not there is any other object to be scanned that is not a foreign object (step S34). If it is determined in step S34 that a scanning object is present, its position in the image is fed back to the scanning direction designation unit 32 (NO branch of step S24). On the other hand, if it is determined in step S34 that no scanning object is present, the processing ends.

[0043] As described above, in the image processing device 3 according to the third embodiment, by changing the scanning directions of the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13 according to the position, it is possible to improve the detection accuracy of the detection target object even if the shooting angle of view of the three-dimensional data acquisition unit 11 and the extraction target area setting unit 12 is narrow. Furthermore, in the image processing device 3 according to the third embodiment, by changing the scanning directions of the three-dimensional data acquisition unit 11 and the two-dimensional data acquisition unit 13, it is possible to reduce the probability that the detection target object will be hidden by another object.

[0044] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0045] 1 to 3 Image processing device 11 Three-dimensional data acquisition unit 12 Extraction target area setting unit 13 Two-dimensional data acquisition unit 14 Object image extraction unit 21 Object recognition unit 22 Notification unit 31 Self-position estimation unit 32 Scan direction designation unit 33 Object recognition unit 100 Computer 101 Calculation unit 102 Memory 103 Three-dimensional data acquisition unit 104 Two-dimensional data acquisition unit

Claims

1. A two-dimensional data acquisition unit that acquires an image, which is two-dimensional data; a three-dimensional data acquisition unit for acquiring three-dimensional data for at least a part of the area captured as the image; an extraction target region setting unit that outputs, as extraction target region coordinates, two-dimensional coordinates of a region including point cloud data in which distance information of the three-dimensional data falls within a predetermined recognition section; an object image extraction unit that extracts an image of an area corresponding to the extraction target area coordinates from the image as an object image; An image processing device comprising:

2. The image processing device according to claim 1 , further comprising an object recognition unit that recognizes an object included in the object image.

3. The image processing device according to claim 2 , wherein the object recognition unit recognizes the object by taking into account the extraction target area coordinates.

4. The image processing device according to claim 2 , further comprising a notification unit that notifies information about the object recognized by the object recognition unit.

5. a self-position estimation unit that estimates a current position of the device and outputs self-position estimation information; The image processing device according to claim 2 , wherein the object recognition unit switches a candidate list in which the object to be recognized is described, depending on the self-location estimation information.

6. The image processing device according to claim 2 , wherein the object includes at least one of a signal located within the shooting range of the two-dimensional data acquisition unit, a stone, a fallen tree, and a person located in a no-entry area set with respect to the shooting range of the two-dimensional data acquisition unit.

7. a self-location estimation unit that estimates a current location of the device and outputs self-location estimation information; a scan direction designation unit that designates an acquisition direction of the two-dimensional data and an acquisition direction of the three-dimensional data, The image processing apparatus according to claim 1 , wherein the two-dimensional data acquisition unit and the three-dimensional data acquisition unit acquire data in a direction specified by the scan direction designation unit.

8. The image processing apparatus according to claim 1 , wherein the three-dimensional data acquisition unit outputs point cloud data that is a set of measurement points whose values ​​change depending on the magnitude of the distance.

9. A two-dimensional data acquisition process for acquiring an image, which is two-dimensional data acquired by a two-dimensional data acquisition unit; a three-dimensional data acquisition process for acquiring three-dimensional data output by a three-dimensional data acquisition unit for at least a part of the area captured as the image; an extraction target region setting process for outputting, as extraction target region coordinates, two-dimensional coordinates of a region including point cloud data in which distance information of the three-dimensional data falls within a predetermined recognition section; an object image extraction process for extracting an image of an area corresponding to the extraction target area coordinates from the image as an object image; An image processing program that causes the calculation unit to execute the above.

10. A two-dimensional data acquisition process for acquiring an image, which is two-dimensional data acquired by a two-dimensional data acquisition unit; a three-dimensional data acquisition process for acquiring three-dimensional data output by a three-dimensional data acquisition unit for at least a part of the area captured as the image; an extraction target region setting process for outputting, as extraction target region coordinates, two-dimensional coordinates of a region including point cloud data in which distance information of the three-dimensional data falls within a predetermined recognition section; an object image extraction process for extracting an image of an area corresponding to the extraction target area coordinates from the image as an object image; The image processing method includes:

Images