Image processing device, mobile body, image processing method, and computer program

Abstract

To provide an image processing device capable of detecting a manhole where vehicles are prohibited from entering and notifying a driver.SOLUTION: An image processing device includes a receiving unit for receiving an image of the surroundings of a vehicle acquired by an imaging unit such as a camera, an overhead image generating unit that generates an overhead image from the image acquired by the receiving unit, a detection unit that detects manholes from the generated overhead image, a no-vehicle-riding detection unit that detects no-vehicle-riding signs from the detected manhole image, and an image processing unit that highlights the positions of no-vehicle-riding manholes included in the generated overhead image and outputs the image to a display unit that displays the highlighted image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, a moving object, an image processing method, a computer program, and the like. [Background technology]

[0002] Manholes marked "No Vehicles Allowed" are sometimes installed around private residences or in parking lots. Manholes marked "No Vehicles Allowed" have low load resistance. Therefore, if a driver does not notice when parking and drives a vehicle over a manhole marked "No Vehicles Allowed," the weight of the vehicle may cause the manhole to break.

[0003] For example, Patent Document 1 discloses a technology in which a manhole is detected by installing a plurality of magnet pieces arranged in a pattern corresponding to the contents of system equipment information in the manhole and having a vehicle detect the manhole by detecting the movement of the magnet pieces.

[0004] Patent Document 2 discloses a technology that prevents a manhole from being mistakenly recognized as an obstacle based on an image captured by a camera, causing the vehicle to make a sudden stop, and excludes manholes from being considered obstacles. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 8-137985 [Patent Document 2] Patent Publication No. 2022-28989 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the conventional technology disclosed in the above-mentioned Patent Document 1, in order to detect manholes, it was necessary to embed additional magnet pieces in existing manholes, and only manholes with such a configuration could be detected.

[0007] Furthermore, the conventional technology disclosed in Patent Document 2 excludes manholes from the list of obstacles, so even if a manhole is detected, the vehicle will continue to drive through it, which may result in the vehicle destroying a manhole that is prohibited to drive through.

[0008] Therefore, one object of the present invention is to provide an image processing device capable of detecting manholes where vehicles are prohibited from entering and notifying the driver of the same. [Means for solving the problem]

[0009] In order to achieve the above object, the image processing device of the present invention includes a receiving unit for receiving an image of the surroundings of a vehicle captured by an imaging unit such as a camera. an overhead image generating unit that generates an overhead image from the image acquired by the receiving unit; A detection unit that detects manholes from the generated overhead image; a vehicle-restricted detection unit that detects signs indicating that vehicle-restricted driving is prohibited from the detected manhole image; The system is characterized by having an image processing unit that highlights the positions of manholes where vehicles are prohibited from entering, which are included in the generated overhead image, and outputs the highlighted image to a display unit. Effect of the Invention

[0010] According to the present invention, it is possible to realize an image processing device that can detect manholes where vehicles are prohibited from entering and notify the driver. [Brief description of the drawings]

[0011] [Figure 1] 1 is a functional block diagram showing an example of the configuration of an image processing device 100 according to a first embodiment. [Diagram 2] 1 is a diagram showing an example of positions of cameras 200a to 200d as an imaging unit mounted on a vehicle 10 in the first embodiment. [Diagram 3] 4 is a flowchart illustrating an example of an image processing method of the image processing device 100 of the first embodiment. [Figure 4]1 is a diagram showing an example of an overhead image of the surroundings of a vehicle 10 generated by an overhead image generating unit 120 in the first embodiment. FIG. [Diagram 5] FIG. 13 is a diagram showing an example of a manhole where vehicles are prohibited from entering. [Figure 6] FIG. 13 is a diagram showing an example of an overhead image on which an image processing unit 140 of the first embodiment has performed highlighting. [Figure 7] 1 is a functional block diagram showing an example of the configuration of an image processing device 100 according to a second embodiment. [Figure 8] 10 is a flowchart illustrating an example of an image processing method of the image processing device 100 of the second embodiment. [Figure 9] FIG. 13 is a diagram showing an example of an overhead image on which an image processing unit 140 of the second embodiment has performed highlighting. [Figure 10] FIG. 13 is a diagram showing an example of an overhead image in which a no-boarding manhole on a vehicle 10 is highlighted by an image processing unit 140 of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0013] (Embodiment 1) Fig. 1 is a functional block diagram showing an example of the configuration of an image processing device 100 according to the first embodiment. Some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like serving as a computer (not shown) included in the image processing device to execute a computer program stored in a memory serving as a storage medium (not shown).

[0014] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0015] In addition, the functional blocks shown in Fig. 1 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 1 also applies to Fig. 7.

[0016] Image processing device 100 has a receiving unit 110, an overhead image generating unit 120, a detecting unit 130, and an image processing unit 140, and image processing unit 140 has a control unit 150. Image processing device 100 is also connected to cameras 200a, 200b, 200c, and 200d as imaging units, receives image signals, and outputs the image signals to display unit 300.

[0017] The control unit 150 has a built-in CPU and the like as a computer, and functions as a control means for controlling the operation of each part of the entire image processing apparatus based on a computer program stored in a memory serving as a storage medium (not shown).

[0018] 2 is a diagram showing an example of the positions of cameras 200a to 200d as imaging units mounted on vehicle 10 in embodiment 1. Cameras 200a to 200d are disposed in front, right, left, and rear of vehicle 10 as a moving body, respectively, and capture images of the surroundings of vehicle 10.

[0019] In the first embodiment, the image processing device 100 and the display unit 300 are disposed inside the vehicle 10, and the display unit 300 is disposed at a position where the driver can see it. However, the image processing device 100 and the display unit 300 may be provided in, for example, a remote terminal.

[0020] Fig. 3 is a flowchart showing an example of an image processing method of the image processing device 100 of embodiment 1. Note that the operation of each step of the flowchart in Fig. 3 is performed sequentially by a CPU or the like as a computer in the control unit 150 executing a computer program stored in a memory.

[0021] In step S101, the receiving unit 110 receives image signals of the surroundings of the vehicle acquired by the cameras 200a to 200d as imaging units connected to the image processing device 100, and supplies the received video signals to the overhead image generating unit 120. Here, step S101 functions as a receiving step for receiving images of the surroundings of the vehicle acquired by the imaging units.

[0022] In step S102, the overhead image generation unit 120 generates an overhead image of the surroundings of the vehicle 10 viewed from above by performing synthesis processing, distortion correction, etc. based on images of four locations on the front, back, left and right sides of the vehicle 10 acquired by the receiving unit 110.

[0023] Here, step S102 functions as an overhead image generating step of generating an overhead image based on the image acquired in the receiving step. The overhead image generating unit 120 supplies the generated overhead image to the detecting unit 130 and the image processing unit 140.

[0024] The detection unit 130 detects an object included in the overhead image based on the overhead image supplied from the overhead image generation unit 120. The detection unit 130 includes a manhole detection unit 131 and a vehicle-entry prohibition detection unit 132.

[0025] In step S103, the manhole detection unit 131 determines whether it has detected a manhole in the overhead image. If it determines that a manhole is present, the process proceeds to step S104; if it determines No, the flow in FIG. 3 is terminated.

[0026] In step S104, the vehicle-entry prohibition detection unit 132 detects a manhole having a notation indicating that vehicle entry is prohibited, based on the overhead image detected by the manhole detection unit 131. Then, in step S104, it is determined whether or not a manhole indicating that vehicle entry is prohibited has been detected.

[0027] Here, step S104 functions as a vehicle-entry prohibition detection step for detecting a manhole for which vehicle entry is prohibited based on the overhead image. Note that the detection by the manhole detection unit 131 and the vehicle-entry prohibition detection unit 132 uses, for example, recognition processing using deep learning.

[0028] Examples of deep learning include YOLO (You Only Look Once) and Faster R-CNN (Regional Convolution Neural Network), which are easy to learn and perform detection quickly.

[0029] If it is determined that the answer is Yes in step S104, the process proceeds to step S105, and if it is determined that the answer is No, the flow of FIG. 3 ends.

[0030] In step S105, the detection unit 130 supplies the position of the manhole where vehicular riding is prohibited to the image processing unit 140. Based on the position information of the manhole where vehicular riding is prohibited supplied from the detection unit 130, the image processing unit 140 generates an image in which the manhole where vehicular riding is prohibited included in the overhead image supplied from the overhead image generation unit 120 is highlighted.

[0031] Here, step S105 functions as an image processing step for generating an image in which manholes for which vehicles are not permitted to enter, which are included in the overhead image, are highlighted.

[0032] Moreover, the image processing unit 140 outputs the generated overhead image to the display unit 300. The display unit 300 includes a display panel such as a liquid crystal display mounted on the vehicle, and displays the supplied overhead image.

[0033] FIG. 4 is a diagram showing an example of an overhead image of the surroundings of the vehicle 10 generated by the overhead image generating unit 120 of the first embodiment. A display method when using an overhead image of the surroundings of the vehicle 10 will be described with reference to FIG.

[0034] 4, there are manholes 20a, 20b, no-vehicle-entry manholes 30a, and no-vehicle-entry manholes 30b. Note that the vehicle 10 displayed in the overhead image does not need to be an image acquired from the camera 200, and may be an image of the vehicle 10 prepared in advance or an image icon of the same size as the vehicle 10.

[0035] FIG. 5 is a diagram showing an example of a manhole where vehicles are prohibited. As shown in FIG. 5, a manhole 30 where vehicles are prohibited has a no-vehicle sign 31 written on the surface of the manhole indicating that "vehicles are prohibited."

[0036] The detection unit 130 that receives the overhead image detects the no-vehicle manholes by image recognition. First, the manhole detection unit 131 detects manholes from their features such as their shapes based on the overhead image of Fig. 4 generated by the overhead image generation unit 120. As a result, the manhole detection unit 131 detects the manholes 20a, 20b, the no-vehicle manhole 30a, and the no-vehicle manhole 30b.

[0037] Next, the vehicle-riding prohibition detection unit 132 detects vehicle-riding prohibited manholes by image recognition of the manholes 20a, 20b, vehicle-riding prohibited manholes 30a, and vehicle-riding prohibited manholes 30b detected by the manhole detection unit 131. That is, the vehicle-riding prohibition detection unit 132 detects the vehicle-riding prohibited manholes 30a and 30b by image recognition of the vehicle-riding prohibited notation 31 included in the image of the manholes.

[0038] Furthermore, the detection unit 130 outputs information on the positions and shapes of the no-vehicle-riding manholes 30a and 30b contained in the overhead image to the image processing unit 140. The image processing unit 140 highlights the no-vehicle-riding manholes 30a and 30b in the overhead image supplied from the overhead image generation unit 120 based on the position information of the no-vehicle-riding manholes 30a and 30b.

[0039] Fig. 6 is a diagram showing an example of an overhead image in which the image processing unit 140 of the first embodiment has performed highlighting. The images of the no-vehicle-entry manhole 30a and the no-vehicle-entry manhole 30b in the overhead image are highlighted in a highlighting color (e.g., red). On the other hand, the images of the manholes 20a and 20b are not highlighted and are displayed as normal images. Such an overhead image is output to the display unit 300, and the overhead image as shown in Fig. 6 is displayed on the display unit 300.

[0040] By performing the above processing, the driver of the vehicle 10 or the operator of the remote terminal can grasp the location of the manhole where vehicles are prohibited from entering by checking the overhead image. Therefore, it is possible to prevent the vehicle from driving over the manhole and destroying it when driving the vehicle.

[0041] (Embodiment 2) Fig. 7 is a functional block diagram showing an example of the configuration of the image processing device 100 of the second embodiment. Hereinafter, the operation of the image processing device 100 in the second embodiment of the present invention will be described with reference to the configuration diagram of Fig. 7. The image processing unit 140 of the image processing device 100 of the second embodiment includes an interface for connecting to the travel route estimation unit 400 and the storage unit 700.

[0042] The travel route estimation unit 400 calculates the steering angle of the vehicle 10 from the rotation angle of a steering wheel 500 provided on the vehicle 10. The larger the rotation angle of the steering wheel, the larger the steering angle of the tires. In addition, the travel route estimation unit 400 identifies the moving direction (forward, backward) of the vehicle 10 based on gear information of a driving gear 600 of the vehicle 10.

[0043] For example, if the vehicle is in reverse gear, it can be estimated that the vehicle will travel backwards. Based on this information, the travel route estimation unit 400 supplies the image processing unit 140 with information on the travel route of the vehicle 10.

[0044] The storage unit 700 stores the position information of the manholes where vehicular access is prohibited supplied from the image processing unit 140, and supplies the stored position information of the manholes where vehicular access is prohibited to the image processing unit 140.

[0045] Next, the operation of the image processing device 100 in the second embodiment will be described with reference to Figures 8 and 9. Figure 8 is a flowchart showing an example of an image processing method of the image processing device 100 in the second embodiment.

[0046] The operation of each step in the flowchart of FIG. 8 is performed in sequence by a CPU or the like serving as a computer in the control unit 150 executing a computer program stored in a memory.

[0047] 9 is a diagram showing an example of an overhead image highlighted by the image processing unit 140 of the second embodiment, and shows an example of the display of an overhead image when reversing (moving backward) to the right in the second embodiment. A driving route 40a and a driving route 40b represent the driving routes in the display image generated by the image processing unit 140.

[0048] Incidentally, steps S201 to S205 in the flowchart of FIG. 8 are the same as steps S101 to S105 in the first embodiment, and therefore a description thereof will be omitted.

[0049] In step S206, if it is determined in step S205 that the manhole has a no-vehicle sign, it is determined whether the detected no-vehicle-entry manhole is on the travel route. That is, in step S206, travel route estimation unit 400 estimates that the travel route is in the right backward direction based on the rightward rotation angle of the steering wheel from steering wheel 500 and the information from gear 600 that the gear is in reverse.

[0050] Then, the travel route estimation unit 400 supplies the travel route information to the image processing unit 140. In step S206, the image processing unit 140 determines whether or not the manhole for which vehicles are not permitted to enter, detected in step S205, is on the travel route indicated by the travel route 40a and the travel route 40b, based on the supplied travel route information.

[0051] If it is determined in step S206 that the manhole is on the travel route, the process proceeds to step S207. If it is determined No in step S206, the flow of Fig. 8 ends. In step S207, any manholes on the travel route that are prohibited from vehicle entry are highlighted.

[0052] That is, in step S206, the image processing unit 140 acquires information on the driving route of the vehicle 10, and in step S207, highlights manholes on the driving route where vehicles are not permitted to enter, in the overhead image.

[0053] 9, for example, of the no-vehicle-entry manhole 30a and the no-vehicle-entry manhole 30b, the manhole that is on the driving route is the no-vehicle-entry manhole 30a. Therefore, in step S207, processing is performed to highlight the image of the no-vehicle-entry manhole 30a.

[0054] In addition, in step S208, the image processing unit 140 generates an overhead image in which the no-vehicle-entry manholes are highlighted, and further adds the driving routes 40a and 40b as driving route information to the overhead image, and displays the generated overhead image on the display unit 300.

[0055] By executing the above image processing method, it is possible to detect manholes on the driving route that are prohibited from being driven over and to notify the driver, so that it is possible to notify the driver of manholes that the driver may run over with high accuracy, and therefore the driver can easily avoid the manholes that are prohibited from being driven over.

[0056] 10 is a diagram showing an example of an overhead image in which a no-boarding manhole on the vehicle 10 is highlighted by the image processing unit 140 of the second embodiment. In FIG. 10, the no-boarding manhole 30a in FIG. 8 is present under the vehicle 10.

[0057] The image processing unit 140 calculates the amount of movement of the vehicle 10 included in the overhead image. Then, the position of the no-ride manhole 30a hidden under the vehicle 10 is estimated and displayed on the vehicle 10.

[0058] The image processing unit 140 supplies the position information of the no-entry manhole hidden under the vehicle 10 to the storage unit 700 before the power of the vehicle 10 is turned off, and the storage unit 700 holds the position information of the manhole when the vehicle is stopped. That is, the storage unit 700 of the image processing unit 140 stores and holds the position information of the no-entry manhole when the power of the vehicle is turned off.

[0059] When the power supply of the vehicle 10 is turned on again, the image processing unit 140 reads out the position information of the manhole where vehicle entry is prohibited stored in the storage unit 700. Then, based on the position information, the position of the manhole where vehicle entry is prohibited under the vehicle 10 is highlighted on the overhead image, and the image is supplied to the display unit 300 for superimposition on the overhead image.

[0060] That is, when a manhole indicating that vehicles are not permitted to enter, which is included in the overhead image, is under the vehicle 10, the image processing unit 140 generates an overhead image in which the position of the manhole indicating that vehicles are not permitted to enter is superimposed on the image of the vehicle 10.

[0061] In this embodiment, by executing such an image processing method, it is possible to prevent the driver from driving over a manhole that is marked as off-limits under the vehicle when starting off and damaging the manhole.

[0062] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications and combinations of the above-mentioned embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention includes the following combinations.

[0063] (Configuration 1) An image processing device comprising: a receiving unit for receiving an image of the surroundings of a vehicle acquired by an imaging unit; an overhead image generating unit for generating an overhead image based on the image acquired by the receiving unit; a no-vehicle-riding detection unit for detecting a no-vehicle-riding manhole based on the overhead image; and an image processing unit for generating an image in which the no-vehicle-riding manhole included in the overhead image is highlighted.

[0064] (Configuration 2) The image processing device according to configuration 1, characterized in that the image processing unit acquires driving route information of the vehicle and highlights the manholes on the driving route where vehicles are not permitted to enter in the overhead image.

[0065] (Configuration 3) The image processing device described in Configuration 1 or 2, characterized in that when the manhole indicating that vehicular access is prohibited included in the overhead image is located under the vehicle, the image processing unit generates the overhead image in which the position of the manhole indicating that vehicular access is prohibited is superimposed on the image of the vehicle.

[0066] (Configuration 4) An image processing device according to any one of configurations 1 to 3, characterized in that the image processing unit has a memory unit for storing and retaining position information of the no-boarding manhole when the power of the vehicle is turned off.

[0067] (Configuration 5) The image processing device described in Configuration 4, characterized in that when the vehicle is powered on, the image processing unit reads out the location information of the no-vehicle manhole from the memory unit and superimposes it on the overhead image.

[0068] (Configuration 6) A moving body including the image processing device according to any one of configurations 1 to 5 and the imaging unit.

[0069] (Method) An image processing method comprising: a receiving step for receiving an image of the surroundings of the vehicle acquired by an imaging unit; an overhead image generating step for generating an overhead image based on the image acquired in the receiving step; a no-vehicle-riding detection step for detecting a no-vehicle-riding manhole based on the overhead image; and an image processing step for generating an image in which the no-vehicle-riding manhole included in the overhead image is highlighted.

[0070] (Program) A computer program for controlling each part of the image processing device according to any one of configurations 1 to 5 or the moving object according to configuration 6 by a computer.

[0071] In order to realize a part or all of the control in the above-described embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0072] 100...Image processing device 110... Receiver 120... Bird's-eye view image generating unit 130...Detection unit 131 Manhole detector 132....No-ride detection unit 140...Image processing unit 150...Control section 200a···Camera 200b···Camera 200c···Camera 200d···Camera 300...Display section 400 ····Route estimation unit 500···Handle 600···gear 700...Storage section 10. Vehicle 20a···Manhole 20b...Manhole 30a No Vehicles Allowed Manhole 30b No vehicle access manhole 31...No driving allowed sign 40a····Route 40b····Running route

Claims

1. a receiving unit for receiving an image of the surroundings of the vehicle acquired by the imaging unit; an overhead image generating unit that generates an overhead image based on the image acquired by the receiving unit; a vehicle-entry-prohibited detection unit that detects a vehicle-entry-prohibited manhole based on the overhead image; and an image processing unit that generates an image in which the manhole marked as "no vehicle access" included in the overhead image is highlighted.

2. The image processing device according to claim 1 , wherein the image processing unit acquires information on a driving route of the vehicle, and highlights the manholes on the driving route where vehicles are not permitted to enter in the overhead image.

3. The image processing device according to claim 1, characterized in that, when the manhole indicating that no vehicle entry is permitted, which is included in the overhead image, is located under the vehicle, the image processing unit generates the overhead image in which the position of the manhole indicating that no vehicle entry is permitted is superimposed on the image of the vehicle.

4. 2. The image processing device according to claim 1, wherein the image processing unit has a storage unit for storing and holding position information of the manhole where the vehicle is prohibited from entering while the power source of the vehicle is turned off.

5. 5. The image processing device according to claim 4, wherein the image processing unit reads out the position information of the manhole where vehicle entry is prohibited from the storage unit and superimposes the position information on the overhead image when the vehicle is powered on.

6. 6. A moving body comprising the image processing device according to claim 1 and the imaging unit.

7. a receiving step for receiving an image of the surroundings of the vehicle acquired by the imaging unit; an overhead image generating step of generating an overhead image based on the image acquired in the receiving step; a no-boarding detection step for detecting a no-boarding manhole based on the overhead image; and generating an image in which the manhole marked as a no-vehicle zone included in the overhead image is highlighted.

8. A computer program for controlling each part of the image processing device according to any one of claims 1 to 5 or the moving body according to claim 6 by a computer.

Images