Image recognition system for vehicle peripheral image

Abstract

In the present invention, a vehicle includes: an on-vehicle imaging device 3 for capturing a vehicle peripheral image; a vehicle-side image recognition part 17 for identifying a road-surface state and an object on a road from the vehicle peripheral image and estimating reliability of a result of the identification; and a vehicle-side communication device 12. A server includes: a server-side image recognition part 31 for identifying a road-surface state and an object on a road from a vehicle peripheral image with higher accuracy than that of the vehicle-side image recognition part 17; and a server-side communication device 13. When reliability is high, the vehicle-side image recognition part 17 uses the road-surface state and the object on the road identified by the vehicle-side image recognition part 17 as a final identification result and, when reliability is low, causes the road-surface state and the object on the road to be identified from the vehicle peripheral image in the server-side image recognition part 31, and sets the identification result as the final identification result of the road-surface state or the object on the road.

Description

Image recognition system for vehicle surroundings images

[0001] The present invention relates to a technology for recognizing images of the surroundings of a vehicle captured by an in-vehicle imaging device.

[0002] 2. Description of the Related Art In recent years, image recognition systems have been developed that detect objects on a road and estimate the state of a road surface from images captured by an in-vehicle imaging device such as an in-vehicle camera.

[0003] For example, in Patent Document 1, an image recognition system is installed in a vehicle, which identifies images captured by an on-board imaging device and uploads information such as the latitude and longitude of the image capture location, as well as information on road objects and road surface conditions, to a server. This information is used in services such as providing route search results that avoid specific road surfaces, such as muddy roads and speed bumps, when others search for routes using a navigation system or the like.

[0004] Also, a system has been devised in which an image captured by an in-vehicle imaging device is transmitted to a server, and the image is recognized by an image recognition system provided in the server.

[0005] JP 2018-206108 A

[0006] When image recognition is performed using an image recognition system installed in a vehicle, as in Patent Document 1, there is a problem in that it is difficult to achieve highly accurate image recognition because the processing speed is slower and the memory is smaller than that of an image recognition system installed in a server.

[0007] On the other hand, installing an image recognition system on a server poses problems such as long delays when transferring images from vehicles to the server. Also, installing an image recognition system on the server that can handle all vehicles on the road would result in enormous initial and maintenance costs.

[0008] The present invention has been made in consideration of such problems, and its purpose is to provide an image recognition system for images around a vehicle that is capable of highly accurate image recognition while reducing the processing power of image recognition on the server and keeping costs down.

[0009] In order to achieve the above-mentioned object, the image recognition system of the present invention comprises an imaging means mounted on a vehicle for imaging the surroundings of the vehicle, a first identification processing means mounted on the vehicle for identifying road surface conditions or road objects from the vehicle surroundings image captured by the imaging means and estimating the reliability of the identification results, a first communication means mounted on the vehicle for sending and receiving image information, a second communication means provided on a server for sending and receiving image information to and from the first communication means, and a second identification processing means provided on the server for identifying road surface conditions or road objects from the vehicle surroundings image with higher accuracy than the first identification processing means, wherein the first identification processing means causes the vehicle surroundings image to be transmitted from the first communication means to the second communication means based on the reliability, and causes the second identification processing means to identify the road surface conditions or road objects.

[0010] In the image recognition system of the present invention, the first identification processing means mounted on the vehicle can estimate the road surface condition or an object on the road from the vehicle surroundings image with good responsiveness.

[0011] Furthermore, when the reliability of the identification made by the first identification processing means is low, the second identification processing means provided on the server side can make the identification with high accuracy.

[0012] This enables highly responsive identification while ensuring the reliability of the identification results. By performing identification in the first identification processing means, the burden of identification on the second identification processing means can be reduced, and the processing power of the second identification processing means on the server side can be reduced, thereby reducing initial costs and maintenance costs.

[0013] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.

[0014] An embodiment of an image recognition system embodying the present invention will now be described.

[0015] FIG. 1 is a configuration diagram of an image recognition system 1 according to this embodiment.

[0016] The image recognition system 1 of the present invention is a system that recognizes road surface conditions around a vehicle and objects on the road from images captured by an on-board imaging device 3 such as an on-board camera mounted on the vehicle.

[0017] As shown in FIG. 1, the image recognition system 1 of this embodiment includes a vehicle-side system 10 mounted on a vehicle side, and a server-side system 11 provided on a server.

[0018] The vehicle-side system 10 and the server-side system 11 can transmit and receive information to and from each other via a vehicle-side communication device 12 and a server-side communication device 13 .

[0019] The vehicle-side system 10 has an in-vehicle imaging device 3 (imaging means), an image acquisition unit 15, an image processing unit 16, a vehicle-side image recognition unit 17 (first identification processing means), a communication processing unit 18, a GPS device 19, and a vehicle-side communication device 12 (first communication means).

[0020] The image acquisition unit 15, image processing unit 16, vehicle-side image recognition unit 17, and communication processing unit 18 are included in a control unit that includes an input / output device, a storage device (ROM, RAM, non-volatile RAM, etc.), a central processing unit (CPU), etc. Note that the image acquisition unit 15, image processing unit 16, vehicle-side image recognition unit 17, and communication processing unit 18 may be partially or entirely configured in separate control units.

[0021] The vehicle-mounted imaging device 3 captures images of the surroundings of the vehicle, particularly the road surface and the area above the road surface in front of the vehicle.

[0022] The image acquisition unit 15 receives an image of the surroundings of the vehicle captured by the in-vehicle imaging device 3 .

[0023] The image processing unit 16 processes the vehicle surroundings image input by the image acquisition unit 15 so that the road surface condition and objects on the road surface can be easily recognized. Note that the image processing unit 16 uses the CPU and memory installed in the vehicle, so it may be configured to perform relatively simple image processing.

[0024] The vehicle-side image recognition unit 17 includes an image classification unit 21 , an object detection unit 22 , a server reprocessing determination unit 23 , and a result transmission unit 24 .

[0025] The image classification unit 21 classifies the input image (the road surface condition in the vehicle surroundings image) using machine learning. A desirable example of the machine learning is a convolutional neural network (CNN). A first reliability R1, which is the reliability of each classification, is output from a learning device that has learned the road surface so that the sum total is 1. The first reliability R1 of the road surface information is output as the road becomes more difficult to recognize, with lower values ​​being output, such as 0.5 for asphalt, 0.3 for dirt, 0.2 for snow, and 0.0 for grass. In addition to asphalt, snow, and grass, road surface conditions include concrete, mud, ice, sand, gravel, rocks (monolithic road surfaces several meters to several tens of meters long that exist in rivers and mountains), stones (larger than gravel, such as natural round stones from rivers and man-made stone-paved road surfaces), cobblestones, bricks, iron plates (temporary surfaces installed during construction or in parking lots), and wooden boards (surfaces of wooden bridges, etc.).

[0026] The object detection unit 22 uses machine learning to output the location and type of road objects in the image from the input vehicle surroundings image. Then, a second reliability R2, which is a reliability for each road object detected, is output from a learning device that has been trained on object types. As the reliability R2 for road object types, for example, the learning device outputs lower values ​​as the object becomes more difficult to recognize, such as 0.4 for vehicles, 0.03 for bicycles, and 0.01 for humans. Note that, since the reliability is calculated for each detected object when detecting road objects, the total value for one vehicle surroundings image does not necessarily equal 1. In addition to vehicles, bicycles, and humans, road objects include, for example, motorcycles (motorcycles), animals, cones, balls, trash cans, trash, potholes, manholes, speed bumps, etc.

[0027] The server reprocessing determination unit 23 determines, based on the results of the image classification unit 21 and the object detection unit 22, whether or not image recognition for the image should be performed again on the server side (whether or not a reprocessing determination should be made by the server reprocessing determination unit 23).

[0028] If the server reprocessing determination unit 23 determines that reprocessing determination is necessary, the result transmission unit 24 outputs an instruction to that effect, the vehicle surroundings image, and the image recognition result (image information) from the vehicle-side image recognition unit 17 to the communication processing unit 18. If the server reprocessing determination unit 23 determines that reprocessing determination is not necessary, the result transmission unit 24 transmits the image recognition result from the vehicle-side image recognition unit 17 to the server side. Note that the vehicle surroundings image may be transmitted to the server side together with the image recognition result.

[0029] The GPS device 19 may be a car navigation system installed in a vehicle.

[0030] The communication processing unit 18 combines the output from the result transmission unit 24 (the vehicle surroundings image and the image recognition result) with the current vehicle position from the GPS device 19 and outputs the combined output from the vehicle-side communication device 12 .

[0031] The vehicle-side communication device 12 and the server-side communication device 13 may be 4G or 5G communication devices.

[0032] The server-side system 11 includes a server-side communication device 13 (second communication means), a communication processing unit 30 , a server-side image recognition unit 31 (second identification processing means), and a storage device 33 .

[0033] The server-side communication device 13 is a device capable of transmitting and receiving data to and from the vehicle-side communication device 12 mounted on the vehicle.

[0034] If the server reprocessing determination unit 23 determines that reprocessing determination is necessary, the communication processing unit 30 outputs the vehicle surroundings image and image classification result transmitted from the vehicle-side system 10 to the server-side image recognition unit 31. If it determines that reprocessing determination is not necessary, the communication processing unit 30 outputs the image and classification result to the storage device 33 of the server-side system 11, and stores the image recognition result.

[0035] The server-side image recognition unit 31 includes a super-resolution unit 40 , an image classification unit 41 , and an object detection unit 42 .

[0036] The super-resolution unit 40 increases the resolution of the input image by using a super-resolution technology (for example, GAN: Generative Adversarial Networks) that generates a higher-resolution image from the vehicle surroundings image captured by the in-vehicle imaging device 3, and outputs the resulting image to the image classification unit 41 and the object detection unit 42.

[0037] The image classification unit 41 classifies images using an algorithm with higher accuracy than the image classification unit 21 of the vehicle-side image recognition unit 17, and records the results in the storage device 33. If the recognition results are to be used on the vehicle side, they are transmitted to the vehicle via communication. The image classification unit 41 may perform machine learning using a CNN with more hidden layers than the image classification unit 21, for example.

[0038] The object detection unit 42 detects objects using a more accurate algorithm than the object detection unit 42 of the vehicle-side image recognition unit 17, and records the results in the storage device 33. When the recognition results are to be used on the vehicle side, the recognition results are transmitted to the vehicle from the server-side communication device 13. The object detection unit 42 may perform machine learning using a CNN with more hidden layers than the vehicle-side object detection unit 22, for example.

[0039] The storage device 33 stores information about the types of road objects or road surface information that may be similar, as will be described later.

[0040] FIG. 2 is a flowchart showing the procedure of the image recognition process in the image recognition system 1 of this embodiment.

[0041] In this system, the control of the vehicle-side system 10 (steps S10 to S80, S120) is repeatedly executed at predetermined time intervals when the vehicle power is on, and the control of the server-side system 11 (steps S90 to S110) is executed continuously.

[0042] First, in step S10, the image acquisition unit 15 acquires an image of the vehicle's surroundings from the in-vehicle image capture device 3. Then, the image processing unit 16 processes the acquired image of the vehicle's surroundings so that the road surface condition and objects on the road surface can be easily recognized. Then, the process proceeds to step S20.

[0043] In step S20, the object detection unit 22 of the vehicle-side image recognition unit 17 detects the location and type of an object (road object) in the image from the vehicle surroundings image processed in step S10. Furthermore, the image classification unit 21 executes classification of the vehicle surroundings image processed in step S10 (acquisition of road surface information). Note that for the location and type of road object and the road surface information (type of road surface), a reliability R is calculated for each road object and for the road surface information. Then, the process proceeds to step S30.

[0044] In step S30, it is determined whether the highest reliability Rmax1 of the multiple reliabilities R (R1 and all R2) calculated in step S20 is equal to or less than a threshold a. The threshold a may be set to a value close to the lower limit of the reliability when the image recognition accuracy in the vehicle-side system 10 is sufficient. If the highest reliability Rmax1 is equal to or less than the threshold a, the process proceeds to step S80. If the highest reliability Rmax1 is higher than the threshold a, the process proceeds to step S40.

[0045] In step S40, the difference ΔR between the highest reliability Rmax1 and the second highest reliability Rmax2 among the multiple reliabilities R calculated in step S20 is calculated, and it is determined whether the reliability difference ΔR is equal to or less than a threshold b. The threshold b may be set appropriately, for example, so as to eliminate failures that cause the reliability to always be a constant value. If the reliability difference ΔR is equal to or less than the threshold b, the process proceeds to step S80. If the reliability difference ΔR is greater than the threshold b, the process proceeds to step S50.

[0046] In step S50, it is determined whether the variance of the multiple reliabilities R calculated in step S20 is equal to or less than a threshold c. The threshold b may be set appropriately, for example, so as to eliminate failures that cause the variance of the multiple reliabilities R to always be a constant value. If the variance of the reliabilities R is equal to or less than the threshold c, the process proceeds to step S80. If the variance of the reliabilities R is greater than the threshold c, the process proceeds to step S60.

[0047] In step S60, the output results from the image classification unit 21 and the object detection unit 22, i.e., the type of road object and road surface information, are determined to determine whether two or more detection results previously defined as similar are included. If multiple potentially similar detection results are known, they can be pre-registered in a storage device. Examples of road objects potentially similar in detection results include potholes and manholes, potholes and puddles, potholes and road stains, speed bumps and curbs, and speed bumps and barrier bars. Examples of road surface conditions potentially similar in detection results include ice and wet asphalt, concrete and snow, bricks and asphalt of the same color, etc. If multiple potentially similar detection results are included, the process proceeds to step S80. If multiple potentially similar detection results are not included, the process proceeds to step S70.

[0048] In step S70, it is determined whether the detection result is a detection result that can be used (utilized) on the server side. If the data is to be used on the server side, the process proceeds to step S110. If the data is not to be used on the server side, the process proceeds to step S120.

[0049] In step S80, the vehicle surroundings image, the vehicle detection results, and a reprocessing instruction are transmitted to the server-side system 11. Then, the process proceeds to step S90.

[0050] In step S90, the super-resolution unit 40 of the server-side image recognition unit 31 in the server-side system 11 increases the resolution of the transmitted vehicle surroundings image. Then, the process proceeds to step S100.

[0051] In step S100, the object detection unit 42 of the server-side image recognition unit 31 detects the location and type of an object (road object) in the image from the vehicle surroundings image whose resolution has been increased in step S90. Also, the image classification unit 41 classifies the image (obtains road surface information) from the vehicle surroundings image whose resolution has been increased in step S90. Then, the process proceeds to step S110.

[0052] In step S110, the road object information and road surface information are stored in the server-side storage device 33 and are used for various subsequent services for the subject vehicle and other vehicles. Then, the present routine is returned.

[0053] In step S120, the road object and road surface information recognized by the vehicle-side image recognition unit 17 is used for various services on the vehicle side.

[0054] The various services for the vehicle may include, for example, a device that issues a warning to the driver in response to an object on the road, or the use of road object information and road surface information in various safety devices such as a traction control device, an anti-skid device, an anti-collision device, etc. Furthermore, the road object information and road surface information may be used when searching for a route in the navigation system of the vehicle.

[0055] As various services for other vehicles, for example, road surface information and road object information may be provided to a server of a navigation system, and provided to many users as information for route searches.

[0056] As described above, the image recognition system 1 of this embodiment has a vehicle-side image recognition unit 17 in the vehicle-side system 10 mounted on the vehicle side, and is capable of recognizing road surface conditions and objects on the road from images of the vehicle's surroundings captured by the on-board imaging device 3, and is also capable of transmitting the images of the vehicle's surroundings to the server-side system 11, allowing the server-side image recognition unit 31 of the server-side system 11 to recognize the road surface conditions and objects on the road from the images of the vehicle's surroundings with high accuracy.

[0057] The information on road surface conditions and road objects can be used for various services, such as route search in the navigation system installed in the vehicle or warning control while the vehicle is traveling, thereby improving the performance of these various services.

[0058] The vehicle-side system 10 identifies road surface conditions and road objects from the vehicle surroundings image, estimates the reliability R of the identification, and if the reliability R is high, sets the road surface conditions and road object information identified by the vehicle-side image recognition unit 17 as the final identification result. This enables highly responsive identification while ensuring the reliability of the identification result.

[0059] In addition, the server-side system 11 is equipped with a server-side image recognition unit 31 that is capable of more accurate recognition than the vehicle-side image recognition unit 17, and when the reliability R of the recognition result of the vehicle-side image recognition unit 17 is low, image recognition is performed in the server-side image recognition unit 31, allowing the road surface condition and objects on the road to be recognized with high accuracy.

[0060] This enables highly responsive identification while ensuring the reliability of the identification results. By performing identification in the vehicle-side image recognition unit 17, the load on the server-side image recognition unit 31 is reduced, and the processing capacity of the server-side image recognition unit 31 is reduced, thereby reducing initial costs and maintenance costs.

[0061] The vehicle-side image recognition unit 17 is equipped with an image classification unit 21 that identifies road surface conditions from images of the vehicle's surroundings and estimates the reliability of the identification results of the road surface conditions (first reliability R1), and an object detection unit 22 that identifies road objects from images of the vehicle's surroundings and estimates the reliability of each road object (second reliability R2).Based on the first reliability R1 and the second reliability R2, it is determined whether or not it is necessary to identify the road surface conditions and road objects in the server-side system 11.Therefore, the server-side image recognition unit 31 performs identification of the road surface conditions and each road object only as much as necessary, making it possible to achieve both improved identification reliability and reduced processing load on the server-side image recognition unit 31 at a high level.

[0062] In detail, the reliability of the identification can be improved by having the server-side image recognition unit 31 identify the object when at least one of the following applies: the highest reliability value among the reliability R1 of the road surface condition and the reliability R2 of each road object is equal to or less than a threshold a; the difference between the highest reliability Rmax1 and the second highest reliability Rmax2 among the multiple reliabilities R (R1, R2) is equal to or less than a threshold b; or the variance of each reliabilty R is equal to or less than a threshold c.

[0063] Furthermore, if a road object identified from an image of the vehicle's surroundings contains pre-set specific information, the server-side image recognition unit 31 can identify the object, thereby improving the identifiability of the specific information.

[0064] In more detail, multiple road objects are identified from the image of the vehicle's surroundings, and if multiple pre-set similar road objects are included, the server-side image recognition unit 31 can identify them, thereby improving the ability to identify similar road objects.

[0065] The present invention is not limited to the above-described embodiment, and can be modified within the scope of the invention.

[0066] For example, the reliability thresholds a, b, and c that determine whether or not the server-side image recognition unit 31 needs to identify an image may be changed based on the usage state, temperature state, and the like of the vehicle-side image recognition unit 17. In more detail, when the control unit including the vehicle-side image recognition unit 17 is processing another application or is in a high temperature state above a predetermined temperature, the thresholds a, b, and c may be changed so that the server-side image recognition unit 31 can more easily identify an image.

[0067] REFERENCE SIGNS LIST 1 Image recognition system 3 In-vehicle imaging device (imaging means) 10 Vehicle-side system 11 Server-side system 17 Vehicle-side image recognition unit (first identification processing means) 12 Vehicle-side communication device (first communication means) 13 Server-side communication device (second communication means) 31 Server-side image recognition unit (second identification processing means) 21 Image classification unit 22 Object detection unit

Claims

1. An image recognition system for vehicle surroundings images comprising: an imaging means mounted on a vehicle for capturing images of the vehicle's surroundings; a first identification processing means mounted on the vehicle for identifying road surface conditions or road objects from vehicle surroundings images captured by the imaging means and estimating the reliability of the identification results; a first communication means mounted on the vehicle for transmitting and receiving image information; a second communication means mounted on a server for transmitting and receiving image information to and from the first communication means; and a second identification processing means mounted on the server for identifying road surface conditions or road objects from the vehicle surroundings images with higher accuracy than the first identification processing means, wherein the first identification processing means causes the vehicle surroundings images to be transmitted from the first communication means to the second communication means based on the reliability, and causes the second identification processing means to identify the road surface conditions or road objects.

2. The image recognition system for vehicle surroundings images described in claim 1, characterized in that the first identification processing means comprises: an image classification unit that identifies road surface conditions from the vehicle surroundings image and estimates a first reliability of the identification result of the road surface conditions; and an object detection unit that identifies road objects from the vehicle surroundings image and estimates a second reliability for each of the road objects, and determines whether or not it is necessary to have the second identification processing means identify the road surface conditions and road objects based on the first reliability and the second reliability.

3. An image recognition system for vehicle surrounding images according to claim 1, characterized in that the results of identification made by the first identification processing means or the second identification processing means are used for route search in a navigation system installed in the vehicle or for warning control while the vehicle is traveling.

4. The image recognition system for vehicle surroundings images described in claim 1, characterized in that the first identification processing means causes the second identification processing means to identify the road object from the vehicle surroundings image when predetermined specific information is included in the vehicle surroundings image.

5. The image recognition system for vehicle surroundings images described in claim 1, characterized in that when the vehicle surroundings image contains multiple similar road objects that have been set in advance, the first identification processing means causes the second identification processing means to identify the road objects from the vehicle surroundings image.

6. The image recognition system for vehicle surroundings images described in claim 1, characterized in that the first identification processing means changes the reliability threshold for determining whether or not image identification is necessary in the second identification processing means based on the usage state or temperature state of the first identification processing means.

Images