Ground object detection device, ground object detection method, and computer program product

Abstract

The present invention relates to a ground object detection device, a ground object detection method, and a computer program for ground object detection. The ground object detection device includes: a first detection unit (41) that detects one or more predetermined ground objects from a first image generated by a first camera unit (11) provided in a vehicle (2) and showing a first range around the vehicle (2); a second detection unit (42) that detects one or more predetermined ground objects from a second image generated by a second camera unit (12) provided in the vehicle (2) and showing a second range closer to the vehicle than the first range; and a switching determination unit (43) that determines whether the vehicle (2) is stopped, and when the vehicle (2) is stopped, the second detection unit (42) detects one or more predetermined ground objects, and on the other hand, when the vehicle (2) is moving, the first detection unit (41) detects one or more predetermined ground objects.

Description

Technical Field

[0001] This invention relates to a ground object detection device, a ground object detection method, and a computer program for detecting ground objects from images. Background Technology

[0002] A technique for detecting ground objects associated with the movement of a vehicle from images generated by a camera mounted on the vehicle, representing the area around the vehicle (see Japanese Patent Application Publication No. 2009-205403).

[0003] The road sign recognition device disclosed in Japanese Patent Application Publication No. 2009-205403 transforms images captured by a vehicle-mounted camera into images viewed from a vertical top-down perspective and concatenates them sequentially in a time series to generate a composite image. Furthermore, the device extracts feature values ​​from this composite image and compares them with a pre-prepared template of feature values ​​as a reference, thereby determining the category of road signs contained in the image. Additionally, the device detects the relative position of the vehicle with respect to the road signs. Summary of the Invention

[0004] Sometimes, obstacles exist on the road that conceal objects on the ground when viewed from vehicle-mounted cameras, such as those on parked vehicles. In locations with such obstacles, it can be difficult to detect the objects in the images generated by the vehicle-mounted cameras because they are not shown in the images. Consequently, data representing such objects may sometimes be unavailable.

[0005] Therefore, the object of the present invention is to provide a ground object detection device that detects ground objects from images generated by a camera mounted on a vehicle.

[0006] According to one embodiment, a ground object detection device is provided. This ground object detection device includes: a first detection unit that detects one or more predetermined ground objects from a first image generated by a first camera unit installed in a vehicle, representing a first range around the vehicle; a second detection unit that detects one or more predetermined ground objects from a second image generated by a second camera unit installed in the vehicle, representing a second range closer to the vehicle than the first range; and a switching determination unit that determines whether the vehicle is stopped based on at least one of the vehicle's speed, steering angle, and gear position; wherein when the vehicle is stopped, the second detection unit detects one or more predetermined ground objects, and on the other hand, the first detection unit detects one or more predetermined ground objects while the vehicle is in motion.

[0007] In this ground object detection device, it is preferable that the second detection unit performs viewpoint transformation on the second image to generate a bird's-eye view image, and detects one or more predetermined ground objects from the generated bird's-eye view image.

[0008] Furthermore, the above-ground object detection device preferably includes a position estimation unit. This position estimation unit estimates the positional relationship between the first above-ground object and the vehicle based on the position of the first above-ground object among one or more predetermined above-ground objects in each of two or more first images obtained at different times during vehicle travel, and the amount of vehicle movement between the acquisition times of the two or more first images. Based on the estimated positional relationship, it estimates the position of the first above-ground object in the actual space. Based on the positional relationship between the vehicle and the first above-ground object, it determines the position of the first above-ground object in the bird's-eye view. Based on the position of the first above-ground object in the bird's-eye view, the position of the first above-ground object in the bird's-eye view, the position of the other above-ground objects detected from the bird's-eye view among one or more predetermined above-ground objects in the bird's-eye view, and the position of the first above-ground object in the actual space, it estimates the positions of the other above-ground objects in the actual space.

[0009] In addition, the aboveground object detection device preferably also includes a data generation unit that generates data representing one or more predetermined aboveground objects detected.

[0010] According to another approach, a method for detecting objects on the ground is provided. This method includes the following actions: determining whether the vehicle is stopped based on at least one of the vehicle's speed, steering angle, and gear position; detecting one or more predetermined objects on the ground from a first image generated by a first camera mounted on the vehicle, representing a first range around the vehicle, during the period when the vehicle is moving; and detecting one or more predetermined objects on the ground from a second image generated by a second camera mounted on the vehicle, representing a second range closer to the vehicle than the first range, during the period when the vehicle is stopped.

[0011] According to another embodiment, a computer program for detecting ground objects is provided. This computer program is used to cause a computer to perform the following actions: determining whether a vehicle is stopped based on at least one of the vehicle's speed, steering angle, and gear position; detecting one or more predetermined ground objects from a first image generated by a first camera unit installed on the vehicle, representing a first range around the vehicle, during the period when the vehicle is moving; and detecting one or more predetermined ground objects from a second image generated by a second camera unit installed on the vehicle, representing a second range closer to the vehicle than the first range, during the period when the vehicle is stopped.

[0012] The ground object detection device of the present invention has the effect that it can generate data representing ground objects based on images generated by a camera mounted on a vehicle. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of a ground object data collection system equipped with ground object detection devices.

[0014] Figure 2 It is a schematic diagram of the vehicle's structure.

[0015] Figure 3 This is a hardware structure diagram of the data acquisition device.

[0016] Figure 4 This is a hardware structure diagram of a server used as an example of a ground object detection device.

[0017] Figure 5 This is a functional block diagram of the processor of the server related to the detection and processing of above-ground objects.

[0018] Figure 6 This is a schematic diagram illustrating an example of the relationship between ground objects detected by the first detection unit and ground objects detected by the second detection unit.

[0019] Figure 7 This is a flowchart of the action process for detecting and processing above-ground objects. Detailed Implementation

[0020] The following description, with reference to the accompanying drawings, describes the ground object detection device, the ground object detection method executed by the ground object detection device, and the computer program for ground object detection. This ground object detection device collects images of ground objects related to vehicle movement and information related to vehicle movement from one or more vehicles capable of communication within a predetermined area.

[0021] The vehicle sending images to the ground object detection device has multiple cameras with different shooting ranges, generating an image representing the area around the vehicle for each camera. Furthermore, for each camera, the vehicle sends a time-series of images generated by that camera, the generation time of each image, and information related to the vehicle's movement to the ground object detection device.

[0022] The ground object detection device detects the timing of a vehicle's parking based on at least one of the vehicle's speed, steering angle, and gear position, which are information related to the vehicle's movement. Then, outside of the vehicle's parking timing, the ground object detection device detects ground objects from images generated by cameras that capture a range relatively far from the vehicle. Conversely, during the vehicle's parking timing, the ground object detection device transforms images generated by cameras that capture a range relatively close to the vehicle into a bird's-eye view image, and detects ground objects from the transformed bird's-eye view image. Furthermore, based on the vehicle's position when two or more images of the same ground object are generated, the ground object detection device estimates the position of the ground object. Then, the ground object detection device generates data containing information indicating the type and position of each detected ground object, as ground object data. It should be noted that, hereinafter, the ground object data will be simply referred to as ground object data.

[0023] The ground objects that are to be inspected include various road signs, traffic lights, and other ground objects that are related to vehicle movement.

[0024] Figure 1 This is a schematic structural diagram of a ground object data collection system equipped with a ground object detection device. In this embodiment, the ground object data collection system 1 includes at least one vehicle 2 and a server 3, which is an example of a ground object detection device. A wireless base station 5 is connected to a communication network 4 connected to the server 3 via a gateway (not shown) or the like. Each vehicle 2 connects to the server 3 via the wireless base station 5 and the communication network 4, for example, by accessing the wireless base station 5. It should be noted that in Figure 1 For simplicity, only one vehicle 2 is shown in the diagram, but the ground data collection system 1 can also have multiple vehicles 2. Similarly, in Figure 1 The diagram shows only one wireless base station 5, but multiple wireless base stations 5 can be connected to the communication network 4.

[0025] Figure 2 This is a schematic diagram of vehicle 2. Vehicle 2 includes a camera 11, a panoramic camera 12, a GPS receiver 13, a wireless communication terminal 14, and a data acquisition device 15. The camera 11, panoramic camera 12, GPS receiver 13, wireless communication terminal 14, and data acquisition device 15 are connected via an in-vehicle network according to a standard such as a controller area network (CLAN) to communicate. Additionally, vehicle 2 may also include a navigation device (not shown) that searches for a predetermined driving route for vehicle 2 and navigates the vehicle 2 along that route.

[0026] Camera 11 is an example of a first imaging unit, comprising a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as CCD or C-MOS, and an imaging optical system that images the area (first range) to be photographed onto the two-dimensional detector. Camera 11 is mounted inside the vehicle compartment of vehicle 2, for example, facing forward. Camera 11 then photographs the area in front of vehicle 2 at a predetermined shooting cycle (e.g., 1 / 30 to 1 / 10 of a second), generating an image reflecting that area. It should be noted that camera 11 may also be mounted in vehicle 2 facing other directions, such as the rear of vehicle 2. The image generated by camera 11 is an example of a first image, which can be either a color image or a grayscale image.

[0027] Each time the camera 11 generates an image, it outputs the generated image to the data acquisition device 15 via the in-vehicle network.

[0028] The panoramic camera 12 is an example of a second camera unit. It includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as CCD or C-MOS, and an imaging optical system that images the area (second range) to be photographed onto the two-dimensional detector. Furthermore, the panoramic camera 12 is mounted on the vehicle 2 to photograph a range closer to the vehicle 2 than the camera 11. For example, the panoramic camera 12 is mounted obliquely downwards on the vehicle 2 such that its photographing range includes the area below and around the vehicle 2. Therefore, the photographing range of the panoramic camera 12 is narrower than that of the camera 11. In this embodiment, the focal length of the panoramic camera 12 is shorter than that of the camera 11, but it is not limited to this. Moreover, the panoramic camera 12 photographs the surrounding area of ​​the vehicle 2 at a predetermined shooting cycle (e.g., 1 / 30 second to 1 / 10 second), generating an image reflecting that surrounding area. It should be noted that the shooting cycle and shooting timing of the camera 11 can be independent of those of the panoramic camera 12. The image generated by the panoramic camera 12 is an example of the second image, which can be either a color image or a grayscale image.

[0029] It should be noted that multiple panoramic cameras 12 with different shooting directions can also be installed on vehicle 2. For example, one panoramic camera 12 can be installed at the front of vehicle 2 facing the lower front of vehicle 2, and another panoramic camera 12 can be installed at the rear of vehicle 2 facing the lower rear of vehicle 2. Alternatively, in order to capture the entire surroundings of vehicle 2, four panoramic cameras 12 can be installed at the front, rear, and left and right sides of vehicle 2.

[0030] Each time the panoramic camera 12 generates an image, it outputs the generated image to the data acquisition device 15 via the in-vehicle network. It should be noted that, in order to distinguish the image generated by the panoramic camera 12 from the image generated by the camera 11, the image generated by the panoramic camera 12 will sometimes be referred to as a panoramic image.

[0031] Each time the panoramic camera 12 generates a panoramic image, it outputs the generated panoramic image to the data acquisition device 15 via the in-vehicle network.

[0032] GPS receiver 13 receives GPS signals from GPS satellites at predetermined intervals and determines the vehicle 2's own position based on the received GPS signals. It should be noted that the predetermined interval for GPS receiver 13 to determine the vehicle 2's own position may differ from the shooting interval of camera 11 and panoramic camera 12. Furthermore, GPS receiver 13 outputs positioning information, representing the positioning result of the vehicle 2's own position based on GPS signals, to data acquisition device 15 via the in-vehicle network at predetermined intervals. It should be noted that vehicle 2 may also have a receiver that follows a satellite positioning system other than GPS receiver 13. In this case, it is sufficient for that receiver to determine the vehicle 2's own position.

[0033] The wireless communication terminal 14, an example of a communications unit, is a device that performs wireless communication processing following a predetermined wireless communication standard. For example, it connects to the server 3 via the wireless base station 5 and the communication network 4. The wireless communication terminal 14 then generates an uplink wireless signal containing image information or driving information received from the data acquisition device 15. The wireless communication terminal 14 then transmits this uplink wireless signal to the wireless base station 5, sending the image information and driving information to the server 3. Additionally, the wireless communication terminal 14 receives downlink wireless signals from the wireless base station 5 and transmits collection instructions from the server 3 contained in these signals to the data acquisition device 15 or an electronic control unit (ECU, not shown) that controls the driving of the vehicle 2.

[0034] Figure 3 This is a hardware structure diagram of the data acquisition device. The data acquisition device 15 generates image information including images generated by camera 11 and panoramic images generated by panoramic camera 12. Furthermore, the data acquisition device 15 generates driving information representing the driving behavior of vehicle 2. For this purpose, the data acquisition device 15 has a communication interface 21, a memory 22, and a processor 23.

[0035] Communication interface 21 is an example of an in-vehicle communication unit, and has an interface circuit for connecting the data acquisition device 15 to the in-vehicle network. Specifically, communication interface 21 is connected to camera 11, panoramic camera 12, GPS receiver 13, and wireless communication terminal 14 via the in-vehicle network. Furthermore, each time communication interface 21 receives an image from camera 11, it transmits the received image to processor 23. Similarly, each time communication interface 21 receives a panoramic image from panoramic camera 12, it transmits the received panoramic image to processor 23. Additionally, each time communication interface 21 receives location information from GPS receiver 13, it transmits the received location information to processor 23. Furthermore, communication interface 21 transmits a collection instruction for image information received from server 3 via wireless communication terminal 14 to processor 23. Finally, communication interface 21 outputs the image information and driving information received from processor 23 to wireless communication terminal 14 via the in-vehicle network.

[0036] The memory 22 may include, for example, volatile semiconductor memory and non-volatile semiconductor memory. The memory 22 may also include other storage devices such as a hard disk drive. Furthermore, the memory 22 stores various data used in the processing associated with image collection executed by the processor 23 of the data acquisition device 15. Such data includes, for example, identification information of the vehicle 2, parameters of the camera 11 such as its setting height, shooting direction, and field of view. Additionally, the memory 22 may store images received from the camera 11, panoramic images received from the panoramic camera 12, and positioning information received from the GPS receiver 13 for a certain period. Furthermore, the memory 22 stores information indicating the area designated by the ground object data collection instruction as the object of ground object data generation and collection (hereinafter, sometimes referred to as the collection object area). Additionally, the memory 22 may also store computer programs for implementing the various processes executed by the processor 23.

[0037] The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuitry. The processor 23 may also include other arithmetic circuitry such as logic units, numerical processing units, or graphics processing units. The processor 23 then stores the images received from the camera 11, the panoramic images received from the panoramic camera 12, the positioning information received from the GPS receiver 13, and the information representing the behavior of the vehicle 2 received from the ECU in the memory 22. Furthermore, during the operation of the vehicle 2, the processor 23 executes processing associated with image acquisition at predetermined intervals (e.g., 0.1 seconds to 10 seconds).

[0038] As part of the image collection process, the processor 23 determines, for example, whether the vehicle 2's location, as indicated by the positioning information received from the GPS receiver 13, is contained within the target collection area. Then, if the vehicle's location is contained within the target collection area, the processor 23 generates image information that includes the image received from the camera 11 and the panoramic image received from the panoramic camera 12.

[0039] The processor 23 includes in the image information the image generated by the camera 11, the time when the image was generated, the direction of travel of the vehicle 2 at that time, and parameters of the camera 11 such as its set height, shooting direction, and field of view. Similarly, the processor 23 includes in the image information a panoramic image, the time when the panoramic image was generated, the direction of travel of the vehicle 2 at that time, and parameters of the panoramic camera 12 such as its set height, shooting direction, and field of view. It should be noted that the processor 23 can obtain information indicating the direction of travel of the vehicle 2 from the ECU of the vehicle 2. Then, each time image information is generated, the processor 23 sends the generated image information to the server 3 via the wireless communication terminal 14. It should be noted that the processor 23 may also include multiple images or multiple panoramic images, the generation time of each image or panoramic image, and the direction of travel of the vehicle 2 in a single image information. Alternatively, the processor 23 may send the parameters of the camera 11 and the panoramic camera 12 separately from the image information to the server 3 via the wireless communication terminal 14.

[0040] Furthermore, the processor 23 generates vehicle 2's driving information after a predetermined time (e.g., the time when the ignition switch of vehicle 2 is turned on), and sends this driving information to the server 3 via the wireless communication terminal 14. The processor 23 includes in the driving information a series of positioning information obtained after the predetermined time, the time when each positioning information was obtained, and information representing the behavior of vehicle 2 for dead reckoning, such as wheel speed, acceleration, and angular velocity obtained from the ECU. Furthermore, the processor 23 may also include vehicle 2's identification information in the driving information and image information.

[0041] Next, server 3 will be described as an example of a ground object detection device.

[0042] Figure 4 This is a hardware structure diagram of server 3, an example of a ground object detection device. Server 3 has a communication interface 31, a storage device 32, a memory 33, and a processor 34. The communication interface 31, storage device 32, and memory 33 are connected to the processor 34 via signal lines. Server 3 may also have input devices such as a keyboard and mouse, and a display device such as an LCD screen.

[0043] Communication interface 31 is an example of a communication unit, and has interface circuitry for connecting server 3 to communication network 4. Furthermore, communication interface 31 is configured to communicate with vehicle 2 via communication network 4 and wireless base station 5. Specifically, communication interface 31 transmits image information and driving information received from vehicle 2 via wireless base station 5 and communication network 4 to processor 34. Additionally, communication interface 31 transmits collection instructions received from processor 34 to vehicle 2 via communication network 4 and wireless base station 5.

[0044] Storage device 32 is an example of a storage unit, such as having a hard disk device or an optical recording medium and its access device. Storage device 32 stores various data and information used in the ground object detection process. For example, storage device 32 stores parameter sets for determining the identifier used to detect ground objects from various images, and identification information for each vehicle 2. Furthermore, storage device 32 stores image information and driving information received from each vehicle 2. Additionally, storage device 32 may also store a computer program executed on processor 34 for performing ground object detection processing.

[0045] Memory 33 is another example of a storage unit, such as a non-volatile semiconductor memory and a volatile semiconductor memory. Furthermore, memory 33 temporarily stores various data generated during the execution of ground object detection processing.

[0046] Processor 34 is an example of a control unit, having one or more CPUs (Central Processing Units) and their peripheral circuitry. Processor 34 may also have other arithmetic circuitry such as logic units, numerical processing units, or graphics processing units. Furthermore, processor 34 performs surface object detection processing.

[0047] Figure 5 This is a functional block diagram of a processor 34 associated with ground object detection and processing. The processor 34 includes a first detection unit 41, a second detection unit 42, a switching determination unit 43, a position estimation unit 44, and a data generation unit 45. These units of the processor 34 are, for example, functional modules implemented by a computer program operating on the processor 34. Alternatively, these units of the processor 34 may also be dedicated arithmetic circuits provided in the processor 34.

[0048] The first detection unit 41 detects ground objects from a series of images generated by the camera 11 in a time sequence, in the interval where the switching determination unit 43 indicates the detection of ground objects, i.e., in the images generated during the movement of the vehicle 2.

[0049] The first detection unit 41, referring to the generation time of each image contained in the image information received from the vehicle 2, determines the image generated by the camera 11 within the interval where ground object detection is indicated by the switching determination unit 43. Then, the first detection unit 41 detects the ground objects represented by the input image (hereinafter, sometimes simply referred to as the input image) by inputting the determined images into a pre-learned recognizer designed to detect ground objects as detection targets. As such a recognizer, the first detection unit 41 can use a deep neural network (DNN) pre-learned to detect ground objects represented by the input image. For example, a DNN with a convolutional neural network (CNN) type architecture, such as Single Shot MultiBox Detector (SSD) or Faster R-CNN, can be used. Alternatively, a DNN with a self-attention network (SAN) type architecture, such as Vision Transformer, can also be used.

[0050] In this case, the recognizer calculates a confidence level representing the probability that a ground object (e.g., traffic lights, lane markings, pedestrian crossings, temporary stop lines, etc.) is present in various regions of the input image for each type of ground object that is the target of detection. The recognizer determines that a ground object of any type is present in a region where the confidence level for any ground object is above a predetermined detection threshold. Then, the recognizer outputs information representing the region in the input image that contains the ground object that is the target of detection (e.g., the bounding rectangle of the ground object that is the target of detection, hereinafter referred to as the object region), and information representing the type of ground object represented in the object region.

[0051] It should be noted that the first detection unit 41 differs from the second detection unit 42, which will be described in detail later, in that it does not convert the image generated by the camera 11 into a bird's-eye view image, but instead uses the image itself as the input image to the recognizer. This is because the camera 11 can capture a position farther away from the vehicle 2 than the panoramic camera 12. Therefore, even if the image is converted into a bird's-eye view image, ground objects that are more than a certain distance away from the vehicle 2 will be represented in a distorted manner in the bird's-eye view image, and no improvement in recognition accuracy can be expected.

[0052] The first detection unit 41 notifies the position estimation unit 44 and the data generation unit 45 of the generation time of each image, as well as information on the object area in each image that contains the detected ground object and the type of the detected ground object.

[0053] The second detection unit 42 detects ground objects from a series of panoramic images generated by the panoramic camera 12 over a time series, within the intervals where ground object detection is indicated by the switching determination unit 43. That is, the second detection unit 42 detects ground objects from each panoramic image generated during the interval when the vehicle 2 is parked.

[0054] The second detection unit 42, referring to the generation time of each panoramic image contained in the image information received from the vehicle 2, determines the panoramic image generated by the panoramic camera 12 within the interval where ground object detection is indicated by the switching determination unit 43. Then, the second detection unit 42 performs viewpoint transformation on each determined panoramic image to generate a bird's-eye view image, and detects ground objects from each generated bird's-eye view image. In this way, the second detection unit 42 detects ground objects from a series of panoramic images generated by the panoramic camera 12, which is set to shoot from a different orientation than the camera 11. Therefore, the second detection unit 42 can also detect ground objects not shown in the images obtained by the camera 11, such as ground objects obscured by obstacles on the road surface like parked vehicles. Furthermore, in the bird's-eye view image, ground objects represented on the road surface, especially road signs, are represented with minimal distortion. Therefore, by detecting ground objects from the bird's-eye view image after transforming the panoramic image, the second detection unit 42 can detect ground objects with good accuracy.

[0055] The second detection unit 42 performs viewpoint transformation processing on each panoramic image, using parameters such as the setting position, shooting direction, and focal length of the panoramic camera 12, from a predetermined height to a virtual viewpoint facing vertically downwards. As a result, the second detection unit 42 generates a bird's-eye view image corresponding to each panoramic image. It should be noted that if the vehicle 2 is equipped with multiple panoramic cameras 12, a single bird's-eye view image can also be generated by separately performing viewpoint transformation on each panoramic image generated by each panoramic camera 12 at the same timing and then combining them. Furthermore, the second detection unit 42 inputs each bird's-eye view image into a pre-learned recognizer that detects ground objects as the detection target, thereby detecting the ground objects represented in the input image. The second detection unit 42 uses the same recognizer as the one used by the first detection unit 41. Alternatively, the second detection unit 42 may use a different recognizer than the one used by the first detection unit 41. In this case, the recognizer used by the second detection unit 42 can be pre-learned using the bird's-eye view image as the input image. In addition, the recognizer used by the second detection unit 42 can also be set as a DNN with a CNN-type or SAN-type architecture.

[0056] According to a variation, the recognizer used by the second detection unit 42 can also be pre-learned by using the panoramic image itself as the input image. In this case, the second detection unit 42 may also detect ground objects from each panoramic image by inputting each panoramic image into the recognizer, without performing viewpoint transformation processing. According to this variation, the computational load is reduced because the viewpoint transformation processing is omitted.

[0057] The second detection unit 42 notifies the location estimation unit 44 and the data generation unit 45 of the generation time of each panoramic image, as well as information representing the object area containing the detected ground objects in each panoramic image and the type of the detected ground objects.

[0058] The switching determination unit 43 determines whether the vehicle 2 has stopped in a certain section based on information representing the behavior of the vehicle 2 contained in the driving information, particularly at least one of the vehicle 2's speed, steering angle, and gear position at various times while driving in the predetermined area. Furthermore, the switching determination unit 43 causes the second detection unit to detect objects on the ground in the section where the vehicle 2 is stopped. On the other hand, the switching determination unit 43 causes the first detection unit 41 to detect objects on the ground in the section where the vehicle 2 is traveling.

[0059] For example, the switching determination unit 43 determines the interval where the vehicle 2's speed is below a predetermined speed threshold as the interval during which the vehicle 2 is parked. Furthermore, generally speaking, the change in the vehicle 2's direction of travel becomes more significant when parking. Therefore, the switching determination unit 43 can also determine the interval during which the vehicle 2 is parked from when the vehicle 2's speed is below the predetermined speed threshold and the steering wheel angle of the vehicle 2 is above a predetermined angle until the vehicle 2's speed exceeds the predetermined speed threshold. Alternatively, sometimes the vehicle 2's direction of travel is switched to reverse when parking. Therefore, the switching determination unit 43 can also determine the interval during which the vehicle 2 is parked from when the vehicle 2 is in reverse gear until the gear shifts to forward gear.

[0060] The switching determination unit 43 notifies the first detection unit 41 of the remaining sections of the travel range of vehicle 2 within the predetermined area, excluding the section where vehicle 2 is parked, and instructs the first detection unit 41 to detect ground objects in the notified sections. Additionally, the switching determination unit 43 notifies the second detection unit 42 of the section where vehicle 2 is parked and instructs the second detection unit 42 to detect ground objects in the notified sections. Furthermore, the switching determination unit 43 notifies the position estimation unit 44 and the data generation unit 45 of the section where vehicle 2 is parked.

[0061] The location estimation unit 44 estimates the location of ground objects detected from any image or any panoramic image.

[0062] In this embodiment, the position estimation unit 44 estimates the position of the vehicle 2 at the time of image generation and the relative positions of detected ground objects with respect to these positions for the section where the vehicle 2 is traveling, according to the so-called Structure from Motion (SfM) method. That is, the position estimation unit 44 estimates the positional relationship between the ground objects of interest and the vehicle 2 based on the positions of the ground objects of interest in two or more images obtained at different time points during the travel of the vehicle 2, and the amount of movement of the vehicle 2 between the generation time points of each image. Then, based on the estimated positional relationship, the position estimation unit 44 estimates the position of the ground objects of interest in actual space.

[0063] Here, the position of each pixel in the image corresponds one-to-one with the orientation from the camera 11 to the object represented by that pixel. Therefore, the position estimation unit 44 can estimate the relative positional relationship between the ground object and the vehicle 2 based on the orientation from the camera 11 corresponding to the feature points representing the ground objects in each image, the amount of movement of the vehicle 2 between the generation times of each image, the direction of travel of the vehicle 2, and the parameters of the camera 11.

[0064] The position estimation unit 44 can set the reference position of vehicle 2 as the vehicle's own position represented by the positioning information obtained by GPS receiver 13 at a predetermined time, which is included in the driving information. Then, the position estimation unit 44 uses information for dead reckoning, such as wheel speed, acceleration, and angular velocity included in the driving information, to estimate the amount of movement of vehicle 2 between the time of image generation.

[0065] During the movement of vehicle 2, the position estimation unit 44 establishes correspondences between identical ground objects detected in multiple images obtained at different times. At this time, the position estimation unit 44 can establish correspondences between feature points contained in object regions representing the same ground object of interest across multiple images, for example, by using an optical flow tracking method. Furthermore, the position estimation unit 44 can estimate the relative position of the ground object of interest with respect to the position of vehicle 2 at the time each image was generated, as well as the position of vehicle 2 at the time each image was generated, through triangulation. In this triangulation, the position estimation unit 44 utilizes the vehicle 2's direction of travel at the time each image was generated, the vehicle 2's position at any given time of image generation, the amount of vehicle 2's movement between the time each image was generated, the parameters of camera 11, and the positions of corresponding feature points in each image.

[0066] The position estimation unit 44 repeatedly performs the above-described processing on the detected ground objects to sequentially estimate the position of the vehicle 2 at the time of each image generation and the relative positions of the ground objects surrounding the vehicle 2 with respect to these positions. Then, based on the estimated relative positions of the ground objects with respect to the vehicle 2 and the reference position of the vehicle 2, the position estimation unit 44 estimates the position of the ground object in the actual space. It should be noted that if the detected ground objects include ground objects with known positions, the position estimation unit 44 may also use the positions of the known ground objects instead of the reference position of the vehicle 2 to estimate the positions of each detected ground object in the actual space.

[0067] Furthermore, the location estimation unit 44 estimates the location of ground objects detected from each bird's-eye view image according to the steps described below. It should be noted that the location estimation unit 44 can perform the same processing on each bird's-eye view image; therefore, the processing for a single bird's-eye view image will be described below.

[0068] The position estimation unit 44 selects ground objects (hereinafter referred to as the first ground object) detected from any image generated by the camera 11 during a predetermined period before switching to ground object detection from the panoramic image. Then, based on the positional relationship between the first ground object and the vehicle 2, and the installation position of the panoramic camera 12, the position estimation unit 44 determines the relative position of the first ground object with respect to the panoramic camera 12 at the time the panoramic image of interest is generated. If the time at which the image detecting the first ground object is generated differs from the time at which the panoramic image of interest is generated, the position estimation unit 44 calculates the amount of movement of the vehicle 2 from the time the image detecting the first ground object is generated to the time at which the panoramic image of interest is generated, referring to driving information. Then, the position estimation unit 44 considers this amount of movement and determines the relative position of the first ground object with respect to the panoramic camera 12. Furthermore, the position estimation unit 44 determines the position of the first ground object on the bird's-eye view image based on the parameters of the panoramic camera 12, the relative position of the first ground object with respect to the panoramic camera 12, and the virtual viewpoint used in viewpoint transformation. Furthermore, the position estimation unit 44 determines the relative position of the second ground object, based on the positional relationship between the first ground object and other ground objects detected from the bird's-eye view (hereinafter referred to as the second ground object) on the bird's-eye view image and the virtual viewpoint used for viewpoint transformation, with the first ground object in actual space as the reference. Then, the position estimation unit 44 estimates the position of the second ground object in actual space based on the position of the first ground object in actual space and the relative position of the second ground object with respect to the first ground object.

[0069] It should be noted that in the modified example where ground objects are detected from the panoramic image when vehicle 2 is parked, the position estimation unit 44 performs the same processing as described above on the second ground object detected from the panoramic image instead of the bird's-eye view, and estimates the position of the second ground object in the actual space. In this case, the position estimation unit 44 uses the parameters of the panoramic camera 12 instead of the parameters of the virtual camera used when generating the bird's-eye view image, thereby determining the relative position of the second ground object in the actual space with the first ground object as a reference.

[0070] The location estimation unit 44 notifies the data generation unit 45 of the actual spatial location of objects in each location.

[0071] The data generation unit 45 generates ground object data representing ground objects detected by the first detection unit 41 and ground objects detected by the second detection unit 42. Specifically, for each ground object detected by the first detection unit 41, the data generation unit 45 includes the type of the ground object notified by the first detection unit 41 and the location of the ground object in actual space estimated by the location estimation unit 44 in the ground object data. Similarly, for each ground object detected by the second detection unit 42, the data generation unit 45 includes the type of ground object notified by the second detection unit 42 and the location of the ground object in actual space estimated by the location estimation unit 44 in the ground object data.

[0072] The generated surface feature data is used for map generation or updating. That is, regarding the surface features represented by the surface feature data, it is sufficient to write the type and location of the surface feature into the map that is being generated or updated.

[0073] Figure 6 This is a schematic diagram illustrating an example of the relationship between ground objects detected by the first detection unit 41 and ground objects detected by the second detection unit 42 in this embodiment. Figure 6In this scenario, when vehicle 2 is at position P1, it is in motion, and the first detection unit 41 detects ground objects from the image generated by camera 11. Therefore, lane markings and other objects present within the camera's field of view 601 are detected as ground objects. Conversely, at position P2, when vehicle 2 is about to stop, the second detection unit 42 detects ground objects from the bird's-eye view image obtained by viewpoint transformation of the panoramic image generated by panoramic camera 12. Therefore, lane markings and other objects present within the range 602 represented by the bird's-eye view image are detected as ground objects. In particular, in this example, the field of view 601 partially overlaps with the range 602 represented by the bird's-eye view image; therefore, road signs 611 located in this overlapping area are detected by both the first detection unit 41 and the second detection unit 42. Therefore, the position of the road sign 611, whose actual location is determined using the SfM method, is estimated on the bird's-eye view image. Furthermore, starting from position P1, a portion 612 of the lane dividing line on the road end is obscured by the parked vehicle 621 and therefore is not reflected in the image generated by camera 11. Thus, the first detection unit 41 cannot detect the portion 612 of the lane dividing line from the image generated by camera 11. However, the area 602 represented by the bird's-eye view image generated at position P2 includes this portion 612. Therefore, the second detection unit 42 is able to detect this portion 612 of the lane dividing line. Furthermore, based on the position of the road sign 611 in actual space and the relative positional relationship between the road sign 611 and the portion 612 of the lane dividing line in the bird's-eye view image, the position of the portion 612 of the lane dividing line in actual space is also inferred.

[0074] Figure 7 This is a flowchart of the action flow for ground object detection processing in server 3. When the processor 34 of server 3 receives image information and driving information from vehicle 2 in a predetermined interval within the collection object area, it performs ground object detection processing according to the following flowchart.

[0075] The switching determination unit 43 of the processor 34 determines the interval in which the vehicle 2 is traveling and the interval in which the vehicle 2 is parked within a predetermined interval (step S101). Then, the switching determination unit 43 instructs the first detection unit 41 to detect objects on the ground for the interval in which the vehicle 2 is traveling, and instructs the second detection unit 42 to detect objects on the ground for the interval in which the vehicle 2 is parked.

[0076] The first detection unit 41 of the processor 34 detects ground objects from a series of images generated by the camera 11 during the travel of the vehicle 2, and estimates the type of the ground object (step S102). Additionally, the second detection unit 42 of the processor 34 transforms each panoramic image generated by the panoramic camera 12 during the travel of the vehicle 2 into a bird's-eye view image, detects ground objects from each bird's-eye view image, and estimates the type of the ground object (step S103). It should be noted that, as described above, the second detection unit 42 can also directly detect ground objects from each panoramic image generated by the panoramic camera 12 during the travel of the vehicle 2.

[0077] The position estimation unit 44 of the processor 34 estimates the position of each ground object detected from each image generated by the camera 11 in the actual space (step S104). Then, the position estimation unit 44 determines the position of the first ground object detected from any image generated by the camera 11 in the bird's-eye view image (step S105). Next, based on the determined position, the position estimation unit 44 estimates the positions of other ground objects in the actual space, the position relationship between the determined position and the positions of other ground objects detected from the bird's-eye view image in the bird's-eye view image, and the position of the first ground object in the actual space (step S106).

[0078] The data generation unit 45 of the processor 34 generates ground object data representing the detected ground objects by including the types of ground objects and their actual locations in space in the ground object data (step S107). Then, the processor 34 ends the ground object detection process.

[0079] As explained above, while the vehicle is in motion, the ground object detection device detects ground objects from a series of time-series images generated by a first camera unit with a relatively wide shooting range that can capture positions relatively far from the vehicle. Therefore, the likelihood of the same ground object appearing in two or more images generated at different locations increases, resulting in the ground object detection device being able to accurately estimate the position of the ground object. On the other hand, when the vehicle is parked, the ground object detection device detects ground objects from a second image generated by a second camera unit that captures images relatively close to the vehicle. Therefore, the ground object detection device can also detect ground objects from the second image that are not visible from the first camera unit due to obstacles, etc. Furthermore, based on the position of the ground object detected from the first image in actual space, the ground object detection device estimates the position of the ground object detected from the second image in actual space. Therefore, the ground object detection device can also accurately estimate the position of ground objects detected from the second image generated when the vehicle is parked, where the position estimation based on SfM is insufficient due to low speed or increased steering angle. Therefore, the aboveground object detection device can appropriately generate aboveground object data representing the detected aboveground objects.

[0080] The processing of each part of the processor 34 of the server 3 based on the above-described embodiments or variations can also be performed by the processor 23 of the data acquisition device 15 mounted on the vehicle 2. In this case, the data acquisition device 15 is another example of a ground object detection device. Moreover, the data acquisition device 15 can also send the generated ground object data to the server 3 via the wireless communication terminal 14. In this case, the switching determination unit 43 can also determine in real time whether the vehicle 2 is parked based on information indicating the behavior of the vehicle 2 obtained from the ECU. Then, based on the determination result, the first detection unit 41 or the second detection unit 42 can also sequentially detect ground objects from any image in the latest image generated by the camera 11 or the latest panoramic image generated by the panoramic camera 12. In addition, the processor 23 of the data acquisition device 15 can also perform part of the processing of each part of the processor 34 of the server 3. For example, the processor 23 can also convert a series of panoramic images of the time series generated by the panoramic camera 12 into bird's-eye view images, and send each bird's-eye view image to the server 3 in the image information. Alternatively, the processor 23 may execute the processing of the switching determination unit 43, and for the section where the switching determination unit 43 determines that the vehicle 2 is traveling, only include the image generated by the camera 11 in the image information. In addition, for the section where the vehicle 2 is parked, the processor 23 may only include the panoramic image generated by the panoramic camera 12 or the bird's-eye view image generated from the panoramic image in the image information.

[0081] Furthermore, the detected ground features can also be used for purposes other than map generation and updating. For example, the types and locations of the detected ground features can also be used in the ECU of vehicle 2 for vehicle 2 control. In this case, the data generation unit can be omitted.

[0082] The computer program that enables the computer to implement the functions of the processor of the above-described embodiments or variations of the above-described object detection device can also be provided in the form of a computer-readable recording medium. It should be noted that the computer-readable recording medium can be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0083] As described above, those skilled in the art can make various modifications according to the embodiments within the scope of this invention.

Claims

1. A device for detecting above-ground objects, wherein, The above-ground object detection device has the following features: The first detection unit detects one or more predetermined ground objects from a first image generated by a first camera unit installed on the vehicle, which represents a first range around the vehicle. The second detection unit performs viewpoint transformation on a second image generated by a second camera unit installed on the vehicle, which represents a second range that is closer to the vehicle than the first range, to generate a bird's-eye view image, and detects the one or more predetermined ground objects from the generated bird's-eye view image; The switching determination unit determines whether the vehicle is stopped based on at least one of the vehicle's speed, steering angle, and gear position. When the vehicle is stopped, the second detection unit detects one or more predetermined ground objects. On the other hand, when the vehicle is in motion, the first detection unit detects one or more predetermined ground objects. as well as Location estimation department, The position estimation unit estimates the positional relationship between the first ground object and the vehicle based on the position of the first ground object among the one or more predetermined ground objects in each of two or more first images obtained at different times during the vehicle's movement, and the amount of vehicle movement between the acquisition times of the two or more first images. Based on the estimated positional relationship, it estimates the position of the first ground object in actual space. The position estimation unit determines the position of the first ground object on the bird's-eye view image based on the positional relationship between the vehicle and the first ground object. Based on the position of the first ground object on the bird's-eye view image, the positional relationship between the position of the first ground object on the bird's-eye view image and the positions of other ground objects detected from the bird's-eye view image among the more than one predetermined ground objects, and the position of the first ground object in the actual space, it estimates the positions of the other ground objects in the actual space.

2. The above-ground object detection device according to claim 1, wherein, The aboveground object detection device also includes a data generation unit that generates data representing the detected one or more predetermined aboveground objects.

3. A method for detecting aboveground features, wherein, The above-ground object detection method includes the following actions: Determine whether the vehicle is stopped based on at least one of the vehicle's speed, steering angle, and gear position. Within the area where the vehicle is traveling, one or more predetermined ground objects are detected from a first image generated by a first camera unit installed on the vehicle, representing a first range around the vehicle. In the area where the vehicle is parked, a bird's-eye view image is generated by a viewpoint transformation of a second image generated by a second camera unit installed on the vehicle, representing a second range closer to the vehicle than the first range. One or more predetermined ground objects are detected from the generated bird's-eye view image. Based on the position of the first ground object in each of the two or more first images obtained at different times during the vehicle's movement, and the amount of vehicle movement between the acquisition times of the two or more first images, the positional relationship between the first ground object and the vehicle is estimated. Based on the estimated positional relationship, the position of the first ground object in actual space is estimated. Based on the positional relationship between the vehicle and the first ground object, the position of the first ground object on the bird's-eye view is determined. Based on the positional relationship between the position of the first ground object on the bird's-eye view and the positions of other ground objects detected from the bird's-eye view among the more than one predetermined ground objects on the bird's-eye view, as well as the position of the first ground object in the actual space, the positions of the other ground objects in the actual space are estimated.

4. A computer program product, comprising a computer program for detecting aboveground features, wherein, The computer program for detecting above-ground features is used to cause the computer to perform the following actions: Determine whether the vehicle is stopped based on at least one of the vehicle's speed, steering angle, and gear position. Within the area where the vehicle is traveling, one or more predetermined ground objects are detected from a first image generated by a first camera unit installed on the vehicle, representing a first range around the vehicle. In the area where the vehicle is parked, a bird's-eye view image is generated by a viewpoint transformation on a second image generated by a second camera unit installed on the vehicle, which represents a second range closer to the vehicle than the first range. One or more predetermined ground objects are detected from the generated bird's-eye view image. Based on the position of the first ground object in each of the two or more first images obtained at different times during the vehicle's movement, and the amount of vehicle movement between the acquisition times of the two or more first images, the positional relationship between the first ground object and the vehicle is estimated. Based on the estimated positional relationship, the position of the first ground object in actual space is estimated. Based on the positional relationship between the vehicle and the first ground object, the position of the first ground object on the bird's-eye view is determined. Based on the positional relationship between the position of the first ground object on the bird's-eye view and the positions of other ground objects detected from the bird's-eye view among the more than one predetermined ground objects on the bird's-eye view, as well as the position of the first ground object in the actual space, the positions of the other ground objects in the actual space are estimated.

Images