Feature detection device, feature detection method, and feature detection computer program

Abstract

To provide a feature detection device which generates data representing a feature on the basis of an image generated by a camera loaded on a vehicle.SOLUTION: A feature detection device comprises: a first detection unit 41 which detects one or more predetermined features from a first image representing a first range around a vehicle 2 generated by a first imaging unit 11 provided on the vehicle 2; a second detection unit 42 which detects one or more predetermined features from a second image generated by a second imaging unit 12 provided on the vehicle 2 to represent a second range close to the vehicle than the first range; and a changeover determination unit 43 which determines whether or not the vehicle 2 is parked, causes the second detection unit 42 to detect the one or more predetermined features when the vehicle 2 is parked and on the other hand, causes the first detection unit 41 to detect the one or more predetermined features during travel of the vehicle 2.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a feature detection device, a feature detection method, and a feature detection computer program for detecting features from an image. [Background technology]

[0002] A technique has been proposed for detecting features related to the movement of a vehicle from an image showing the surroundings of the traveling vehicle, which is generated by a camera mounted on the vehicle (see Patent Document 1).

[0003] The road sign recognition device disclosed in Patent Document 1 converts images captured by an onboard camera into images viewed vertically down and connects them in chronological order to generate a composite image. Furthermore, this device extracts features of the composite image and compares them with a previously prepared template of reference features to determine the type of road sign contained in the image. This device then detects the relative position of the vehicle with respect to the road sign. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-205403 A Summary of the Invention [Problem to be solved by the invention]

[0005] There may be obstacles on the road, such as parked vehicles, that hide features from the view of the vehicle-mounted camera. In places where such obstacles exist, it may be difficult to detect the features from the images generated by the vehicle-mounted camera because the features to be detected are not shown in the images. As a result, it may not be possible to generate data representing such features.

[0006] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to provide a feature detection device that detects features from an image generated by a camera mounted on a vehicle. [Means for solving the problem]

[0007] According to one embodiment, there is provided a feature detection device, which includes a first detection unit that detects one or more predetermined features from a first image that is generated by a first imaging unit provided in the vehicle and represents a first range around the vehicle, a second detection unit that detects one or more predetermined features from a second image that is generated by a second imaging unit provided in the vehicle and represents a second range that is closer to the vehicle than the first range, and a switching determination unit that determines whether the vehicle is parked based on at least one of the vehicle speed, steering angle, and shift position, and causes the second detection unit to detect the one or more predetermined features when the vehicle is parked, and causes the first detection unit to detect the one or more predetermined features while the vehicle is traveling.

[0008] In this feature detection device, it is preferable that the second detection section performs viewpoint conversion on the second image to generate a bird's-eye image, and detects one or more predetermined features from the generated bird's-eye image.

[0009] Moreover, it is preferable that the feature detection device further includes a position estimation unit that estimates a positional relationship between the first feature and the vehicle based on the position of a first feature among one or more specified features in each of two or more first images obtained at different times while the vehicle is traveling and the amount of movement of the vehicle between the timing of acquisition of each of the two or more first images, estimates the position of the first feature in real space based on the estimated positional relationship, identifies the position of the first feature on the bird's-eye image based on the positional relationship between the vehicle and the first feature, and estimates the positions of the other features in real space based on the positional relationship between the position of the first feature on the bird's-eye image and the positions on the bird's-eye image of other features detected from the bird's-eye image of the one or more specified features and the position of the first feature in real space.

[0010] Moreover, the feature detection device preferably further comprises a data generation unit that generates data representative of the detected one or more predetermined features.

[0011] According to another aspect, there is provided a feature detection method, which includes: determining whether or not a vehicle is parked based on at least one of a speed, a steering angle, and a shift position of the vehicle, detecting one or more predetermined features from a first image representing a first range around the vehicle generated by a first imaging unit provided in the vehicle in a section where the vehicle is traveling, and detecting one or more predetermined features from a second image representing a second range closer to the vehicle than the first range generated by a second imaging unit provided in the vehicle in a section where the vehicle is parked.

[0012] According to yet another aspect, there is provided a computer program for detecting features, the computer program for detecting features including instructions for causing a computer to execute the following: determining whether or not a vehicle is parked based on at least one of a speed, a steering angle, and a shift position of the vehicle, detecting one or more predetermined features from a first image representing a first range around the vehicle generated by a first imaging unit provided in the vehicle in a section where the vehicle is traveling, and detecting one or more predetermined features from a second image representing a second range closer to the vehicle than the first range generated by a second imaging unit provided in the vehicle in a section where the vehicle is parked. Effect of the Invention

[0013] The feature detection device according to the present disclosure advantageously has the ability to generate data representing features based on an image generated by a camera mounted on a vehicle. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic configuration diagram of a feature data collection system in which a feature detection device is implemented. [Diagram 2] FIG. 1 is a schematic configuration diagram of a vehicle. [Diagram 3] FIG. 2 is a hardware configuration diagram of a data acquisition device. [Figure 4] FIG. 2 is a hardware configuration diagram of a server that is an example of a feature detection device. [Diagram 5]FIG. 2 is a functional block diagram of a server's processor related to feature detection processing. [Figure 6] 4 is a schematic diagram showing an example of a relationship between a feature detected by a first detection unit and a feature detected by a second detection unit. FIG. [Figure 7] 13 is an operational flowchart of feature detection processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The following describes a feature detection device, a feature detection method executed by the feature detection device, and a computer program for feature detection, with reference to the drawings. The feature detection device collects, for a given area, images showing features related to vehicle travel and information related to vehicle travel from one or more vehicles with which communication is possible.

[0016] The vehicle that transmits images to the feature detection device has multiple cameras with different shooting ranges, and generates an image showing the surroundings of the vehicle for each camera. The vehicle then transmits to the feature detection device, for each camera, a series of images in time series generated by the camera, the generation time of each image, and information related to the traveling of the vehicle.

[0017] The feature detection device detects the timing when the vehicle is parked based on at least one of the vehicle speed, steering angle, and shift position included in the information on the vehicle's travel. The feature detection device detects features from an image generated by a camera that can capture a relatively far range from the vehicle among the multiple cameras at times other than when the vehicle is parked. On the other hand, the feature detection device converts an image generated by a camera that captures a relatively close range to the vehicle among the multiple cameras into a bird's-eye image at the time when the vehicle is parked, and detects features from the converted bird's-eye image. Furthermore, this feature detection device estimates the position of the same feature based on the position of the vehicle when each of two or more images in which the same feature is detected was generated. The feature detection device generates data including information representing the type and position of each detected feature as data representing the feature. Note that, hereinafter, data representing the feature is simply referred to as feature data.

[0018] The features to be detected include, for example, various road signs, various road markings, traffic lights, and other features related to vehicle movement.

[0019] FIG. 1 is a schematic diagram of a feature data collection system in which a feature detection device is implemented. In this embodiment, the feature data collection system 1 has at least one vehicle 2 and a server 3 which is an example of a feature detection device. Each vehicle 2 is connected to the server 3 via the wireless base station 5 and the communication network 4 by, for example, accessing a wireless base station 5 which is connected to the communication network 4 to which the server 3 is connected via a gateway (not shown). Note that, for the sake of simplicity, only one vehicle 2 is illustrated in FIG. 1, but the feature data collection system 1 may have multiple vehicles 2. Similarly, only one wireless base station 5 is illustrated in FIG. 1, but multiple wireless base stations 5 may be connected to the communication network 4.

[0020] 2 is a schematic diagram of the vehicle 2. The vehicle 2 has a camera 11, a panoramic camera 12, a GPS receiver 13, a wireless communication terminal 14, and a data acquisition device 15. The camera 11, the panoramic camera 12, the GPS receiver 13, the wireless communication terminal 14, and the data acquisition device 15 are communicably connected via an in-vehicle network conforming to a standard such as a controller area network. The vehicle 2 may further have a navigation device (not shown) that searches for a planned travel route of the vehicle 2 and navigates the vehicle 2 to travel according to the planned travel route.

[0021] The camera 11 is an example of a first imaging unit, and has a two-dimensional detector configured with an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS, and an imaging optical system that forms an image of an area to be photographed (first range) on the two-dimensional detector. The camera 11 is attached, for example, in the passenger compartment of the vehicle 2 so as to face the front of the vehicle 2. The camera 11 photographs the area in front of the vehicle 2 at a predetermined photographing period (for example, 1 / 30 to 1 / 10 seconds) and generates an image showing the area in front of the vehicle 2. The camera 11 may be attached to the vehicle 2 so as to face another direction, for example, the rear of the vehicle 2. The image generated by the camera 11 is an example of a first image, and may be a color image or a gray image.

[0022] Each time the camera 11 generates an image, it outputs the generated image to the data acquisition device 15 via the in-vehicle network.

[0023] The panoramic camera 12 is an example of a second imaging unit, and has a two-dimensional detector configured with an array of photoelectric conversion elements sensitive to visible light, such as CCD or C-MOS, and an imaging optical system that forms an image of the area to be photographed (second range) on the two-dimensional detector. The panoramic camera 12 is attached to the vehicle 2 so as to photograph an area closer to the vehicle 2 than the camera 11. For example, the panoramic camera 12 is attached to the vehicle 2 facing diagonally downward so that the lower area around the vehicle 2 is included in the photographing range. Therefore, the photographing range of the panoramic camera 12 is narrower than the photographing range of the camera 11. In this embodiment, the focal length of the panoramic camera 12 is shorter than the focal length of the camera 11, but is not limited to this. The panoramic camera 12 photographs the surrounding area of ​​the vehicle 2 at a predetermined photographing period (for example, 1 / 30 second to 1 / 10 second) and generates an image in which the surrounding area is captured. The shooting cycle and shooting timing of the camera 11 can be independent of the shooting cycle and shooting timing of the panoramic camera 12. The image generated by the panoramic camera 12 is an example of the second image, and may be a color image or a gray image.

[0024] Note that the vehicle 2 may be provided with a plurality of panoramic cameras 12 with different shooting directions. For example, one panoramic camera 12 may be attached to the front end of the vehicle 2 so as to face the downward front side of the vehicle 2, and another panoramic camera 12 may be attached to the rear end of the vehicle 2 so as to face the downward rear side of the vehicle 2. Alternatively, four panoramic cameras 12 may be attached to the front end, rear end, and each of the left and right sides of the vehicle 2 so as to be able to shoot the entire surroundings of the vehicle 2.

[0025] Every time an image is generated, the panoramic camera 12 outputs the generated image to the data acquisition device 15 via the in-vehicle network. In the following, in order to distinguish the image generated by the panoramic camera 12 from the image generated by the camera 11, the image generated by the panoramic camera 12 may be referred to as a panoramic image.

[0026] Every time the panoramic camera 12 generates a panoramic image, it outputs the generated panoramic image to the data acquisition device 15 via the in-vehicle network.

[0027] The GPS receiver 13 receives GPS signals from GPS satellites at predetermined intervals, and determines the self-position of the vehicle 2 based on the received GPS signals. The predetermined interval at which the GPS receiver 13 determines the self-position of the vehicle 2 may be different from the image capturing interval by the camera 11 and the image capturing interval by the panoramic camera 12. The GPS receiver 13 outputs positioning information representing the positioning result of the self-position of the vehicle 2 based on the GPS signals to the data acquisition device 15 via the in-vehicle network at predetermined intervals. The vehicle 2 may have a receiver that complies with a satellite positioning system other than the GPS receiver 13. In this case, the receiver may determine the self-position of the vehicle 2.

[0028] The wireless communication terminal 14 is an example of a communication unit, and is a device that executes wireless communication processing conforming to a predetermined wireless communication standard. For example, by accessing the wireless base station 5, the wireless communication terminal 14 is connected to the server 3 via the wireless base station 5 and the communication network 4. The wireless communication terminal 14 generates an uplink wireless signal including the image information or driving information received from the data acquisition device 15. The wireless communication terminal 14 transmits the uplink wireless signal to the wireless base station 5 to transmit the image information and driving information to the server 3. The wireless communication terminal 14 also receives a downlink wireless signal from the wireless base station 5, and passes on collection instructions and the like from the server 3, which are included in the wireless signal, to the data acquisition device 15 or an electronic control unit (ECU, not shown) that controls the driving of the vehicle 2.

[0029] 3 is a hardware configuration diagram of the data acquisition device. The data acquisition device 15 generates image information including an image generated by the camera 11 and a panoramic image generated by the panoramic camera 12. Furthermore, the data acquisition device 15 generates driving information representing the driving behavior of the vehicle 2. To this end, the data acquisition device 15 has a communication interface 21, a memory 22, and a processor 23.

[0030] The communication interface 21 is an example of an in-vehicle communication unit, and has an interface circuit for connecting the data acquisition device 15 to the in-vehicle network. That is, the communication interface 21 is connected to the camera 11, the panoramic camera 12, the GPS receiver 13, and the wireless communication terminal 14 via the in-vehicle network. Then, the communication interface 21 passes the received image to the processor 23 every time it receives an image from the camera 11. Similarly, the communication interface 21 passes the received panoramic image to the processor 23 every time it receives a panoramic image from the panoramic camera 12. Also, the communication interface 21 passes the received positioning information to the processor 23 every time it receives positioning information from the GPS receiver 13. Furthermore, the communication interface 21 passes the collection instruction for image information received from the server 3 via the wireless communication terminal 14 to the processor 23. Furthermore, the communication interface 21 outputs the image information and the driving information received from the processor 23 to the wireless communication terminal 14 via the in-vehicle network.

[0031] The memory 22 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 may further include other storage devices such as a hard disk device. The memory 22 stores various data used in the process related to image collection executed by the processor 23 of the data acquisition device 15. Such data includes, for example, identification information of the vehicle 2, and parameters of the camera 11 such as the installation height, shooting direction, and angle of view of each of the camera 11 and the panoramic camera 12. The memory 22 may also store images received from the camera 11, panoramic images received from the panoramic camera 12, and positioning information received from the GPS receiver 13 for a certain period of time. The memory 22 also stores information indicating an area (hereinafter, sometimes referred to as a collection target area) that is to be the target of generating and collecting feature data, which is specified in an instruction to collect feature data. The memory 22 may also store computer programs for implementing each process executed by the processor 23.

[0032] The processor 23 has one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The processor 23 stores in the memory 22 the images received from the camera 11, the panoramic images received from the panoramic camera 12, the positioning information received from the GPS receiver 13, and the information representing the behavior of the vehicle 2 received from the ECU. Furthermore, the processor 23 executes processing related to image collection at predetermined intervals (for example, every 0.1 to 10 seconds) while the vehicle 2 is traveling.

[0033] As a process related to image collection, the processor 23 determines whether or not the vehicle position of the vehicle 2 represented by the positioning information received from the GPS receiver 13 is included in the collection target area. If the vehicle position is included in the collection target area, the processor 23 generates image information including the image received from the camera 11 and the panoramic image received from the panoramic camera 12.

[0034] The processor 23 includes, in the image information, parameters of the camera 11, such as an image generated by the camera 11, the time when the image was generated, the traveling direction of the vehicle 2 at that time, and the installation height, shooting direction, and angle of view of the camera 11. Similarly, the processor 23 includes, in the image information, a panoramic image, the time when the panoramic image was generated, the traveling direction of the vehicle 2 at that time, and parameters of the panoramic camera 12, such as the installation height, shooting direction, and angle of view of the panoramic camera 12. The processor 23 may obtain information indicating the traveling direction of the vehicle 2 from the ECU of the vehicle 2. Then, every time the processor 23 generates image information, it transmits the generated image information to the server 3 via the wireless communication terminal 14. The processor 23 may include, in one piece of image information, a plurality of images or a plurality of panoramic images, the generation time of each image or each panoramic image, and the traveling direction of the vehicle 2. The processor 23 may transmit the parameters of the camera 11 and the parameters of the panoramic camera 12 to the server 3 via the wireless communication terminal 14 separately from the image information.

[0035] Furthermore, the processor 23 generates driving information of the vehicle 2 from a predetermined timing (for example, the timing when the ignition switch of the vehicle 2 is turned on) onward, and transmits the driving information to the server 3 via the wireless communication terminal 14. The processor 23 includes in the driving information a series of positioning information obtained from the predetermined timing onward, the time when each piece of positioning information was obtained, and information indicating the behavior of the vehicle 2 used for dead reckoning, such as the wheel speed, acceleration, and angular velocity obtained from the ECU. Furthermore, the processor 23 may include identification information of the vehicle 2 in the driving information and image information.

[0036] Next, the server 3, which is an example of a feature detection device, will be described. 4 is a hardware configuration diagram of a server 3 which is an example of a feature detection device. The server 3 has a communication interface 31, a storage device 32, a memory 33, and a processor 34. The communication interface 31, the storage device 32, and the memory 33 are connected to the processor 34 via signal lines. The server 3 may further have input devices such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0037] The communication interface 31 is an example of a communication unit, and has an interface circuit for connecting the server 3 to the communication network 4. The communication interface 31 is configured to be able to communicate with the vehicle 2 via the communication network 4 and the wireless base station 5. That is, the communication interface 31 passes the image information and driving information received from the vehicle 2 via the wireless base station 5 and the communication network 4 to the processor 34. The communication interface 31 also transmits the collection instruction received from the processor 34 to the vehicle 2 via the communication network 4 and the wireless base station 5.

[0038] The storage device 32 is an example of a storage unit, and includes, for example, a hard disk device or an optical recording medium and an access device therefor. The storage device 32 stores various data and information used in the feature detection process. For example, the storage device 32 stores a parameter set for identifying a classifier for detecting features from various images, and identification information of each vehicle 2. The storage device 32 also stores image information and traveling information received from each vehicle 2. Furthermore, the storage device 32 may store a computer program for executing the feature detection process, which is executed on the processor 34.

[0039] The memory 33 is another example of a storage unit, and includes, for example, a non-volatile semiconductor memory and a volatile semiconductor memory. The memory 33 temporarily stores various data generated during the execution of the feature detection process.

[0040] The processor 34 is an example of a control unit and includes one or more central processing units (CPUs) and their peripheral circuits. The processor 34 may further include other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The processor 34 executes feature detection processing.

[0041] 5 is a functional block diagram of the processor 34 related to the feature detection process. The processor 34 has a first detection unit 41, a second detection unit 42, a switching determination unit 43, a position estimation unit 44, and a data generation unit 45. Each of these units in the processor 34 is a functional module realized by, for example, a computer program running on the processor 34. Alternatively, each of these units in the processor 34 may be a dedicated arithmetic circuit provided in the processor 34.

[0042] The first detection unit 41 detects features from each image generated by the camera 11 in a section of a time-series series of images generated by the camera 11 where feature detection has been instructed by the switching determination unit 43, i.e., from each image generated while the vehicle 2 is traveling.

[0043] The first detection unit 41 refers to the generation time of each image included in the image information received from the vehicle 2, and identifies images generated by the camera 11 in a section instructed by the switching determination unit 43 to detect features. The first detection unit 41 then inputs each identified image to a classifier that has been trained in advance to detect features to be detected, thereby detecting features shown in the input image (hereinafter, may be simply referred to as an input image). The first detection unit 41 may use a deep neural network (DNN) that has been trained in advance to detect features shown in the input image from the input image as such a classifier. As such a DNN, for example, a DNN having a convolutional neural network (CNN) type architecture such as Single Shot MultiBox Detector (SSD) or Faster R-CNN is used. Alternatively, as such a DNN, a DNN having a self attention network (SAN) type architecture such as Vision Transformer may be used.

[0044] In this case, the classifier calculates a certainty indicating the likelihood that a feature to be detected is represented in various regions on the input image for each type of feature (e.g., traffic lights, lane markings, pedestrian crossings, stop lines, etc.). The classifier determines that a feature of a certain type is represented in a region where the certainty for that type of feature is equal to or greater than a predetermined detection threshold. The classifier then outputs information indicating a region on the input image that contains the feature to be detected (e.g., a circumscribing rectangle of the feature to be detected, hereinafter referred to as an object region) and information indicating the type of feature represented in the object region.

[0045] Unlike the second detection unit 42, the details of which will be described later, the first detection unit 41 does not convert the image generated by the camera 11 into a bird's-eye image, but uses the image itself as an input image to the classifier. This is because the camera 11 can capture images at positions farther from the vehicle 2 than the panoramic camera 12, and therefore even if the image is converted into a bird's-eye image, objects that are farther away from the vehicle 2 than a certain distance will be distorted in the bird's-eye image, and improvement in classification accuracy cannot be expected.

[0046] The first detection unit 41 notifies the position estimation unit 44 and the data generation unit 45 of the generation time of each image, information representing the object area including the detected feature in each image, and the type of the detected feature.

[0047] The second detection unit 42 detects features from the panoramic images generated in a section instructed by the switching determination unit 43 to detect features, out of the series of time-series panoramic images generated by the panoramic camera 12. That is, the second detection unit 42 detects features from each of the panoramic images generated in the section when the vehicle 2 is parked.

[0048] The second detection unit 42 refers to the generation time of each panoramic image included in the image information received from the vehicle 2, and identifies the panoramic image generated by the panoramic camera 12 in the section instructed by the switching determination unit 43 to detect features, among the panoramic images generated by the panoramic camera 12. The second detection unit 42 then performs viewpoint conversion on each of the identified panoramic images to generate bird's-eye images, and detects features from each of the generated bird's-eye images. In this manner, the second detection unit 42 detects features from a series of panoramic images generated by the panoramic camera 12 installed so as to capture a different direction from the camera 11. Therefore, the second detection unit 42 can also detect features that are not shown in the image by the camera 11, for example, features hidden by obstacles located on the road surface such as parked vehicles. Furthermore, in the bird's-eye image, features that appear on the road surface, such as road markings, are particularly shown without much distortion. Therefore, by detecting features from the bird's-eye image converted from the panoramic image, the second detection unit 42 can detect features with high accuracy.

[0049] The second detection unit 42 performs viewpoint conversion processing for each panoramic image to a virtual viewpoint directed vertically downward from a predetermined height, using parameters such as the installation position, shooting direction, and focal length of the panoramic camera 12. As a result, the second detection unit 42 generates a bird's-eye image corresponding to each panoramic image. In addition, when a plurality of panoramic cameras 12 are provided on the vehicle 2, each panoramic image generated at the same timing by each panoramic camera 12 may be converted in viewpoint and synthesized to generate one bird's-eye image. Then, the second detection unit 42 detects the features shown in the input image by inputting each bird's-eye image to a classifier that has been trained in advance to detect the features to be detected. As such a classifier, the second detection unit 42 uses the same classifier as the classifier used by the first detection unit 41. Alternatively, the second detection unit 42 may use a classifier different from the classifier used by the first detection unit 41. In this case, the classifier used by the second detection unit 42 may be trained in advance using the bird's-eye image as the input image. In addition, the classifier used by the second detection unit 42 can also be a DNN having a CNN-type or SAN-type architecture.

[0050] According to a modified example, the classifier used by the second detection unit 42 may be trained in advance so that the panoramic image itself is used as an input image. In this case, the second detection unit 42 may detect features from each panoramic image by inputting each panoramic image to the classifier without performing viewpoint conversion processing. According to this modified example, since the viewpoint conversion processing is omitted, the computation load is reduced.

[0051] The second detection unit 42 notifies the position estimation unit 44 and the data generation unit 45 of the generation time of each panoramic image, information representing the object area including the detected feature in each panoramic image, and the type of the detected feature.

[0052] The switching determination unit 43 determines whether the vehicle 2 is parked in any section based on information indicating the behavior of the vehicle 2 included in the travel information, particularly at least one of the speed, steering angle, and shift position of the vehicle 2 at each time when the vehicle 2 travels in a specific area. The switching determination unit 43 then causes the second detection unit to detect features in the section when the vehicle 2 is parked. On the other hand, the switching determination unit 43 causes the first detection unit 41 to detect features in the section in which the vehicle 2 is traveling.

[0053] For example, the switching determination unit 43 determines a section where the vehicle speed of the vehicle 2 is equal to or lower than a predetermined speed threshold as a section where the vehicle 2 is parked. In addition, generally, when parking, the change in the traveling direction of the vehicle 2 is large. Therefore, the switching determination unit 43 may determine a section from when the vehicle speed of the vehicle 2 is equal to or lower than a predetermined speed threshold and the steering angle of the vehicle 2 is equal to or greater than a predetermined angle until the vehicle speed of the vehicle 2 exceeds the predetermined speed threshold as a section where the vehicle 2 is parked. Alternatively, the traveling direction of the vehicle 2 may be switched to a reverse direction when parking. Therefore, the switching determination unit 43 may determine a section from when the shift position of the vehicle 2 is in the reverse position until the shift position is next shifted to the drive position as a section where the vehicle 2 is parked.

[0054] The switching determination unit 43 notifies the first detection unit 41 of the remaining sections of the driving sections when the vehicle 2 drove through the specified area, excluding the section when the vehicle 2 was parked, and instructs the first detection unit 41 to detect features for the notified sections. The switching determination unit 43 also notifies the second detection unit 42 of the section when the vehicle 2 was parked, and instructs the second detection unit 42 to detect features for the notified sections. Furthermore, the switching determination unit 43 notifies the position estimation unit 44 and the data generation unit 45 of the section when the vehicle 2 was parked.

[0055] The position estimation unit 44 estimates the position of a feature detected from any of the images or any of the panoramic images.

[0056] In this embodiment, for the section in which the vehicle 2 is traveling, the position estimation unit 44 estimates the position of the vehicle 2 at the time of generating each image and the relative position of each detected feature with respect to these positions according to a so-called Structure from Motion (SfM) technique. That is, the position estimation unit 44 estimates the positional relationship between the feature of interest and the vehicle 2 based on the position of the feature of interest in each of two or more images obtained at different times while the vehicle 2 is traveling and the amount of movement of the vehicle 2 between the times when each image was generated. Then, the position estimation unit 44 estimates the position of the feature of interest in real space based on the estimated positional relationship.

[0057] Here, the position of each pixel on the image has a one-to-one correspondence with the direction from the camera 11 to the object depicted in that pixel. Therefore, the position estimation unit 44 can estimate the relative positional relationship between the feature and the vehicle 2 based on the direction from the camera 11, which corresponds to the feature point representing the feature in each image, the amount of movement of the vehicle 2 between the times when each image was generated, the traveling direction of the vehicle 2, and the parameters of the camera 11.

[0058] The position estimation unit 44 can set the reference position of the vehicle 2 to the vehicle position of the vehicle 2 represented in the positioning information by the GPS receiver 13 at a predetermined timing, which is included in the travel information. After that, the position estimation unit 44 estimates the amount of movement of the vehicle 2 between the generation times of each image, using information used for dead reckoning, such as the wheel speed, acceleration, and angular velocity, which is included in the travel information.

[0059] The position estimation unit 44 associates the same feature detected in each of a plurality of images obtained at different times while the vehicle 2 is traveling. In this case, the position estimation unit 44 can associate feature points included in an object region in which the same feature of interest is represented between a plurality of images by using, for example, a tracking method using optical flow. The position estimation unit 44 can estimate the relative position of the feature of interest with respect to the position of the vehicle 2 at the time each image is generated and the position of the vehicle 2 at the time each image is generated by triangulation. In this case, the position estimation unit 44 may use the traveling direction of the vehicle 2 at the time each image is generated, the position of the vehicle 2 at the time any one of the images is generated, the amount of movement of the vehicle 2 between the times each image is generated, the parameters of the camera 11, and the positions of the corresponding feature points in each image for the triangulation.

[0060] The position estimation unit 44 repeats the above process for the multiple detected features, thereby sequentially estimating the position of the vehicle 2 at the time of generating each image and the relative positions of the features located around the vehicle 2 with respect to that position. The position estimation unit 44 may then estimate the position of the feature in real space based on the relative position of the feature with respect to the estimated position of the vehicle 2 and the reference position of the vehicle 2. Note that if the detected features include features whose positions are known, the position estimation unit 44 may estimate the position of each detected feature in real space using the position of the known feature instead of the reference position of the vehicle 2.

[0061] The position estimation unit 44 estimates the position of the feature detected from each bird's-eye image in the following procedure. Note that the position estimation unit 44 only needs to execute the same process for each bird's-eye image, so the process for one bird's-eye image will be described below.

[0062] The position estimation unit 44 selects a feature (hereinafter referred to as a first feature) detected from any of the images generated by the camera 11 within a predetermined period immediately before switching to feature detection from the panoramic image. Then, the position estimation unit 44 specifies the relative position of the first feature with respect to the panoramic camera 12 at the time of generation of the panoramic image of interest, based on the positional relationship between the first feature and the vehicle 2 and the installation position of the panoramic camera 12. If the generation time of the image in which the first feature is detected is different from the generation time of the panoramic image of interest, the position estimation unit 44 refers to the travel information and determines the amount of movement of the vehicle 2 from the generation time of the image in which the first feature is detected to the generation time of the panoramic image of interest. Then, the position estimation unit 44 may specify the relative position of the first feature with respect to the panoramic camera 12, taking into account the amount of movement. Furthermore, the position estimation unit 44 specifies the position of the first object on the bird's-eye image based on the parameters of the panoramic camera 12, the relative position of the first object with respect to the panoramic camera 12, and the virtual viewpoint used for the viewpoint conversion. Furthermore, the position estimation unit 44 specifies the relative position of the second object with respect to the first object in real space based on the positional relationship on the bird's-eye image between the first object and another object (hereinafter referred to as the second object) detected from the bird's-eye image and the virtual viewpoint used for the viewpoint conversion. Then, the position estimation unit 44 estimates the position of the second object in real space based on the position of the first object in real space and the relative position of the second object with respect to the first object.

[0063] In a modified example in which a feature is detected from a panoramic image when the vehicle 2 is parked, the position estimation unit 44 may estimate the position of the second feature in real space by performing the same process as the above process on the second feature detected from the panoramic image instead of the bird's-eye image. In this case, the position estimation unit 44 may specify the relative position of the second feature in real space with respect to the first feature by using the parameters of the panoramic camera 12 instead of the parameters of the virtual camera used when generating the bird's-eye image.

[0064] The position estimation unit 44 notifies the data generation unit 45 of the position of each feature in real space.

[0065] The data generation unit 45 generates feature data representing each feature detected by the first detection unit 41 and each feature detected by the second detection unit 42. Specifically, for each feature detected by the first detection unit 41, the data generation unit 45 includes in the feature data the type of the feature notified by the first detection unit 41 and the position of the feature in real space estimated by the position estimation unit 44. Similarly, for each feature detected by the second detection unit 42, the data generation unit 45 includes in the feature data the type of the feature notified by the second detection unit 42 and the position of the feature in real space estimated by the position estimation unit 44.

[0066] The generated feature data is used to generate or update a map. That is, for each feature represented in the feature data, the type and position of the feature may be written to the map to be generated or updated.

[0067] FIG. 6 is a schematic diagram showing an example of the relationship between features detected by the first detection unit 41 and features detected by the second detection unit 42 according to this embodiment. In FIG. 6, when the vehicle 2 is at position P1, the vehicle 2 is traveling, and the first detection unit 41 detects features from the image generated by the camera 11. Therefore, lane markings and the like present within the shooting range 601 of the camera 11 are detected as features. On the other hand, at position P2, the vehicle 2 is about to park, and the second detection unit 42 detects features from the bird's-eye image obtained by performing viewpoint conversion on the panoramic image generated by the panoramic camera 12. Therefore, lane markings and the like present within the range 602 shown in the bird's-eye image are detected as features. In particular, in this example, the shooting range 601 and the range 602 shown in the bird's-eye image partially overlap, and therefore the road marking 611 located at the partially overlapping portion is detected by both the first detection unit 41 and the second detection unit 42. Therefore, the position on the bird's-eye image of the road marking 611, whose position in real space is estimated by the SfM technique, is specified. Furthermore, from position P1, a part 612 of the lane marking on the road edge side is hidden by a parked vehicle 621 from position P1, and is therefore not captured in the image generated by the camera 11. Therefore, the first detection unit 41 cannot detect the part 612 of the lane marking from the image generated by the camera 11. However, the part 612 of the lane marking is included in the range 602 represented in the bird's-eye image generated at position P2. Therefore, the second detection unit 42 can detect the part 612 of the lane marking. Furthermore, the position of the part 612 of the lane marking in real space is also estimated based on the position of the road marking 611 in real space and the relative positional relationship between the road marking 611 and the part 612 of the lane marking on the bird's-eye image.

[0068] 7 is an operation flowchart of feature detection processing in the server 3. When the processor 34 of the server 3 receives image information and driving information from the vehicle 2 for a predetermined section within the collection target area, it may execute the feature detection processing in accordance with the operation flowchart shown below.

[0069] The switching determination unit 43 of the processor 34 determines, in a predetermined section, a section in which the vehicle 2 is traveling and a section in which the vehicle 2 is parked (step S101). Then, the switching determination unit 43 instructs the first detection unit 41 to detect features in the section in which the vehicle 2 is traveling, and instructs the second detection unit 42 to detect features in the section in which the vehicle 2 is parked.

[0070] The first detection unit 41 of the processor 34 detects features from each of the series of images generated by the camera 11 in the section where the vehicle 2 is traveling and estimates the type of the feature (step S102). The second detection unit 42 of the processor 34 converts each panoramic image generated by the panoramic camera 12 in the section where the vehicle 2 is parked into a bird's-eye image, detects features from each bird's-eye image, and estimates the type of the feature (step S103). As described above, the second detection unit 42 may directly detect features from each panoramic image generated by the panoramic camera 12 in the section where the vehicle 2 is parked.

[0071] The position estimation unit 44 of the processor 34 estimates the position in real space of each of the features detected from each image generated by the camera 11 (step S104). Furthermore, the position estimation unit 44 specifies the position in the bird's-eye image of the first feature detected from any of the images generated by the camera 11 (step S105). Then, the position estimation unit 44 estimates the position in real space of the other features based on the positional relationship on the bird's-eye image between the specified position and the other features detected from the bird's-eye image and the position in real space of the first feature (step S106).

[0072] The data generation unit 45 of the processor 34 generates feature data representing each detected feature by including in the feature data the type of each detected feature and its position in real space (step S107). Then, the processor 34 ends the feature detection process.

[0073] As described above, while the vehicle is traveling, the feature detection device detects features from a series of time-series images generated by the first imaging unit, which has a relatively wide shooting range and can shoot positions relatively far from the vehicle. Therefore, the same feature is more likely to appear in two or more images generated at different positions, and as a result, the feature detection device can accurately estimate the position of the feature. On the other hand, when the vehicle is parked, the feature detection device detects features from a second image generated by the second imaging unit, which shoots an image relatively close to the vehicle. Therefore, the feature detection device can detect features from the second image that were not visible from the first imaging unit due to an obstacle or the like. Furthermore, the feature detection device estimates the position in real space of the feature detected from the second image based on the position in real space of the feature detected from the first image. Therefore, this feature detection device can accurately estimate the positions of features detected from the second image generated when the vehicle is parked, where the accuracy of position estimation by SfM is insufficient due to low speed driving or large steering angle. Therefore, this feature detection device can appropriately generate feature data representing the detected features.

[0074] The processing of each part of the processor 34 of the server 3 according to the above embodiment or modification may be executed by the processor 23 of the data acquisition device 15 mounted on the vehicle 2. In this case, the data acquisition device 15 is another example of a feature detection device. The data acquisition device 15 may transmit the generated feature data to the server 3 via the wireless communication terminal 14. In this case, the switching determination unit 43 may determine in real time whether the vehicle 2 is parked or not based on information indicating the behavior of the vehicle 2 acquired from the ECU. Based on the determination result, the first detection unit 41 or the second detection unit 42 may sequentially detect features from either the latest image generated by the camera 11 or the latest panoramic image generated by the panoramic camera 12. In addition, a part of the processing of each part of the processor 34 of the server 3 may be executed by the processor 23 of the data acquisition device 15. For example, the processor 23 may convert each of a series of time-series panoramic images generated by the panoramic camera 12 into a bird's-eye image, and transmit each bird's-eye image to the server 3 by including it in the image information. Alternatively, the processor 23 may execute the process of the switching determination unit 43, and for a section where the switching determination unit 43 determines that the vehicle 2 is traveling, the image information may include only the image generated by the camera 11. Furthermore, for a section where the vehicle 2 is parked, the processor 23 may include only the panoramic image generated by the panoramic camera 12 or the bird's-eye image generated from the panoramic image, in the image information.

[0075] Furthermore, the detected features may be used for purposes other than generating and updating the map. For example, the type and position of the detected features may be used in the ECU of the vehicle 2 to control the vehicle 2. In this case, the data generating unit may be omitted.

[0076] A computer program that causes a computer to realize the functions of each unit of the processor of the feature detection device according to the above embodiment or modified example may be provided in a form stored in a computer-readable recording medium. Note that the computer-readable recording medium may be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0077] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0078] 1. Geophysical data collection system 2 Vehicles 11 Camera 12 Panoramic Camera 13 GPS receiver 14 Wireless communication terminals 15 Data Acquisition Device 21 Communication Interface 22 Memory 23 Processors 3 Server 31 Communication Interface 32 Storage Device 33 Memory 34 processors 41 First detection unit 42 Second detection unit 43 Switching decision unit 44 Position estimation part 45 Data Generation Division 4. Communication Network 5. Radio base station

Claims

1. a first detection unit configured to detect one or more predetermined features from a first image representing a first range around the vehicle, the first image being generated by a first imaging unit provided in the vehicle; a second detection unit that performs viewpoint conversion on a second image that is generated by a second imaging unit provided on the vehicle and represents a second range that is closer to the vehicle than the first range to generate a bird's-eye image, and detects the one or more predetermined features from the generated bird's-eye image; a switching determination unit that determines whether the vehicle is parked based on at least one of a speed, a steering angle, and a shift position of the vehicle, and causes the second detection unit to detect the one or more predetermined features when the vehicle is parked, and causes the first detection unit to detect the one or more predetermined features while the vehicle is traveling; a position estimation unit that estimates a positional relationship between the first feature and the vehicle based on a position of the first feature among the one or more specified features in each of two or more first images obtained at different times while the vehicle is traveling and an amount of movement of the vehicle between the acquisition timings of each of the two or more first images, estimates a real space position of the first feature based on the estimated positional relationship, specifies a position of the first feature on the bird's-eye image based on the positional relationship between the vehicle and the first feature, and estimates a position of the other feature in real space based on a positional relationship between the position of the first feature on the bird's-eye image and a position on the bird's-eye image of another feature detected from the bird's-eye image among the one or more specified features and the position of the first feature in real space; A feature detection device having the following features:

2. 2. The feature detection apparatus of claim 1, further comprising a data generator configured to generate data representative of the one or more detected predetermined features.

3. determining whether the vehicle is parked based on at least one of a vehicle speed, a steering angle, and a shift position; Detecting one or more predetermined features from a first image representing a first range around the vehicle, the first image being generated by a first imaging unit provided in the vehicle, during a section in which the vehicle is traveling; generating a bird's-eye image by performing viewpoint conversion on a second image, which is generated by a second imaging unit provided in the vehicle and represents a second range closer to the vehicle than the first range, in a section in which the vehicle is parked, and detecting the one or more predetermined features from the generated bird's-eye image; estimating a positional relationship between the first feature and the vehicle based on a position of a first feature among the one or more predetermined features in each of two or more first images obtained at different times while the vehicle is traveling and an amount of movement of the vehicle between the acquisition times of each of the two or more first images, and estimating a position of the first feature in real space based on the estimated positional relationship; identifying a position of the first object on the bird's-eye image based on a positional relationship between the vehicle and the first object, and estimating a position of the other object in real space based on a positional relationship between the position of the first object on the bird's-eye image and a position on the bird's-eye image of another object detected from the bird's-eye image among the one or more specified objects, and on the position of the first object in real space; 23. A feature detection method comprising:

4. determining whether the vehicle is parked based on at least one of a vehicle speed, a steering angle, and a shift position; Detecting one or more predetermined features from a first image representing a first range around the vehicle, the first image being generated by a first imaging unit provided in the vehicle, during a section in which the vehicle is traveling; generating a bird's-eye image by performing viewpoint conversion on a second image, which is generated by a second imaging unit provided in the vehicle and represents a second range closer to the vehicle than the first range, in a section in which the vehicle is parked, and detecting the one or more predetermined features from the generated bird's-eye image; estimating a positional relationship between the first feature and the vehicle based on a position of a first feature among the one or more predetermined features in each of two or more first images obtained at different times while the vehicle is traveling and an amount of movement of the vehicle between the acquisition times of each of the two or more first images, and estimating a position of the first feature in real space based on the estimated positional relationship; identifying a position of the first object on the bird's-eye image based on a positional relationship between the vehicle and the first object, and estimating a position of the other object in real space based on a positional relationship between the position of the first object on the bird's-eye image and a position on the bird's-eye image of another object detected from the bird's-eye image among the one or more specified objects, and on the position of the first object in real space; A computer program for detecting features, which causes a computer to carry out the steps of:

5. A first detection unit that detects one or more predetermined features from a first image representing a first range around the vehicle, the first image being generated by a first imaging unit provided in the vehicle; a second detection unit that detects the one or more predetermined features from a second image that is generated by a second imaging unit provided on the vehicle and represents a second range that is closer to the vehicle than the first range; and a switching determination unit that determines whether the vehicle is parked based on at least one of a speed, a steering angle, and a shift position of the vehicle, and causes the second detection unit to detect the one or more predetermined features when the vehicle is parked, and causes the first detection unit to detect the one or more predetermined features while the vehicle is traveling; a position estimation unit that estimates a positional relationship between the first feature and the vehicle based on a position of a first feature among the one or more specified features in each of two or more first images obtained at different times while the vehicle is traveling and an amount of movement of the vehicle between the acquisition timings of each of the two or more first images, estimates a position of the first feature in real space based on the estimated positional relationship, specifies a position of the first feature on the second image based on the positional relationship between the vehicle and the first feature, and estimates a position of the other feature in real space based on a positional relationship between the position of the first feature on the second image and a position on the second image of another feature detected from the second image among the one or more specified features and the position of the first feature in real space; A feature detection device having the following features:

Images