Data collection device, method, computer program for collecting data, and data collection instruction device

By detecting snow cover and selecting appropriate feature types in the data collection device, and using a classifier to process camera images, the data collection problem of map updating in snowy environments was solved, and effective map data collection under snowy conditions was achieved.

CN118968437APending Publication Date: 2024-11-15WOVEN BY TOYOTA INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410584118.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-15
Filing Date
2024-05-11
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When there is snow on the road, existing technologies struggle to effectively collect feature data about the area around vehicles to update maps.

Method used

Snow accumulation is detected by setting up a decision unit in the data collection device. Different feature types are selected according to the snow accumulation, and a trained classifier is used to detect features from camera images to generate detection data.

Benefits of technology

Even when roads are covered in snow, it can accurately collect feature data of roads and the surrounding environment, ensuring the integrity and accuracy of map updates.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968437A_ABST
    Figure CN118968437A_ABST
Patent Text Reader

Abstract

A data collection device includes: a determination unit 42 that determines whether there is accumulated snow around a vehicle 2; a setting means (43) that sets the type of the feature to be detected on the basis of the determination result of the presence or absence of accumulated snow; and a detection unit (44) that detects the set type of feature from an image representing an area around the vehicle (2) generated by a camera (21) mounted on the vehicle (2), and generates detection data representing the detected feature.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a data collection device and method for generating or updating a map, a computer program for collecting data, and a data collection instruction device. Background Art

[0002] The high-precision map that the vehicle automatic driving system refers to for the automatic driving control of the vehicle needs to accurately represent information about the road. Therefore, data representing the characteristics of the road or the road around the predetermined area obtained by the sensor installed on the vehicle actually traveling in the predetermined area is collected from the vehicle. However, depending on the weather around the vehicle, it is difficult to detect such characteristics from the sensor signal. Therefore, a technology for collecting data by referring to weather information has been proposed (see Japanese Unexamined Patent Publication JP2008-39687A).

[0003] In the technology disclosed in JP2008-39687A, a vehicle-side device of a road map updating system sequentially determines whether a road on which a vehicle is traveling is a new road not included in the road map data. The vehicle-side device transmits an image of the vehicle's surroundings captured by the vehicle-mounted camera to a map management device based on the fact that the image captured by the vehicle-mounted camera when the road on which the vehicle is traveling is determined to be a new road can be used to update the center-side road map data based on weather information. Summary of the invention

[0004] In some areas, snow may remain on the road for a long time. In this case, the above-mentioned technology cannot collect images and update map information while snow still remains. Therefore, it is desirable to be able to collect data representing features that can be used to generate or update map information even when snow remains on the road.

[0005] An object of the present invention is to provide a data collection device capable of collecting characteristic data even when snow is accumulated on a road.

[0006] According to one embodiment, a data collection device is provided. The data collection device includes: a determination unit that determines whether there is snow accumulation around a vehicle; a setting unit that sets the type of feature to be detected based on the determination result of whether there is snow accumulation; and a detection unit that detects the set type of feature from an image representing an area around the vehicle generated by a camera installed on the vehicle, and generates detection data representing the detected feature.

[0007] The setting unit of the data collection device preferably sets a three-dimensional structure on or around the road as the type of feature to be detected when it is determined that there is snow around the vehicle; and sets a predetermined feature including a surface structure formed along the road surface or the ground around the road as the type of feature to be detected when it is determined that there is no snow around the vehicle.

[0008] In this case, the detection unit preferably detects the three-dimensional structure by inputting the image into a first classifier when it is determined that there is snow around the vehicle, and the first classifier has been trained to detect the three-dimensional structure from the image; and detects the predetermined feature by inputting the image into a second classifier when it is determined that there is no snow around the vehicle, and the second classifier has been trained to detect the predetermined feature from the image.

[0009] According to another embodiment of the present invention, a data collection instruction device is provided. The data collection instruction device includes: a determination unit that determines whether there is snow accumulation in a predetermined area; and a notification processing unit that notifies a snowplow of a collection instruction to collect detection data representing a predetermined type of feature in the predetermined area via a communication device when there is snow accumulation, and notifies a vehicle other than the snowplow of the collection instruction via a communication device when there is no snow accumulation.

[0010] According to another embodiment of the present invention, a method for collecting data is provided. The method includes: determining whether there is snow accumulation around a vehicle; setting the type of feature to be detected based on the determination result of whether there is snow accumulation; detecting the set type of feature from an image representing an area around the vehicle generated by a camera mounted on the vehicle; and generating detection data representing the detected feature.

[0011] According to another embodiment of the present invention, a computer program product for collecting data is provided. The computer program product includes instructions that cause a processor mounted on a vehicle to perform processing, the processing including: determining whether there is snow accumulation around the vehicle; setting the type of feature to be detected based on the determination result of whether there is snow accumulation; detecting the set type of feature from an image representing the area around the vehicle generated by a camera mounted on the vehicle; and generating detection data representing the detected feature.

[0012] The data collection device according to the present disclosure has an advantageous effect of being able to collect data representing characteristics even when there is snow on the road. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The configuration of a data collection system equipped with a data collection device is schematically shown.

[0014] Figure 2 The configuration of a vehicle is schematically shown.

[0015] Figure 3 The figure shows the hardware configuration of a data collection device according to an embodiment.

[0016] Figure 4 It is a functional block diagram of a processor of a data collection device.

[0017] Figure 5A An example of the type of features to detect when there is snow accumulation is shown.

[0018] Figure 5B Another example of the type of features to detect when there is no snow accumulation is shown.

[0019] Figure 6 It is an operational flow chart of data collection and processing.

[0020] Figure 7 A hardware configuration of a server as an example of a data collection instruction device is shown.

[0021] Figure 8 It is a functional block diagram of a server's processor.

[0022] Fig. 9 It is an operational flow chart of the processing related to the instruction of collecting data. DETAILED DESCRIPTION

[0023] The following will describe a data collection device, a method and a computer program for collecting data executed by the data collection device, and a data collection instruction device in conjunction with the accompanying drawings. The data collection device is mounted on a vehicle, and based on an image representing an area around the vehicle, generates data representing predetermined features on or around a road (hereinafter referred to as "detection data") for generating or updating a map. To this end, the data collection device determines whether there is snow accumulation around the vehicle, and sets the type of feature to be detected based on the result of the determination. In particular, when there is snow accumulation, the data collection device sets a feature of a type that can be detected even if there is snow accumulation as a feature to be detected.

[0024] Figure 1 The configuration of a data collection system equipped with a data collection device is schematically shown. In this embodiment, the data collection system 1 includes a data collection device 3 installed on at least one vehicle 2, and a server 4. The at least one vehicle 2 can be a snowplow or a vehicle other than a snowplow, such as an ordinary passenger car, a bus or a truck. The data collection device 3 is connected to a wireless base station 6, which is connected to a communication network 5 connected to the server 4 via a gateway (not shown), so that the data collection device 3 is connected to the server 4 via the wireless base station 6 and the communication network 5. Figure 1A single snowplow 2a and a single ordinary passenger car 2b are shown as the at least one vehicle 2, but the data collection system 1 may include a plurality of snowplows, each of which is equipped with a data collection device 3. Similarly, the data collection system 1 may include a plurality of vehicles other than snowplows, each of which is equipped with a data collection device 3. When the server 4 uniformly notifies each vehicle 2 of the collection instruction regardless of the presence of snow accumulation, as in the modified example described below, the data collection system 1 may include only snowplows or vehicles other than snowplows. In addition, the communication network 5 may be connected to a plurality of wireless base stations 6.

[0025] First, the vehicle 2 and the data collection device 3 will be described. As described above, the data collection system 1 may include a plurality of vehicles 2, each of which is equipped with a data collection device 3, and the vehicle 2 may be a snowplow 2a or a vehicle 2b other than a snowplow. However, a single vehicle 2 and a single data collection device 3 are described below because, with respect to data collection processing, each vehicle 2 and each data collection device 3 have the same configuration and perform the same processing.

[0026] Figure 2 The configuration of the vehicle 2 is schematically shown. In addition to the data collection device 3, the vehicle 2 also includes a camera 21 for photographing the area around the vehicle 2, a GPS receiver 22, and a wireless communication terminal 23. The camera 21, the GPS receiver 22, the wireless communication terminal 23, and the data collection device 3 are communicatively connected via an in-vehicle network conforming to a standard such as a controller area network. The vehicle 2 may also include a distance sensor (not shown) for measuring the distance to an object in the area around the vehicle 2, such as a LiDAR sensor.

[0027] The camera 21 is an example of an image capturing unit, and includes a two-dimensional detector composed of an array of photoelectric transducers such as CCD or C-MOS that are sensitive to visible light, and a focusing optical system that forms an image of a target area on the two-dimensional detector. The camera 21 is installed, for example, inside the vehicle 2, so as to face the front of the vehicle 2. The camera 21 captures an image of the area in front of the vehicle 2 at predetermined shooting cycles (e.g., 1 / 30 second to 1 / 10 second) and generates an image representing the area. The image acquired by the camera 21 may be a color image or a grayscale image. The orientation of the camera 21 is not limited to the forward direction of the vehicle 2. In particular, the camera 21 of the snowplow 2a can be installed to capture the area behind the snowplow 2a. Then an image representing the road surface after the snow is removed will be generated. In addition, the vehicle 2 may include a plurality of cameras 21 that take photos in different orientations or have different focal lengths.

[0028] Each time an image is generated, the camera 21 outputs the generated image to the data collection device 3 via the in-vehicle network.

[0029] The GPS receiver 22 receives GPS signals from GPS satellites at predetermined time intervals and determines the position of the vehicle 2 based on the received GPS signals. The GPS receiver 22 outputs positioning information indicating the result of determining the position of the vehicle 2 based on the GPS signals to the data collection device 3 via the in-vehicle network at predetermined time intervals. The vehicle 2 may include a receiver conforming to other satellite positioning systems instead of the GPS receiver 22. In this case, the receiver determines the position of the vehicle 2.

[0030] The wireless communication terminal 23 is an example of a communication device, is a device that performs wireless communication processing that complies with a predetermined wireless communication standard, and, for example, accesses the wireless base station 6 to connect to the server 4 via the wireless base station 6 and the communication network 5. In other words, a communication channel is established between the wireless communication terminal 23 and the server 4 via the wireless base station 6 and the communication network 5. The wireless communication terminal 23 receives a downlink radio signal from the server 4, the downlink radio signal including a collection instruction signal or a signal indicating a collection target area, and outputs the received signal to the data collection device 3. Further, the wireless communication terminal 23 generates an uplink radio signal, the uplink radio signal including a specified type of collection target data (e.g., detection data or an image) received from the data collection device 3. The wireless communication terminal 23 transmits the uplink radio signal to the wireless base station 6, thereby transmitting the collection target data to the server 4.

[0031] Figure 3 2 shows the hardware configuration of the data collection device 3. The data collection device 3 temporarily stores the images received from the camera 21. In addition, the data collection device 3 generates detection data based on, for example, the images, and temporarily stores the generated detection data. The data collection device 3 transmits the detection data and the images to the server 4 via the wireless communication terminal 23. To achieve this, the data collection device 3 includes a communication interface 31, a memory 32, and a processor 33.

[0032] The communication interface 31 is an example of an in-vehicle communication unit, which includes an interface circuit for connecting the data collection device 3 to the in-vehicle network. In other words, the communication interface 31 is connected to the camera 21, the GPS receiver 22, and the wireless communication terminal 23 via the in-vehicle network. Each time an image is received from the camera 21, the communication interface 31 passes the received image to the processor 33. Each time positioning information is received from the GPS receiver 22, the communication interface 31 passes the received positioning information to the processor 33. Each time information from the server 4, such as a collection instruction signal, is received from the wireless communication terminal 23, the communication interface 31 passes the information to the processor 33. Further, the communication interface 31 outputs the data received from the processor 33, such as collection target data, to the wireless communication terminal 23 via the in-vehicle network.

[0033] The memory 32 is an example of a storage unit, which includes, for example, volatile and non-volatile semiconductor memories. The memory 32 may also include other memories, such as a hard disk drive. The memory 32 stores various types of data used in the data collection-related processing performed by the processor 33 of the data collection device 3. For example, the memory 32 stores identification information of the vehicle 2 and parameters of the camera 21 such as focal length, orientation, and installation position. In addition, the memory 32 stores various parameters for specifying a determiner for determining whether snow is present and various parameters for specifying a classifier for detecting features from an image received from the self-camera 21. The memory 32 also stores positioning information received from the GPS receiver 22 and information indicating a collection target area received from the server 4. The memory 32 may also store a computer program for implementing the processing performed by the processor 33.

[0034] The memory 32 includes a storage area for storing images (hereinafter referred to as "image storage area") and a storage area for storing detection data (hereinafter referred to as "detection storage area"). The images received by the data collection device 3 from the camera 21 are temporarily stored in the image storage area. The detection data generated by the processor 33 is temporarily stored in the detection storage area.

[0035] The processor 33 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 33 may also include other computing circuits such as a logic arithmetic unit, an arithmetic unit, or a graphics processing unit. The processor 33 performs data collection processing during the driving of the vehicle 2.

[0036] Figure 4 : is a functional block diagram of the processor 33 of the data collection device 3. The processor 33 includes an image storage unit 41, a determination unit 42, a setting unit 43, a detection unit 44, and a communication processing unit 45. These units included in the processor 33 are functional modules, for example, implemented by a computer program executed by the processor 33, or they may also be dedicated operation circuits provided in the processor 33.

[0037] Each time the data collection device 3 receives an image from the camera 21, the image storage unit 41 writes the image into the image storage area of ​​the memory 32. Specifically, each time an image is received from the camera 21, the image storage unit 41 associates the position of the vehicle 2 indicated by the latest positioning information and the orientation of the vehicle 2 indicated by the orientation sensor (not shown) with the image. The image storage unit 41 may also associate parameters of the camera 21, such as the focal length, the installation position, and the orientation of the camera 21, with the image. When the image is uploaded to the server 4, the information associated with the image is uploaded to the server 4 together with the image itself.

[0038] The determination unit 42 determines whether there is snow around the vehicle 2. In this embodiment, the determination unit 42 inputs the latest image received from the camera 21 into a determiner that is pre-trained to be able to identify snow-covered areas and other areas. When the size of the snow-covered area indicated by the result output by the determiner is not less than a predetermined snow determination threshold, the determination unit 42 determines that there is snow around the vehicle 2. As such a determiner, the determination unit 42 can use a deep neural network (DNN) for semantic segmentation, such as a fully convolutional network (Fully Convolutional Network) or U-Net.

[0039] Alternatively, the determination unit 42 may determine whether there is snow accumulation around the vehicle 2 based on weather information received from a server (not shown) for distributing weather information via the wireless communication terminal 23. In this case, the determination unit 42 determines whether the current position of the vehicle 2 indicated by the latest positioning information received from the GPS receiver 22 is within the snow accumulation area indicated by the weather information. When the current position of the vehicle 2 is within the snow accumulation area, the determination unit 42 determines that there is snow accumulation around the vehicle 2.

[0040] Hereinafter, the state where it is determined that there is snow around the vehicle 2 will be simply referred to as “snow accumulation”, and the state where it is determined that there is no snow around the vehicle 2 will be simply referred to as “no snow accumulation.” The determination unit 42 notifies the setting unit 43 of the result of the determination whether there is snow accumulation.

[0041] The setting unit 43 sets the type of feature to be detected based on the determination result of whether there is snow notified by the determination unit 42. In the present embodiment, when it is determined that there is snow around the vehicle 2, the setting unit 43 sets the three-dimensional structure on or around the road as the type of feature to be detected. Examples of such three-dimensional structures include poles or arrow signs indicating the position of the road shoulder, traffic signs, signboards, and traffic lights. This is because, even if the road surface around the vehicle 2 is covered with snow, such a three-dimensional structure is likely to be visible without being covered by snow. Since the surface of the traffic sign may be covered with snow, the setting unit 43 preferably sets only the traffic sign itself as the detection target, and does not set the information represented on the traffic sign as the detection target. Similarly, the setting unit 43 preferably detects the signboard itself, rather than the information represented on the signboard. The positions of the traffic signs, signboards, and traffic lights are indicated on a "high-precision map" or a "dynamic map" so that these positions can be used, for example, for positioning processing to detect the position of the vehicle. Examples of the three-dimensional structure to be detected may also include a snow wall formed as a result of snow removal. However, the position of the snow wall may change, which is preferably used to update the dynamic map, but not to update the high-precision map. A high-precision map is a map that includes information (such as lane lines) for identifying the lane-level position of the vehicle. A dynamic map is a map that includes semi-dynamic or semi-static information such as traffic information and dynamic information such as information about other vehicles in addition to this information.

[0042] When it is determined that there is no snow accumulation around the vehicle 2, the setting unit 43 sets a predetermined feature including a surface structure formed along the road surface or the ground around the road as the type of feature to be detected. Examples of such surface structures include road markings, such as lane lines, stop lines, or crosswalks, and curbs. Examples of the predetermined features to be detected when it is determined that there is no snow accumulation around the vehicle 2 may include three-dimensional structures to be detected when there is snow accumulation, such as poles and traffic signs.

[0043] Figure 5A and Figure 5B Examples of the types of features to be detected when there is snow and when there is no snow are shown. Figure 5A In the example shown in , snow is detected in an image 500 representing the area around the vehicle 2. Due to the snow, no features on the road surface are visible. Therefore, three-dimensional structures such as poles 501 and traffic lights 502 are set as features to be detected.

[0044] exist Figure 5B In the example shown in , in the image 510 representing the area around the vehicle 2, snow is not detected and features on the road surface are visible. Therefore, road markings, such as lane lines 511, are set as features to be detected.

[0045] The setting unit 43 notifies the detection unit 44 of the set type of feature to be detected.

[0046] The detection unit 44 detects the type of feature set by the setting unit 43 from the latest image generated by the camera 21 at predetermined intervals during the travel of the vehicle 2 or each time the vehicle 2 travels a predetermined distance. The detection unit 44 generates detection data indicating the type and position of the feature detected in the image.

[0047] For example, the detection unit 44 inputs the image into a classifier to detect the features of the type to be detected represented in the input image. As such a classifier, the detection unit 44 may use a DNN having a Convolutional Neural Network (CNN) type architecture, such as a Single Shot MultiBox Detector or Faster R-CNN. Alternatively, as such a classifier, the detection unit 44 may use a DNN having a Self-Attention Network (SAN) type architecture, such as Vision Transster, or a classifier based on other machine learning techniques, such as an AdaBoost classifier. Such a classifier is pre-trained using a large number of training images representing the features of the type to be detected according to a predetermined training technique (such as back propagation) to detect features from the image. The classifier outputs information indicating an area in the input image that includes the features of the type to be detected, for example, a bounding rectangle of the features to be detected (hereinafter referred to as an "object area") and information indicating the type of features represented in the object area.

[0048] In this embodiment, a first classifier used when determining that there is snow around the vehicle 2 and a second classifier used when determining that there is no snow around the vehicle 2 are prepared separately. More specifically, the first classifier is pre-trained to detect the features of the type to be detected from the image when there is snow. The second classifier is pre-trained to detect the features of the type to be detected from the image when there is no snow. The first classifier and the second classifier may be classifiers based on different architectures or the same architecture. When the setting unit 43 notifies the type of features in the case of determining that there is snow around the vehicle 2, the detection unit 44 inputs the image into the first classifier, thereby detecting the features of the type to be detected when there is snow from the image. When the setting unit 43 notifies the type of features in the case of determining that there is no snow around the vehicle 2, the detection unit 44 inputs the image into the second classifier, thereby detecting the features of the type to be detected when there is no snow from the image. In this way, by preparing separate classifiers for the case of snow and the case of no snow, the features of the type to be detected by each classifier can be limited, so that the detection unit 44 can accurately detect the features of the type to be detected.

[0049] Alternatively, a common classifier may be used when determining that there is snow around the vehicle 2, and when determining that there is no snow around the vehicle 2. In this case, the classifier is pre-trained to detect from the image the features of the type to be detected when there is no snow and the features of the type to be detected when there is snow. The detection unit 44 inputs the image into the classifier regardless of whether there is snow. However, when there is snow, the detection unit 44 ignores the detection results output from the classifier except for the detection results of the features of the type to be detected when there is snow. Conversely, when there is no snow, the detection unit 44 ignores the detection results output from the classifier except for the detection results of the features of the type to be detected when there is no snow. Using a common classifier for the case where there is snow and the case where there is no snow can reduce the necessary hardware resources.

[0050] The detection unit 44 estimates the position of the feature represented in the object area detected from the image based on the direction from the camera 21 to the position corresponding to the centroid of the object area, the position and driving direction of the vehicle 2 when the image is generated, and the parameters of the camera 21 such as the orientation, focal length, and installation position. Specifically, the detection unit 44 can estimate the position of the feature by "Structure From Motion (SfM)". In this case, the detection unit 44 uses optical flow to associate object areas representing the same feature in two images acquired at different timings. The detection unit 44 estimates the position of the feature by triangulation based on the position and driving direction of the vehicle 2 when the two images are collected, the parameters of the camera 21, and the position of the object area in each image. The detection unit 44 generates detection data including information indicating the type and estimated position of the detected feature. In the detection data, the detection unit 44 may also include information indicating the position and driving direction of the vehicle 2 when the image is generated, and may also include information indicating the size and position of the object area in the image. The detection unit 44 generates a single piece of detection data for each detected feature. Therefore, when multiple features are detected from the image, multiple pieces of detection data are generated from the image.

[0051] Each time detection data is generated, the detection unit 44 writes the generated detection data into the detection storage area of ​​the memory 32 .

[0052] The communication processing unit 45 transmits the collection target data stored in the memory 32 to the server 4 via the wireless communication terminal 23 .

[0053] The communication processing unit 45 determines whether the vehicle 2 has entered the collection target area by referring to the information indicating the collection target area and the latest positioning information. When the vehicle 2 enters the collection target area, the communication processing unit 45 transmits the collection target data of the collection target area specified in the collection instruction signal to the server 4 via the wireless communication terminal 23.

[0054] For example, the collection target area is specified in units of one or more continuous road sections or areas of predetermined shapes. When the collection target area is specified as a road section, the information indicating the collection target area includes identification information for identifying the road section and information indicating the position of the end of the road section where entry or exit is possible. When the collection target area is an area of ​​predetermined shape, the information indicating the collection target area includes information indicating the position of the outer edge of the area. The communication processing unit 45 determines that the vehicle 2 has entered the collection target area when the position of the vehicle 2 indicated by the latest positioning information is within the area or road section indicated by the information indicating the collection target area.

[0055] When the detection data is designated as the collection target data, the communication processing unit 45 transmits the detection data stored in the detection storage area of ​​the memory 32 to the server 4 via the wireless communication terminal 23 in order of occurrence time.

[0056] When detection data and images are designated as collection target data, the communication processing unit 45 transmits the detection data stored in the detection storage area of ​​the memory 32 and the images stored in the image storage area of ​​the memory 32 to the server 4 via the wireless communication terminal 23 in chronological order.

[0057] The image designated as the target data for collection may be a sub-image. In this case, for each image stored in the image storage area, the communication processing unit 45 cuts out an area estimated to represent the road surface from the image to generate a sub-image. The communication processing unit 45 transmits the detection data stored in the detection storage area of ​​the memory 32 and the generated sub-image to the server 4 via the wireless communication terminal 23 in the order of occurrence time.

[0058] Further, the communication processing unit 45 deletes the collection target data that has been transmitted to the server 4 by the wireless communication terminal 23 from the memory 32 .

[0059] When the vehicle 2 leaves the collection target area, the communication processing unit 45 identifies the collection target data that has not been transmitted at the time of leaving by referring to the memory 32. When the transmission of the identified collection target data to the server 4 is completed, the communication processing unit 45 ends the transmission of the collection target data. The communication processing unit 45 determines whether the vehicle 2 has left the collection target area by referring to the information indicating the collection target area and the latest positioning information, which is similar to determining whether the vehicle 2 has entered the collection target area.

[0060] Figure 6 The processor 33 executes the data collection process according to the operation flowchart described below every predetermined period or every time the vehicle 2 travels a predetermined distance.

[0061] The image storage unit 41 of the processor 33 writes the image received by the data collection device 3 from the camera 21 into the image storage area of ​​the memory 32 (step S101). The determination unit 42 of the processor 33 determines whether there is snow around the vehicle 2 (step S102).

[0062] When there is snow around the vehicle 2 ("Yes" in step S102), the setting unit 43 of the processor 33 sets the three-dimensional structure on or around the road as the type of feature to be detected (step S103). When there is no snow around the vehicle 2 ("No" in step S102), the setting unit 43 sets a predetermined feature including a surface structure formed along the road surface or the ground around the road as the type of feature to be detected (step S104).

[0063] After step S103 or S104, the detection unit 44 of the processor 33 detects the set type of feature from the image generated by the camera 21 and generates detection data representing the detected feature (step S105). The detection unit 44 writes the generated detection data into the detection storage area of ​​the memory 32 (step S106). Then, the processor 33 terminates the processing related to the generation of the detection data.

[0064] Since it is assumed that the presence or absence of snow rarely changes locally, the processor 33 may perform the processing of steps S102 to S104 only once before or after the vehicle 2 enters the collection target area.

[0065] As described above, the data collection device sets the type of feature to be detected according to whether there is snow accumulation around the vehicle. In particular, when there is snow accumulation, the data collection device sets the type of feature that is used to generate or update a map and can be detected even when the road surface is covered with snow as the type of feature to be detected. Therefore, the data collection device can collect detection data even when there is snow accumulation, thereby preventing the collection of detection data from being stopped for a long time.

[0066] According to a modified example, the setting unit 43 can set the type of feature to be detected for each area in the image. More specifically, in an area where the result output from the determiner of the determination unit 42 indicates that there is snow accumulation, the setting unit 43 sets a three-dimensional structure as the type of feature to be detected, as in the above-mentioned embodiment. In an area where the result output from the determiner indicates that there is no snow accumulation, the setting unit 43 sets a predetermined feature including a structure on the surface as the type of feature to be detected. For each area, the detection unit 44 then detects the feature of the type set for the area. More specifically, among the features detected by inputting the image into the first classifier, the detection unit 44 generates detection data only for the features included in the area determined to have snow accumulation. Similarly, among the features detected by inputting the image into the second classifier, the detection unit 44 generates detection data only for the features included in the area determined to have no snow accumulation.

[0067] According to the present modification example, even when there are both areas around the vehicle where snow has been removed and areas where snow has not been removed, the data collection device can appropriately detect features.

[0068] The server 4 is described below. The server 4 is an example of a data collection instruction device, and it selects the type of vehicle to be instructed to collect collection target data according to whether there is snow accumulation. In addition, the server 4 specifies the type of data to be collected for the collection target area, and notifies the selected type of vehicle of a collection instruction signal indicating an instruction to collect the specified type of collection target data. In addition, the server 4 stores the detection data or image transmitted from the data collection device 3 installed on each vehicle 2 notified by the collection instruction signal. Based on the detection data or image, the server 4 generates or updates a map of the collection target area. In addition, the server 4 notifies each vehicle 2 of the collection target area.

[0069] Figure 7 4 shows the hardware configuration of the server 4. The server 4 includes a communication interface 51, a storage device 52, a memory 53 and a processor 54. The communication interface 51, the storage device 52 and the memory 53 are connected to the processor 54 via a signal line. The server 4 may also include input devices such as a keyboard, a mouse, and a display device such as a liquid crystal display.

[0070] The communication interface 51 is an example of a communication unit, which includes an interface circuit for connecting the server 4 to the communication network 5. The communication interface 51 is configured to be able to communicate with the data collection device 3 installed on each vehicle 2 via the communication network 5 and the wireless base station 6. More specifically, the communication interface 51 transfers the collection target data and other data received from the data collection device 3 of each vehicle 2 to the processor 54 via the wireless base station 6 and the communication network 5. The communication interface 51 transmits the collection instruction signal and other signals received from the processor 54 to the data collection device 3 of each vehicle 2 via the communication network 5 and the wireless base station 6.

[0071] The storage device 52 is an example of a storage unit, which includes, for example, a hard disk drive or an optical medium and an access device thereof. The storage device 52 stores information of a specified type, collection target data collected for each road section, and other data. For each vehicle 2, the storage device 52 also stores vehicle type information indicating the vehicle type (snow removal vehicle or other type of vehicle) and identification information. The storage device 52 can also store a computer program executed by the processor 54 for performing data collection processing on the server 4 side, and a map to be generated or updated based on the collection target data.

[0072] The memory 53 is another example of a storage unit, which includes, for example, nonvolatile and volatile semiconductor memories. The memory 53 temporarily stores various types of data generated during execution of data collection processing, and various types of data obtained by communicating with the vehicle 2, such as detection data and images.

[0073] The processor 54 is an example of a control unit, which includes one or more central processing units (CPUs) and their peripheral circuits. The processor 54 may also include other computing circuits such as a logic arithmetic unit or an arithmetic unit. The processor 54 performs data collection processing on the server 4 side.

[0074] Figure 8 4 is a functional block diagram of the processor 54 of the server 4. The processor 54 includes a determination unit 61 and a notification processing unit 62. These units included in the processor 54 are functional modules, for example, implemented by a computer program executed by the processor 54, or they may also be dedicated operation circuits provided in the processor 54.

[0075] The determination unit 61 determines whether there is snow accumulation in the collection target area. For example, the determination unit 61 determines whether there is snow accumulation in the collection target area based on weather information received from a server (not shown) for distributing weather information via the communication interface 51. To achieve this, the determination unit 61 compares the collection target area with the snow accumulation area indicated by the weather information. When at least a predetermined percentage of the collection target area is within the snow accumulation area indicated by the weather information, the determination unit 61 determines that there is snow accumulation in the collection target area. Alternatively, when a reference point set in the collection target area (e.g., the center or centroid of the collection target area) is within the snow accumulation area indicated by the weather information, the determination unit 61 may determine that there is snow accumulation in the collection target area.

[0076] When the server 4 receives an image generated in the collection target area from the vehicle 2 within a recent predetermined period (e.g., several hours to one day), the determination unit 61 can determine whether there is snow accumulation based on the image. In this case, the determination unit 61 determines whether there is snow accumulation based on a determination result obtained by inputting the image into a determiner for determining whether there is snow accumulation, similar to the determination unit 42 of the data collection device 3. When it is determined that there is snow accumulation, the determination unit 61 determines that there is snow accumulation in the collection target area based on the image.

[0077] The determination unit 61 notifies the notification processing unit 62 of the result of determining whether or not snow is accumulated in the collection target area.

[0078] The notification processing unit 62 selects the type of vehicle to be notified of the collection instruction based on the result of the determination of whether there is snow accumulation in the collection target area. More specifically, when there is snow accumulation in the collection target area, the notification processing unit 62 selects a snowplow as the type of vehicle to be notified of the collection instruction. When there is no snow accumulation in the collection target area, the notification processing unit 62 selects a vehicle other than a snowplow as the type of vehicle to be notified of the collection instruction. The notification processing unit 62 identifies the selected type of vehicle among the vehicles 2 by referring to the vehicle type information and the identification information.

[0079] The notification processing unit 62 refers to the type designation information corresponding to the collection target area. Based on the type designation information, the notification processing unit 62 identifies the type of collection target data designated for the collection target area. Examples of the type of collection target data include an image representing the surrounding environment of the vehicle 2 generated by the camera 21 mounted on the vehicle 2, a sub-image representing the road surface cut out from the image, and detection data representing features detected from the image. Therefore, for each collection target area, the type designation information indicates an image, a sub-image, or detection data as the type of collection target data.

[0080] The notification processing unit 62 includes information for specifying the type of the identified collection target data in the collection instruction signal. When the type specifying information indicates detection data, the notification processing unit 62 specifies only the detection data as the collection target data. When the type specifying information indicates an image, the notification processing unit 62 specifies both the detection data and the image as the collection target data. When the type specifying information indicates a sub-image, the notification processing unit 62 specifies both the detection data and the sub-image as the collection target data.

[0081] The notification processing unit 62 transmits the generated collection instruction signal to each vehicle 2 of the selected type via the communication interface 51 , the communication network 5 , and the wireless base station 6 .

[0082] When a plurality of collection target areas are set, the processor 54 executes the processing of the determination unit 61 and the notification processing unit 62 for each collection target area.

[0083] The processor 54 can set a collection target area based on the number of detection data collected for each road section stored in the storage device 52. For example, the processor 54 sets the road section where the number of detection data collected in the most recent predetermined period (e.g., a few weeks to a few months) is less than a predetermined collection threshold as a collection target area. Alternatively, among the various road sections represented in the map to be updated, the processor 54 can set the road section where the time elapsed since the last update is not less than a predetermined elapsed time threshold as a collection target area. The processor 54 then notifies the data collection device 3 of each vehicle 2 of the set collection target area via the communication interface 51.

[0084] In addition, the processor 54 can generate or update a map using the collected detection data and images. The map to be generated or updated is a map for autonomous driving control of the vehicle, and it can be a high-precision map or a dynamic map. The processor 54 performs registration between the features represented in the multiple detection data collected by a single vehicle 2 for a continuous interval and the corresponding features represented in the previously collected detection data or in the map to be updated. Among the features represented in the multiple detection data collected for the continuous interval, the processor 54 identifies the features that do not have corresponding features in the previously collected detection data or in the map to be updated as newly installed features. The processor 54 adds information about the identified features (position and type) to the map to be generated or updated. Further, among the features represented in the map to be updated, the processor 54 can identify the features that do not have corresponding features among the features represented in the multiple detection data collected for the continuous interval as removed features. The processor 54 can delete information about the features identified as removed from the map to be updated.

[0085] Fig. 9 The processor 54 of the server 4 executes the processing related to the instruction to collect data according to the operation flowchart described below at a predetermined timing (for example, at a predetermined date and time).

[0086] The determination unit 61 of the processor 54 determines whether there is snow accumulation in the collection target area (step S201). When there is snow accumulation in the collection target area ("Yes" in step S201), the notification processing unit 62 of the processor 54 selects a snowplow as the type of vehicle to be notified of the collection instruction (step S202). When there is no snow accumulation in the collection target area ("No" in step S201), the notification processing unit 62 selects a vehicle other than a snowplow as the type of vehicle to be notified of the collection instruction (step S203).

[0087] After step S202 or S203, the notification processing unit 62 notifies the selected vehicle of the collection instruction signal via the communication interface 51, the communication network 5 and the wireless base station 6 (step S204). Then, the processor 54 terminates the processing related to the instruction to collect data.

[0088] As described above, the data collection instruction device selects the type of vehicle to be instructed to collect the collection target data including the detection data according to whether there is snow accumulation in the collection target area. In particular, when there is snow accumulation in the collection target area, the data collection instruction device selects a snowplow as the vehicle to be instructed to collect the collection target data. Therefore, even when there is snow accumulation, the data collection instruction device can cause the detection data to be generated and collected when the road surface is visible to a certain extent. Therefore, even when there is snow accumulation, the data collection instruction device can prevent the accuracy of the features represented in the detection data from decreasing, and causes the data collection device to collect the detection data of the features (such as road markings) drawn on the road surface. When there is no snow accumulation, the data collection instruction device causes other vehicles except the snowplow (which does not often travel except during the removal of snow accumulation) to generate and collect detection data, so that the detection data can be appropriately collected even when there is no snow accumulation.

[0089] According to a modified example, the server 4 may uniformly notify each vehicle 2 of the collection instruction regardless of whether there is snow accumulation in the collection target area. When the server 4 specifies the vehicle to be instructed to collect data, as described above, assuming that the data collection device installed on each vehicle collects detection data representing the characteristics of the type to be collected when there is no snow accumulation, the processing of the determination unit 42 and the setting unit 43 may be omitted.

[0090] As described above, those skilled in the art can make various modifications according to the embodiments within the scope of the present invention.

Claims

1. Data collection device, include: a determination unit, which determines whether there is snow around the vehicle; a setting unit that sets the type of feature to be detected based on the determination result of whether snow accumulation exists; as well as A detection unit detects a feature of a set type from an image representing an area around the vehicle generated by a camera mounted on the vehicle, and generates detection data representing the detected feature.

2. The data collection device according to claim 1, in, The setting unit: When it is determined that there is snow around the vehicle, a three-dimensional structure on or around the road is set as the type of feature to be detected, and When it is determined that there is no snow around the vehicle, a predetermined feature including an on-surface structure formed along the road surface or the ground around the road is set as the type of feature to be detected.

3. The data collection device according to claim 2, in, The detection unit: When it is determined that there is snow around the vehicle, detecting the three-dimensional structure by inputting the image into a first classifier, the first classifier having been trained to detect the three-dimensional structure from the image, and When it is determined that there is no snow around the vehicle, the predetermined feature is detected by inputting the image into a second classifier, which has been trained to detect the predetermined feature from the image.

4. Data collection command device, include: a determination unit, which determines whether there is snow accumulation in a predetermined area; as well as A notification processing unit notifies the snowplow of a collection instruction for collecting detection data representing a predetermined type of feature in the predetermined area via a communication device when there is snow accumulation, and notifies the collection instruction to vehicles other than the snowplow via the communication device when there is no snow accumulation.

5. The methods used to collect data, include: Determine whether there is snow around the vehicle; Based on the determination result of whether snow is present, setting the type of feature to be detected; detecting features of a set type from an image representing an area surrounding the vehicle generated by a camera mounted on the vehicle; and Detection data representing the detected features is generated.

6. A computer program product for collecting data, the computer program causing a processor installed on a vehicle to perform a process, the process include: determining whether there is snow around the vehicle; Based on the determination result of whether snow is present, setting the type of feature to be detected; detecting features of a set type from an image representing an area surrounding the vehicle generated by a camera mounted on the vehicle; and Detection data representing the detected features is generated.