Information processing device and information processing system

Abstract

To improve travel safety of an automatic driving vehicle.SOLUTION: On-vehicle moving images including the same road zone are respectively acquired from a first vehicle which is an automatic driving vehicle and a second vehicle which is a manual driving vehicle, and a first travel track corresponding to the first vehicle and a second travel track corresponding to the second vehicle are identified based on the on-vehicle moving images. Identified is a target object existing at a first point which is a point where there is a deviation exceeding a predetermined value between the first travel track and the second travel track. Thus, the existence of an object, which is hard to deal with for the automatic driving vehicle, on the road can be determined.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to autonomous vehicles. [Background technology]

[0002] Various attempts have been made to improve the driving stability of autonomous vehicles. In this regard, Patent Literature 1 discloses a system that detects differences in driving operations between a manually driven vehicle and an autonomous vehicle and records points where the differences exceed a threshold value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-145016 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to improve the driving stability of autonomous vehicles. [Means for solving the problem]

[0005] A first aspect of the present disclosure is an information processing device having a control unit that performs the following operations: acquiring in-vehicle videos including the same road section from a first vehicle, which is an autonomous vehicle, and a second vehicle, which is a manually driven vehicle; identifying a first driving trajectory corresponding to the first vehicle and a second driving trajectory corresponding to the second vehicle based on the in-vehicle videos; and determining an object present at a first point, which is a point at which a deviation between the first driving trajectory and the second driving trajectory exceeds a predetermined value.

[0006] In addition, a second aspect of the present disclosure is an information processing system that includes an on-board device mounted on an autonomous vehicle or a manually driven vehicle, and a server device, wherein the on-board device has a first control unit that transmits an on-board video captured by an on-board camera to the server device, and the server device has a second control unit that performs the following operations based on the on-board video including the same road section: identifying a first driving trajectory corresponding to the autonomous vehicle and a second driving trajectory corresponding to the manually driven vehicle, and determining an object present at a first point, which is a point where there is a deviation between the first driving trajectory and the second driving trajectory that exceeds a predetermined value.

[0007] Another aspect of the present disclosure is a method executed by the above-described apparatus, or a computer-readable storage medium non-transitoryly storing a program for executing the method. Effect of the Invention

[0008] According to the present disclosure, it is possible to improve the driving stability of autonomous vehicles. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of a vehicle system. [Diagram 2] FIG. 2 shows a system configuration of an in-vehicle device 100. [Diagram 3] FIG. 2 is a schematic diagram of video data and position information data stored in a storage unit 102. [Figure 4] FIG. 2 is a diagram showing the components of the server device 200 in detail. [Diagram 5] An example of an image that has been labeled through segmentation. [Figure 6] 1 is a plan view showing the trajectory of a probe car 10 at a certain location. [Figure 7] A two-dimensional plot of the change in position over time. [Figure 8] 1 shows an example of data stored in a video database 202A. [Figure 9]13 is an example of location data 202C. [Figure 10] 4 is a flowchart of a process executed by the in-vehicle device 100. [Figure 11] 13 is a flowchart of a process for generating a map showing the locations of unsuitable points. [Figure 12] 13 shows an example of video data extracted in step S21. [Figure 13] 4 shows an example of map data generated by the determining unit 2012. [Figure 14] 4 is a flowchart of a process for instructing an autonomous vehicle 300 to operate. [Figure 15] 4 is a flowchart of a process executed by the autonomous vehicle 300. [Figure 16] 13 is an example of target object data according to the second embodiment. [Figure 17] 13 is an example of spot data according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Autonomous vehicles control acceleration, deceleration and steering based on information obtained through sensing. However, when an unexpected object is present near the road, accurate judgment cannot be made, which can lead to problems such as an inability to drive smoothly or reduced driving safety. For this reason, there are attempts to identify points where an autonomous vehicle cannot pass smoothly by detecting differences between the operation signals generated by the autonomous vehicle and the driving operations performed by the driver.

[0011] However, because there are various reasons why an autonomous vehicle cannot drive smoothly, there are cases where an appropriate response cannot be taken simply by identifying the location. For example, in places where the sidewalk is narrow and pedestrians frequently step onto the road, it is better to allocate more resources to pedestrian detection processing, and in places where one-way alternating traffic is being implemented due to road construction, it is better to allocate more resources to processing for recognizing instructions from traffic controllers. However, existing technologies do not determine what is present at points where autonomous vehicles cannot pass smoothly, and therefore cannot take appropriate action to improve the driving stability of autonomous vehicles. The information processing device according to the present disclosure solves such a problem.

[0012] An information processing device according to one aspect of the present disclosure is characterized in that it has a control unit that executes the following operations: acquiring in-vehicle videos including the same road section from a first vehicle, which is an autonomous vehicle, and a second vehicle, which is a manually driven vehicle; identifying a first driving trajectory corresponding to the first vehicle and a second driving trajectory corresponding to the second vehicle based on the in-vehicle videos; and determining an object present at a first point, which is a point at which there is a deviation between the first driving trajectory and the second driving trajectory that exceeds a predetermined value.

[0013] The first vehicle is an autonomous vehicle, i.e., a vehicle that drives autonomously based on the results of sensing. The second vehicle is a vehicle that drives under the control of a driver. The in-vehicle video is a video image captured by a camera mounted on the vehicle. The control unit identifies the travel paths of the first vehicle and the second vehicle based on the in-vehicle video including the same road section, and compares them with each other. Generally, when some problem occurs in the driving of an autonomous vehicle, the trajectory tends to become more erratic compared to that of a manually driven vehicle. Therefore, by comparing the trajectories of a manually driven vehicle and an autonomous vehicle on the same road section, it is possible to identify the points (first points) where the autonomous vehicle was unable to drive smoothly. The travel trajectory may be a set of multiple pieces of position information, or a set of transitions of positions in multiple time steps. The travel trajectory may be based on lane boundary lines. The travel trajectory may be identified based on the positions of lane boundary lines detected from an in-vehicle video, for example.

[0014] Furthermore, the control unit determines an object present at the first location. The object may be, for example, a parked vehicle, road construction, a pedestrian, a bicycle, a fallen object, a puddle, or other obstacle that may hinder automated driving. Note that the object may be something other than an object, such as a depression or crack in the road, or a faded white line. The object may be determined, for example, by performing pattern matching on multiple frames included in a video. The control unit does not necessarily need to identify what the object is, but it is sufficient if it can assign some kind of class or label to the object by machine learning or the like.

[0015] The information about the first location can be used to improve the driving stability or driving safety of the autonomous vehicle. For example, information about an object present at the first location may be transmitted to a third vehicle, which is an autonomous vehicle. Note that the first vehicle and the third vehicle may be the same vehicle.

[0016] Furthermore, the control unit may assign a score to the identified first point. The score may be, for example, a value related to the smoothness of traffic that changes due to the object. For example, the greater the object that is an obstacle to traffic, the greater the score that is assigned to the identified first point.

[0017] The score may be stored in association with the first location. For example, by mapping the score to a road map, a map showing locations where the autonomous vehicle requires caution when traveling can be generated. Furthermore, such a map can be used to control the autonomous vehicle. For example, if the information processing device is a device that provides route information to the autonomous vehicle, a route that avoids the first location can be generated based on the map. Furthermore, if the information processing device is a device that controls the operation of the autonomous vehicle, an instruction can be issued to the autonomous vehicle not to pass through the first location. Furthermore, such a map can be distributed to the autonomous vehicle.

[0018] Specific embodiments of the present disclosure will be described below with reference to the drawings. Unless otherwise specified, the hardware configuration, module configuration, functional configuration, and the like described in each embodiment are not intended to limit the technical scope of the disclosure to only those.

[0019] First Embodiment An overview of a vehicle system according to a first embodiment will be described with reference to FIG. The vehicle system according to this embodiment includes a probe car 10, a server device 200, and an autonomous vehicle 300.

[0020] The probe car 10 is a vehicle for collecting data. The probe car 10 may be an autonomous vehicle or a vehicle driven by a driver. The probe car 10 may be, for example, a general vehicle that has entered into a data provision contract with a business operator that provides a service. The autonomous vehicle 300 is an automatically driven vehicle that provides a predetermined service. The autonomous vehicle 300 may be a vehicle that transports passengers or cargo, or may be a mobile store vehicle. The autonomous vehicle 300 autonomously travels according to commands transmitted from the server device 200, and can provide a predetermined service.

[0021] The server device 200 is a device that controls the operation of the autonomous vehicle 300. In addition, the server device 200 determines points where there is an impediment to the traveling of the autonomous vehicle 300 based on the data collected from the probe car 10, and determines the traveling stability of the autonomous vehicle based on the result of the determination. A predetermined process is performed to improve the performance.

[0022] Each element that makes up the system will be explained. The probe car 10 is a connected car having a communication function with an external network. The probe car 10 is equipped with an in-vehicle device 100.

[0023] The in-vehicle device 100 is a computer for collecting information. In this embodiment, the in-vehicle device 100 has a camera installed facing the front of the vehicle, and transmits the captured video to the server device 200 at a predetermined timing. Hereinafter, the video captured by the in-vehicle device 100 is referred to as an in-vehicle video. The in-vehicle device 100 may be a device (such as a car navigation device) that provides information to a passenger of the probe car 10, or may be an electronic control unit (ECU) included in the probe car 10. The in-vehicle device 100 may also be a data communication module (DCM) having a communication function.

[0024] The in-vehicle device 100 can be configured as a computer having a processor such as a CPU or GPU, a main memory such as a RAM or ROM, and an auxiliary memory such as an EPROM, a hard disk drive, or a removable medium. The auxiliary memory stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions that match a predetermined purpose, as described below, can be realized. However, some or all of the functions may be realized by hardware circuits such as ASIC or FPGA.

[0025] FIG. 2 is a diagram showing a system configuration of the in-vehicle device 100. As shown in FIG. The in-vehicle device 100 includes a control unit 101 , a storage unit 102 , a communication unit 103 , an input / output unit 104 , a camera 105 , a position information acquisition unit 106 , and an acceleration sensor 107 .

[0026] The control unit 101 is a calculation unit that executes a predetermined program to realize various functions of the in-vehicle device 100. The control unit 101 may be realized by, for example, a CPU. The control unit 101 is configured to have, as functional modules, a video acquisition unit 1011 and a video transmission unit 1012. These functional modules may be realized by causing a CPU to execute stored programs.

[0027] The video acquisition unit 1011 captures video via the camera 105 described below, and stores the video in the storage unit 102. When the power of the device is turned on, the video acquisition unit 1011 generates a new storage area (for example, a folder, a directory, etc.).

[0028] During operation of the in-car device 100, the video acquisition unit 1011 captures video via the camera 105 and stores the obtained data (video data) in the storage unit 102. The video data is stored in file units. There is an upper limit (e.g., 1 minute or 5 minutes) on the length of a video corresponding to one file, and when the upper limit is exceeded, a new file is generated. If the storage capacity is insufficient, the video acquisition unit 1011 deletes the oldest file to secure free space and then continues shooting. Furthermore, the video acquisition unit 1011 acquires vehicle position information via the position information acquisition unit 106, which will be described later, at predetermined intervals (for example, every second), and stores the information as position information data. 3 is a schematic diagram of the video data and position information data stored in the storage unit 102. As shown in the figure, the video data and the position information data correspond one-to-one. By storing both the video data and the position information data in association with each other, it becomes possible to identify the traveling position of the vehicle after the fact.

[0029] The video transmission unit 1012 transmits the stored video data to the server device 200 at a predetermined timing. The predetermined timing may be a timing that arrives periodically. For example, the video transmission unit 1012 may transmit to the server device 200 the video data recorded in the immediately preceding file at the timing when a new file is generated.

[0030] The storage unit 102 is a memory device including a main storage device and an auxiliary storage device. The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and by loading the programs stored therein into the main storage device and executing them, it is possible to realize various functions that meet a predetermined purpose, as described below. The primary storage device may include a random access memory (RAM) or a read only memory (ROM). The secondary storage device may include an erasable programmable read only memory (EPROM) or a hard disk drive (HDD). It may also include a disk drive (HDD, Hard Disk Drive). may include removable media, i.e., portable recording media.

[0031] The storage unit 102 stores data generated by the control unit 101, that is, video data and position information data.

[0032] The communication unit 103 is a wireless communication interface for connecting the in-vehicle device 100 to a network. The communication unit 103 is configured to be able to communicate with the server device 200 according to a communication standard such as a mobile communication network, a wireless LAN, or Bluetooth (registered trademark).

[0033] The input / output unit 104 is a unit that receives an input operation performed by a user and presents information to the user. The input / output unit 104 is configured to include, for example, a liquid crystal display, a touch panel display, and a hardware switch.

[0034] The camera 105 is an optical unit including an image sensor for acquiring images. The camera 105 is mounted facing forward of the vehicle. The location information acquisition unit 106 calculates location information based on a positioning signal transmitted from a positioning satellite (also referred to as a GNSS satellite). The location information acquisition unit 106 may include an antenna that receives radio waves transmitted from the GNSS satellite. The acceleration sensor 107 is a sensor that measures the acceleration applied to the device. The measurement result is supplied to the control unit 101, and the control unit 101 can thereby determine that an impact has been applied to the vehicle.

[0035] Next, the server device 200 will be described. The server device 200 is a device that controls the operation of the autonomous vehicle 300. The server device 200 also has a function of determining points where there is an impediment to the traveling of the autonomous vehicle, based on video data acquired from a plurality of probe cars 10 (in-vehicle devices 100). In the following description, a point where the driving of an autonomous vehicle is hindered is referred to as an unsuitable point.

[0036] FIG. 4 is a diagram showing in detail components of the server device 200 included in the vehicle system according to this embodiment.

[0037] The server device 200 can be configured as a computer having a processor such as a CPU or GPU, a main storage device such as a RAM or ROM, and an auxiliary storage device such as an EPROM, a hard disk drive, or a removable medium. The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and the programs stored therein are loaded into the working area of ​​the main storage device and executed. Each component is controlled through the execution of the programs, thereby realizing each function that meets a predetermined purpose, as described below. However, some or all of the functions may be realized by hardware circuits such as ASICs and FPGAs.

[0038] The server device 200 includes a control unit 201 , a storage unit 202 , and a communication unit 203 . The control unit 201 is a calculation device that manages the control performed by the server device 200. The control unit 201 can be realized by a calculation processing device such as a CPU. The control unit 201 is configured to have, as functional modules, a video management unit 2011, a determination unit 2012, and an operation command unit 2013. Each functional module may be realized by causing a CPU to execute a stored program.

[0039] The video management unit 2011 executes a process of collecting video data transmitted from a plurality of probe cars 10 (vehicle-mounted devices 100) and storing the data in a storage unit 202 (video database 202A) described later. In this embodiment, the video management unit 2011 also divides the received video data into predetermined road segments (for example, based on map data 202B described later).

[0040] The determination unit 2012 determines points (unsuitable points) that are impediment to the driving of the autonomous vehicle based on the collected video data. Here, a point where there is an obstacle to the traveling of the autonomous vehicle will be described. The autonomous vehicle 300 according to this embodiment is a vehicle that recognizes the positions of lanes and obstacles based on images acquired by a stereo camera and travels. On the other hand, there are many cases where autonomous vehicles cannot travel smoothly due to various objects in the road environment. For example, when there is an unexpected parked vehicle on the road, the vehicle may drive slower than necessary to prevent pedestrians from jumping out. In addition, when lane restrictions are in place due to construction work, the vehicle may not understand the instructions of the traffic controller and may stop driving temporarily.

[0041] Therefore, in this embodiment, the determination unit 2012 determines points where there is an impediment to the traveling of the autonomous vehicle, based on video data collected by multiple probe cars 10. Also, based on the result of the determination, the operation command unit 2013, which will be described later, takes measures to improve the stability or safety of the autonomous vehicle.

[0042] First, the determination unit 2012 compares the trajectories of multiple probe cars (automatic and manually driven vehicles) that traveled along the same road segment in close proximity to each other in time, based on the collected in-vehicle video.

[0043] The vehicle trajectory can be identified based on the positions of lane boundary lines extracted from frames contained in a video. The presence of lane boundaries can be recognized, for example, by segmentation technology. Segmentation technology is a technology for classifying objects included in an image into multiple classes, and can be realized mainly by a machine learning model. By performing segmentation, it is possible to assign labels such as "sky," "nature," "other vehicles," "buildings," "white lines (lane boundaries)," "road," and "own vehicle" to multiple objects included in an image. Figure 5 shows an example of an image that has been labeled by segmentation.

[0044] In addition, the determination unit 2012 can determine in which lane the probe car 10 is traveling, based on the position of a pixel to which a label corresponding to a lane boundary line has been assigned. In addition, the determination unit 2012 determines where the center of the vehicle is located on the lane based on the position of the pixel to which the label corresponding to the lane boundary line is assigned and the mounting position of the camera 105. In this example, the line traced by the center of the vehicle is the vehicle's trajectory. Figure 6 is a plan view showing the trajectory traced by the probe car 10 on a road segment. This example shows how the probe car 10 avoids obstacles on the road. The determination unit 2012 performs trajectory identification for each road segment.

[0045] As described above, the determination unit 2012 compares the trajectories of an automatically driven vehicle and a manually driven vehicle. 6(A) shows a trajectory corresponding to the probe car 10, which is a manually driven vehicle, and FIG. 6(B) shows a trajectory corresponding to the probe car 10, which is an automatically driven vehicle. The black circles in the diagrams represent the passage of time. For example, if the interval between the black circles is 2 seconds, this means that the vehicle corresponding to FIG. 6(A) takes 8 seconds to perform an avoidance action, and the vehicle corresponding to FIG. 6(B) takes 14 seconds to perform an avoidance action. If there is a deviation between the two trajectories that exceeds a predetermined value, it is estimated that the autonomous vehicle is not able to travel smoothly for some reason.

[0046] FIG. 7 is a diagram in which the transition of positions over time is plotted on a two-dimensional plane. The deviation between the trajectories can be determined as the deviation degree between the lines shown in the figure. In the example of FIG. 7, the trajectories are plotted on a two-dimensional plane, but the trajectories may be plotted in a three-dimensional space having an X-axis, a Y-axis, and a time axis. The determination unit 2012 performs the deviation degree determination for each road segment. In addition, when there are multiple probe cars that have passed through the same road segment, the multiple probe cars may be divided into a group of automatically driven vehicles and a group of manually driven vehicles, and a representative trajectory for each group may be obtained, and then the degree of deviation between the respective trajectories may be obtained. The representative trajectory for each group may be obtained, for example, by an average value. The determining unit 2012 reflects the result of the determination in the spot data 202C, which will be described later. In this embodiment, the spot data 202C is data in which the deviation degree of the trajectory is associated with a road segment.

[0047] Secondly, the determination unit 2012 determines an inappropriate point based on the point data 202C. The inappropriate point may be, for example, a predetermined range including the point where the deviation occurs. The predetermined range may be, for example, a road segment. Information on the inappropriate point generated by the determination unit 2012 is transmitted to the traffic command unit 2013.

[0048] The operation command unit 2013 generates an operation plan for a specific autonomous vehicle 300 and transmits the generated operation plan to the autonomous vehicle 300. An operation plan is data that instructs the autonomous vehicle 300 on tasks to be executed. If the autonomous vehicle 300 is a vehicle that transports passengers, the tasks include a task of getting passengers on and off, a task of driving to a specified point, and the like. If the autonomous vehicle 300 is a vehicle that transports luggage, the tasks include a task of picking up luggage, a task of driving to a specified point, a task of handing over luggage, and the like. If the autonomous vehicle 300 is a mobile store, the tasks include a task of driving to a specified point, a task of opening a store at the arrival point, and the like. The operation command unit 2013 generates an operation plan that combines multiple tasks, and the autonomous vehicle 300 completes the tasks in order according to the operation plan, thereby providing a predetermined service. The operation plan may include a route along which the autonomous vehicle 300 will travel.

[0049] Furthermore, the operation command unit 2013 takes a predetermined measure to improve the driving stability of the autonomous vehicle based on the determined unsuitable points. Examples of measures to improve the stability or safety of an autonomous vehicle include the following: (1) Regenerate a route that avoids unsuitable points When a route that avoids unsuitable points is regenerated, the operation plan is corrected based on the route obtained by the regeneration. (2) Determine the resources required to pass through the bad spots For example, the trip plan may include data instructing the autonomous vehicle 300 to allocate more resources to pedestrian detection when traversing a road segment with a high pedestrian presence on the roadway, thereby allowing the autonomous vehicle 300 to more safely traverse the road segment. Note that the above is merely an example, and any process other than the example may be executed as long as it can improve the stability or safety of the autonomous vehicle. The content of the process may vary depending on the target object.

[0050] The storage unit 202 includes a main storage device and an auxiliary storage device. The main storage device is a memory in which the programs executed by the control unit 201 and data used by the control programs are expanded. The auxiliary storage device is a device in which the programs executed by the control unit 201 and data used by the control programs are stored.

[0051] The storage unit 202 also stores a video database 202A, map data 202B, and spot data 202C. The video database 202A is a database that stores a plurality of in-vehicle videos transmitted from the in-vehicle device 100. Fig. 8 is an example of data stored in the video database 202A. As shown in the figure, the video database 202A includes an identifier of the vehicle that transmitted the in-vehicle video, the type of vehicle (whether it is an autonomous vehicle or not), the shooting date and time, video data, a road segment identifier, and the like. The stored data may be deleted at a predetermined timing (for example, when a predetermined time has elapsed since it was received).

[0052] The map data 202B is a database that stores road maps. A road map can be represented, for example, by a set of nodes and links. The map data 202B includes definitions of the nodes, links, and road segments included in the links. A road segment is a unit section obtained by dividing a road link into a predetermined length. Each road segment may be associated with location information (latitude, longitude), an address, a place name, a road name, and the like. In this embodiment, trajectories are compared on a road segment basis.

[0053] The spot data 202C is data that records the result of the determination made by the determination unit 2012. As described above, the determination unit 2012 compares the trajectories of multiple probe cars 10 that have passed through a certain road segment, and reflects the result in the spot data 202C.

[0054] FIG. 9 is an example of the point data 202C. In this example, the spot data 202C has fields for date and time, spot, trajectory data 1, trajectory data 2, deviation, and object. The date and time field stores information about the date and time (time period) when the probe car traveled. The location field stores data for identifying the target location. Examples of such data include a road segment identifier, latitude, and longitude. The trajectory data field 1 stores trajectory data of the probe car 10, which is an autonomously driven vehicle. The trajectory data field 2 stores trajectory data of the probe car 10, which is a manually driven vehicle. The deviation field stores the calculated deviation. The object field stores information about the object determined by the server device 200.

[0055] The communication unit 203 is a communication interface for connecting the server device 200 to a network. The communication unit 203 includes, for example, a network interface board and a wireless communication interface for wireless communication.

[0056] 2 and 4 are merely examples, and all or part of the functions shown in the figures may be executed using specially designed circuits. Furthermore, programs may be stored or executed using a combination of a main memory device and an auxiliary memory device other than those shown in the figures.

[0057] Next, the processes executed by each device included in the vehicle system will be described in detail. 10 is a flowchart of the process executed by the in-vehicle device 100. The process shown in the figure is repeatedly executed by the control unit 101 while the in-vehicle device 100 is supplied with power.

[0058] In step S11, the video acquisition unit 1011 captures an in-vehicle video using the camera 105. In this step, the video acquisition unit 1011 records the video signal output from the camera 105 in a file as video data. As described with reference to FIG. 3, the file is divided into files of a predetermined length. If the storage area of ​​the storage unit 102 is insufficient, files are overwritten starting from the oldest file. Also, in this step, the video acquisition unit 1011 periodically acquires location information via the location information acquisition unit 106, and records the acquired location information in the location information data.

[0059] In step S12, the video acquisition unit 1011 determines whether a protection trigger has occurred. For example, a protection trigger occurs when an impact is detected by the acceleration sensor 107 or when a user presses a save button provided on the device body. In this case, the process proceeds to step S13, and the video acquisition unit 1011 moves the file currently being recorded to a protected area. The protected area is an area in which files are not automatically overwritten. This makes it possible to protect files that record important scenes.

[0060] If a protection trigger has not occurred, the process proceeds to step S14, where it is determined whether a file switch has occurred. As described above, the length of a video corresponding to one file has an upper limit (e.g., 1 minute, 5 minutes), and if the upper limit is exceeded, a new file is generated. If a switch has occurred, the process proceeds to step S15. Otherwise, the process returns to step S11. In step S15, the video transmission unit 1012 transmits the target video data to the server device 200 together with the position information data.

[0061] When the server device 200 (video management unit 2011) receives the video data, it identifies the road segment on which the probe car 10 has traveled based on the position information data, divides the video data for each road segment, and stores the divided data in the video database 202A. In the following description, one piece of video data corresponds to one road segment.

[0062] Next, the process executed by the server device 200 will be described in detail. 11 is a flowchart of a process for generating a map showing the locations of inappropriate points based on collected video data. This process is executed by the determination unit 2012 at a predetermined timing after the video data is accumulated.

[0063] First, in step S21, a set of moving image data to be processed is extracted from moving image database 202A. In this step, a set of moving image data that satisfies the following conditions is extracted. (1) They correspond to the same road segment (2) The image was taken within a specified period (for example, from 10 minutes before the current time to the current time). An example of the video data extracted in this step is shown in Fig. 12. In this example, four sets of video data are extracted. In addition, even if vehicles are traveling on the same road segment, if the lanes they are traveling on are different, the trajectories of the vehicles should not be compared. Therefore, if the target road has multiple lanes, the extraction process may be performed for each lane. For example, vehicles traveling on the same lane at the time of entering and exiting the road segment may be targeted for extraction.

[0064] In step S22, from among the extracted video data sets, any sets that do not include both images captured by an autonomously driven vehicle and images captured by a manually driven vehicle are deleted. In the example of Fig. 12, the video data set corresponding to segment D is to be deleted.

[0065] The processes in steps S23 and S24 are performed for each of the extracted pairs. In step S23, the trajectory of each probe car is identified based on multiple video data. In this step, as described above, the process of extracting lane boundaries is executed for each frame included in the in-vehicle video, and the position of the probe car (based on the lane) is calculated based on the position of the extracted lane boundary, and the trajectory is identified. In addition, if the target group includes two or more autonomous vehicles and two or more manually driven vehicles, the vehicles may be divided into a group of autonomous vehicles and a group of manually driven vehicles, and a trajectory representative of each group may be obtained.

[0066] In step S24, the degree of deviation between the trajectory corresponding to the autonomously driven vehicle and the trajectory corresponding to the manually driven vehicle is calculated, and it is determined whether the calculated degree of deviation exceeds a threshold value. The deviation may be calculated based on the geographical location alone, or on both the geographical location and the time. In the latter case, even if the trajectory is the same, if the time required for passing through the trajectory is different, it is considered to be a deviation.

[0067] If the calculated deviation exceeds the threshold, the process proceeds to step S25. If the calculated deviation does not exceed the threshold, the process proceeds to the next pair. In step S25, an object present in the road segment where the deviation occurred is determined based on the video data. The object may be determined by pattern matching or the like, or may be determined using a machine learning model. Note that the object does not need to be specifically identified in the determination. For example, if it is known that the same label is assigned to the same object, the object may be identified in a later process. In this example, the image is used to determine the object, but the object may be determined using another sensor mounted on the probe car 10.

[0068] The determination unit 2012 reflects the calculated deviation degree and data related to the determined object in the spot data 202C. Note that, although the object is specifically identified in the example of Fig. 9, a label indicating the result of classifying the object may be stored in the object field.

[0069] In step S26, the determination unit 2012 generates data (hereinafter, map data) in which the inappropriate points are mapped onto a road map based on the point data 202C. The map data includes information on points (or road segments) whose deviation degree exceeds a predetermined value and objects present at the points (road segments). FIG. 13 shows an example of map data generated by the determination unit 2012. The generated map data is stored in the storage unit 202 in association with the date and time when the in-vehicle video was captured.

[0070] Next, the server device 200 notifies the autonomous vehicle 300 of the route based on the generated map data. The process of issuing an instruction to operate the vehicle will now be described. 14 is a flowchart of a process in which server device 200 commands autonomous vehicle 300 to operate. The process shown in the figure is executed by operation command unit 2013 when a trigger occurs to dispatch autonomous vehicle 300. For example, in the case of providing a transportation service by autonomous vehicles, the process shown in the figure is started when a vehicle dispatch request is received from a passenger.

[0071] In step S31, a vehicle to be dispatched is determined from among a plurality of autonomous vehicles 300 under the control of the system based on a vehicle dispatch request or the like. The vehicle to be dispatched can be determined based on, for example, the content of the requested service, the current location of each vehicle, the task being performed by each vehicle, and the like. For example, if the requested service is a passenger transportation service, an autonomous vehicle 300 that has a passenger transportation function and can arrive at a specified location within a predetermined time is selected. For this reason, the server device 200 may hold data regarding the status of the autonomous vehicle 300.

[0072] In step S32, an operation plan corresponding to the selected autonomous vehicle 300 is generated. The operation plan is a collection of tasks to be executed by the autonomous vehicle 300. The tasks include, for example, a task of moving to a specified point, a task of boarding and disembarking passengers, a task of loading and unloading baggage, and the like. The tasks also include a route along which the autonomous vehicle 300 will travel. In this embodiment, the operation command unit 2013 performs a route search to determine a travel route for the autonomous vehicle 300.

[0073] Next, in step S33, it is determined whether the generated route includes an inappropriate point. If the generated route includes an inappropriate point, the process proceeds to step S34. If the generated route does not include an inappropriate point, the process proceeds to step S35. In step S34, measures are taken to improve the stability or safety of the autonomous vehicle. As described above, in this step, "processing for avoiding passing through inappropriate points" and "processing for safely passing through inappropriate points" can be performed. In step S35, the generated operation plan is transmitted to the target autonomously driving vehicle 300.

[0074] 15 is a flowchart of a process executed by the autonomous vehicle 300 that has received an operation plan. This process is started when the autonomous vehicle 300 receives the operation plan from the server device 200.

[0075] First, in step S41, the vehicle starts traveling to a destination point (that is, a point specified by the server device 200) according to a specified route.

[0076] When the autonomous vehicle 300 approaches the target point (step S42), it searches for a nearby location where the vehicle can be parked, and then executes the task (step S43).

[0077] When the task is completed, the autonomous vehicle 300 determines whether there is a next destination point according to the operation plan (step S44), and if there is a next destination point, it continues operation. If there is no next destination point (if all tasks included in the operation plan have been completed), it returns to the base.

[0078] As described above, the server device 200 according to the first embodiment identifies points where an autonomous vehicle cannot pass smoothly, based on the in-vehicle video transmitted from the probe car 10, and judges objects present at the points. Furthermore, the server device 200 takes measures against the inappropriate points based on the judgment results. This makes it possible to avoid troubles that prevent the autonomous vehicle from running smoothly due to the external environment.

[0079] (Modification 1 of the first embodiment) In the first embodiment, the server device 200 generates an operation plan for the autonomous vehicle 300. However, the server device 200 may not generate an operation plan, but may only determine inappropriate points (and generate map data). For example, the server device 200 may be configured to distribute only the generated map data to the autonomous vehicle 300. In this case, the autonomous vehicle 300 may autonomously generate an appropriate operation plan based on the received map data. The server device 200 may generate data in a format other than a map. For example, the location data (data in which inappropriate locations are associated with objects) as shown in FIG. 9 may be distributed to a plurality of autonomous vehicles.

[0080] The server device 200 may also be a device that provides a route search service specialized for autonomous vehicles. For example, in response to a request from the autonomous vehicle 300, the server device 200 may perform a route search that takes into account unsuitable points and return the result.

[0081] (Modification 2 of the first embodiment) In the first embodiment, a new operation plan for the autonomous vehicle 300 is generated based on the result of the determination made by the determination unit 2012. However, an operation plan for an autonomous vehicle 300 that has already started operation may be modified. For example, when an inappropriate point is newly generated, an autonomous vehicle 300 that is scheduled to pass through the inappropriate point may be instructed to change the route so as to avoid the inappropriate point. Also, when an inappropriate point is newly generated, an autonomous vehicle 300 that is scheduled to pass through the inappropriate point may be instructed to suspend or stop operation. Such a response can also be taken when there are restrictions on the driving route, such as when the autonomous vehicle 300 can only travel on routes permitted in advance (for example, when the autonomous vehicle 300 is a route bus). Furthermore, the autonomous vehicle 300 may be notified of resources required to pass through the inappropriate point. For example, if there is a special obstacle on the path of the autonomous vehicle 300, additional data for recognizing the obstacle may be transmitted to the autonomous vehicle 300. To perform such processing, the server device 200 may store the contents of the operation plans transmitted to the multiple autonomous vehicles 300.

[0082] Second Embodiment In the first embodiment, the autonomous vehicle is controlled based on the result of associating the unsuitable points with the object. In contrast, in the second embodiment, a score is assigned to each object, and the suitability of the route is determined based on the result of the scoring. In the second embodiment, the server device 200 stores data (object data) in which a score is assigned to each object. Fig. 16 shows an example of the object data. In this embodiment, the score is a value related to the smoothness of traffic, specifically, a value that indicates the degree to which an object is an obstacle to traffic. Note that objects may be subdivided into multiple items (categories, subcategories, etc.). In the illustrated example, different scores are assigned to road construction works with lane restrictions and road construction works without lane restrictions.

[0083] In the second embodiment, in step S25, the determination unit 2012 determines the object and calculates a score for the determined object based on the object data. Fig. 17 is an example of spot data to which a score has been assigned. In this embodiment, in step S26, map data associated with a score is generated. By using such map data, the driving route of the autonomous vehicle can be evaluated. For example, in the process of step S34, the scores corresponding to the unsuitable points included in the generated driving route are summed up to calculate the suitability (or unsuitability) of the entire route. Based on this, it becomes possible to make a decision as to whether or not to regenerate a route. In addition, when the server device 200 is a device that performs route search, it becomes possible to make a decision as to whether or not to select a route whose score satisfies a predetermined threshold value from among multiple route candidates.

[0084] (Other variations) The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate without departing from the spirit and scope of the present disclosure. For example, the processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0085] In addition, in the description of the embodiment, the probe car 10 and the autonomous vehicle 300 are illustrated as separate vehicles, but the autonomous vehicle 300 may function as a probe car. In addition, in the description of the embodiment, the probe car which is an autonomous driving vehicle and the probe car which is a manually driven vehicle are illustrated as separate vehicles, but the two may be the same vehicle. In other words, the same probe car may be capable of switching driving modes.

[0086] The results of the object determination may be used for further processing. For example, if an object is found to be present in a location where the autonomous vehicle's trajectory is disrupted, a corresponding image may be acquired and used as training data to train a machine learning model.

[0087] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0088] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0089] 10. Probe car 100...In-vehicle equipment 101,201...Control unit 102,202...Storage section 103,203···Communications Department 104...Input / output section 105···Camera 106...Location information acquisition unit 107 Acceleration Sensor 200 Server device 300 Autonomous Vehicles

Claims

1. Acquiring vehicle-mounted videos including the same road section from a first vehicle, which is an autonomous vehicle equipped with a first vehicle-mounted camera, and a second vehicle, which is a manually-driven vehicle equipped with a second vehicle-mounted camera; classifying a plurality of objects included in the images of the in-vehicle video into a plurality of classes, and assigning at least a lane boundary label, a label of the first vehicle, and a label of the second vehicle to the plurality of objects; Identifying a line traced by the center of the first vehicle, which is obtained by calculating the position of the center of the first vehicle relative to the lane boundary line at a predetermined period based on the position of a pixel labeled with the lane boundary line in an image of the in-vehicle video and the installation position of the first in-vehicle camera relative to the first vehicle, as a first driving trajectory corresponding to the first vehicle, and identifying a line traced by the center of the second vehicle, which is obtained by calculating the position of the center of the second vehicle relative to the lane boundary line at the predetermined period based on the position of a pixel labeled with the lane boundary line in an image of the in-vehicle video and the installation position of the second in-vehicle camera relative to the second vehicle, as a second driving trajectory corresponding to the second vehicle; Determining an object, including at least any one of a parked vehicle, a road construction, a pedestrian, a bicycle, a fallen object, and a puddle, that exists at a first point, which is a point where a geographical position and a time difference between the first travel path and the second travel path exceeds a predetermined value; An information processing device having a control unit that executes the above.

2. The control unit transmits information regarding the object present at the first location to a third vehicle that is an autonomous driving vehicle. The information processing device according to claim 1 .

3. The control unit determines the object based on an image corresponding to a point where the deviation occurs. The information processing device according to claim 1 .

4. The control unit calculates a score regarding a traffic smoothness for the object. The information processing device according to claim 3 .

5. Further comprising a storage unit that stores scores by category for the plurality of objects. The information processing device according to claim 4.

6. The control unit associates the calculated score with the first location.

6. The information processing device according to claim 4 or 5.

7. The control unit generates a map in which the calculated score is associated with the first location.

6. The information processing device according to claim 4 or 5.

8. The control unit transmits the generated map to a third vehicle that is an autonomous driving vehicle. The information processing device according to claim 7.

9. The control unit generates an operation plan for a third vehicle, which is an autonomous driving vehicle, based on the generated map. The information processing device according to claim 7.

10. An information processing system including a first in-vehicle device mounted on an autonomous driving vehicle, a second in-vehicle device mounted on a manually driven vehicle, and a server device, The first in-vehicle device is A first control unit that transmits an in-vehicle video captured by a first in-vehicle camera installed in the autonomous driving vehicle to the server device, The second in-vehicle device is A second control unit transmits an on-board video captured by a second on-board camera installed in the manually-driven vehicle to the server device, The server device includes: acquiring in-vehicle videos including the same road section from the first in-vehicle device and the second in-vehicle device, respectively; classifying a plurality of objects included in the images of the in-vehicle video including the same road section into a plurality of classes, and assigning at least a label of a lane boundary line, a label of the autonomous vehicle, and a label of the manually driven vehicle to the plurality of objects; Identifying a line traced by the center of the autonomous vehicle, which is obtained by calculating the position of the center of the autonomous vehicle relative to the lane boundary line at a predetermined period based on the position of a pixel labeled with the lane boundary line in an image of the in-vehicle video including the same road section and the installation position of the first in-vehicle camera relative to the autonomous vehicle, as a first driving trajectory corresponding to the autonomous vehicle, and identifying a line traced by the center of the manually driven vehicle, which is obtained by calculating the position of the center of the manually driven vehicle relative to the lane boundary line at the predetermined period based on the position of a pixel labeled with the lane boundary line in an image of the in-vehicle video including the same road section and the installation position of the second in-vehicle camera relative to the manually driven vehicle, as a second driving trajectory corresponding to the manually driven vehicle; Determining an object, including at least any one of a parked vehicle, a road construction, a pedestrian, a bicycle, a fallen object, and a puddle, that exists at a first point, which is a point where a geographical position and a time difference between the first travel path and the second travel path exceeds a predetermined value; A third control unit that executes Information processing system.

11. The third control unit transmits information regarding the object present at the first location to a third vehicle that is an autonomous driving vehicle. The information processing system according to claim 10.

12. The third control unit determines the object based on an image corresponding to a point where the deviation occurs. The information processing system according to claim 10.

13. The third control unit calculates a score regarding a traffic smoothness for the object. The information processing system according to claim 12.

14. The server device further includes a storage unit that stores scores by category for the plurality of objects. The information processing system according to claim 13.

15. The third control unit associates the calculated score with the first location.

15. An information processing system according to claim 13 or 14.

16. The third control unit generates a map in which the calculated score is associated with the first location.

15. An information processing system according to claim 13 or 14.

17. The third control unit transmits the generated map to a third vehicle that is an autonomous driving vehicle.

17. The information processing system according to claim 16.

Images