Information processing apparatus, information processing method, and program

Abstract

To improve recognition accuracy of a road environment.SOLUTION: A predetermined target object is recognized based on an image acquired by a camera included in a first moving body, a first probability is estimated, the first probability being a probability that the first target object to be recognized cannot be normally recognized because the first target object is shielded by a second moving body, and a travel plan of the first moving body is corrected based on the first probability.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to vehicle technology. [Background technology]

[0002] There is a technology for generating road map data in real time while sensing the road environment. In this regard, for example, Patent Document 1 discloses a device for generating a target trajectory of a vehicle based on the results of sensing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-100827 [Patent Document 2] Japanese Patent Application Publication No. 2022-123239 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to improve the accuracy of road environment recognition. [Means for solving the problem]

[0005] One aspect of the present disclosure is The information processing device has a control unit that performs the following operations: recognizes a specified object based on an image acquired by a camera possessed by a first moving body; estimates in advance a first probability that the first object to be recognized will not be recognized correctly because it is blocked by a second moving body; and modifies the driving plan of the first moving body based on the first probability.

[0006] One aspect of the present disclosure is This is an information processing method executed by a computer, comprising: a first step of recognizing a predetermined object based on an image acquired by a camera carried by a first moving body; a second step of estimating in advance a first probability that the first object to be recognized will be blocked by a second moving body and therefore will not be recognized correctly; and a third step of modifying the driving plan of the first moving body based on the first probability.

[0007] Another aspect is a program for causing a computer to execute the above information processing method, or a computer-readable storage medium non-temporarily storing the program. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to improve the accuracy of recognizing road environments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining a problem in the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating the hardware configuration of the in-vehicle device 10. [Figure 3] FIG. 2 is a diagram illustrating the software configuration of the in-vehicle device 10. [Figure 4] 3 shows an example of guidance data stored in the in-vehicle device 10. [Figure 5] A diagram explaining the road network topology. [Figure 6] FIG. 10 is a diagram for explaining the flow of data in the first processing. [Figure 7] FIG. 10 is a diagram for explaining the flow of data in the second processing. [Figure 8] 10 is a flowchart of a process executed in the first process. [Figure 9] 10 is a flowchart of a process executed in the second process. [Figure 10] 10 is a flowchart illustrating the process of step S24 in detail. DETAILED DESCRIPTION OF THE INVENTION

[0010] In recent years, there has been much research into autonomous driving systems in which vehicles autonomously navigate along pre-set routes. Autonomous vehicles determine their own position and attitude by comparing pre-stored road maps with the results of sensing the road environment.

[0011] However, such systems face the challenge of constantly maintaining up-to-date road maps. For example, if a building or structure along the road is demolished, an inconsistency occurs with the road map, which can cause the vehicle to be unable to correctly identify its own location. A similar problem occurs when some lanes are closed due to construction work or other reasons. While there are methods for updating road maps based on information collected by probe cars, this problem cannot be completely solved due to the time lag.

[0012] To address this issue, research is being conducted into technologies that allow vehicles to recognize road environments in real time without storing road maps on the vehicle side. For example, a vehicle can store only data for route guidance (guidance data) and determine its own driving trajectory based on the results of real-time recognition of road areas (drivable areas).

[0013] Data for route guidance (guidance data) is data that records the general locations and directions of intersections, forks, etc. The vehicle compares this data with the sensing results to determine, for example, points along the route where it should turn right or left. For example, if a vehicle traveling on a highway needs to exit at an interchange with a certain name, the vehicle can recognize that the interchange it needs to exit is approaching by reading the name of the interchange written on a guide sign ahead.

[0014] However, when a vehicle is driving while sensing the road environment, there may be cases where the object to be recognized is hidden and cannot be recognized depending on the positional relationship with other vehicles. For example, if a large vehicle is driving directly in front of the vehicle, the camera may not be able to capture an overhead guide sign (such as an interchange guide), and the vehicle may pass the target interchange. The information processing device according to the present disclosure solves such problems.

[0015] An information processing device according to one embodiment of the present disclosure has a control unit that performs the following operations: recognizes a predetermined object based on an image acquired by a camera possessed by a first moving body; estimates in advance a first probability that the first object to be recognized will not be recognized correctly because it is blocked by a second moving body; and modifies a driving plan of the first moving body based on the first probability.

[0016] The first moving body and the second moving body are typically vehicles. The information processing device according to the present disclosure may be an in-vehicle device mounted on a first moving body, or may be a server device that performs processing based on images acquired by the first moving body and issues commands to the first moving body. The control unit executes a first process of recognizing a predetermined object based on an image acquired by a camera (for example, an in-vehicle camera). The predetermined object is any object used to determine the route of the vehicle. For example, the control unit may determine whether the object is right or left based on the recognized guide sign. You can recognize intersections where you should turn.

[0017] The control unit also estimates the probability that the first object will not be recognized correctly due to being occluded by a second moving object. The second moving object is another moving object (such as a vehicle) that may be located near the first moving object. The first probability may be calculated based on, for example, the proportion of the occluded portion of the first object relative to the whole, because it is expected that the greater the proportion of the occluded portion of the first object relative to the whole, the more likely it is that the first object will fail to be recognized. The first probability may also be calculated based on the probability that the second moving object will occlude the first object, because the higher the probability that the first object will be occluded, the more likely it is that recognition of the first object will fail.

[0018] The first probability can be calculated based on, for example, a result of estimating a relative positional relationship between the first moving body and the second moving body at the time when the first moving body passes near the first object. Therefore, the control unit may estimate the relative positional relationship and calculate the first probability based on the estimation result.

[0019] The control unit modifies the driving plan of the first moving body to change its positional relationship with the second moving body based on the first probability. For example, when the estimated first probability is equal to or greater than a predetermined value, the control unit modifies the driving plan of the first moving body so as to change the estimated relative positional relationship. For example, the control unit may modify the driving plan so that the first moving body is spaced a predetermined distance or more from the second moving body in a section where the first object is within the field of view of the camera. This ensures a field of view sufficient to recognize the first object.

[0020] In addition, the first moving body may be an autonomous vehicle that travels along a predetermined route, and the control unit may acquire information in advance about one or more of the first objects that need to be recognized on the predetermined route.

[0021] The first object may be, for example, an object used to guide the location of a road branch point. Examples of such an object include signs and billboards used to guide intersections and interchanges. The control unit may acquire information about such a first object (e.g., external characteristics, name, approximate location information, etc.) in advance. This allows the control unit to determine a section in which the first object is estimated to be visible.

[0022] The control unit may also acquire travel data relating to the travel of one or more second moving objects that are traveling near the first moving object. The travel data is data for predicting the movement of the second moving body. The travel data may be generated based on the results of observing the second moving body from the outside. Furthermore, if communication with a device that controls the travel of the second moving body is possible, data indicating a travel plan for the second moving body may be acquired from the device as the travel data.

[0023] For example, if the second moving body is an autonomous vehicle or a semi-autonomous vehicle, the control unit may acquire driving data contained in a V2X message transmitted from a device that controls the driving of the second moving body.

[0024] The control unit may generate travel data based on the results of sensing the second moving object. For example, by analyzing the speed and movement of the second moving object captured by the camera, the control unit can estimate the relative positional relationship between the first moving object and the second moving object in the immediate past.

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The present disclosure is not limited to the configurations of the embodiments.

[0026] (First embodiment) An overview of a vehicle system according to a first embodiment will be described. The vehicle system according to this embodiment includes a vehicle 1 and an on-board device 10 mounted on the vehicle 1. The on-board device 10 is connected to an on-board camera 11.

[0027] The problem to be solved by the system will be described with reference to FIG. An in-vehicle device 10 mounted on a vehicle 1 recognizes the road area around the vehicle based on images captured by an in-vehicle camera 11, and executes control to make the vehicle 1 travel autonomously using the recognition results.

[0028] A road area is typically an area in which the vehicle 1 can travel. A road area can be recognized by detecting road boundaries (road edges), but the objects of recognition are not limited to road edges. For example, lane boundary lines, lane centerlines, road centerlines, etc. may also be recognized.

[0029] The vehicle-mounted device 10 stores data (guidance data) for guiding the vehicle to the destination, and drives the vehicle 1 along the recognized road area in accordance with the guidance data. The guidance data typically defines waypoints (nodes in a road network) such as intersections, branches, and interchanges on a route. The guidance data includes data that provides guidance on the direction in which the vehicle should proceed at the waypoints, the characteristics of the waypoints, and methods for recognizing the waypoints.

[0030] 1(A) is a plan view showing a vehicle 1 traveling on a highway. In this example, the guidance data defines an interchange from which the vehicle 1 should exit as a route point, and the vehicle 1 exits from the specified interchange in accordance with the guidance data. The presence of an interchange can be recognized by, for example, using a guide sign installed nearby. For example, when the in-vehicle device 10 recognizes a sign indicating that "the target interchange is 500 m away" using the in-vehicle camera, the in-vehicle device 10 starts a lane change to exit the interchange.

[0031] However, depending on the positional relationship with other vehicles, the camera may not be able to capture such an object. For example, as shown in Figure 1(B), if a large vehicle is present between vehicle 1 and the object (guide sign), the view of the vehicle-mounted camera may be blocked, making it impossible to visually capture the object. In such cases, the presence of an interchange may not be detected, and problems may occur, such as being unable to exit correctly.

[0032] Therefore, the in-vehicle device 10 of this embodiment acquires information regarding the movement of other vehicles (hereinafter referred to as "other vehicles") traveling near the vehicle, and adjusts the positional relationship with the other vehicles in advance so that the other vehicles do not block the vehicle when passing near a specified object. For example, by taking measures in advance such as "driving at a distance from large vehicles that may cause obstructions" or "moving to a position where obstructions will not occur," it will be possible to reliably capture objects that need to be recognized for autonomous driving.

[0033] [Hardware configuration] Next, the hardware configuration of each device that constitutes the system will be described. First, we will explain the components of the vehicle 1. Fig. 2 is a diagram schematically illustrating an example of the hardware configuration of an on-board device 10 mounted on the vehicle 1. The vehicle 1 is configured to include the on-board device 10, an on-board camera 11, and a group of sensors 12.

[0034] The in-vehicle device 10 can be configured as a computer having a processor (CPU, GPU, etc.), a main memory device (RAM, ROM, etc.), and an auxiliary memory device (EPROM, hard disk drive, removable media, etc.). The auxiliary memory device stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that meet predetermined purposes, as described below, can be realized. However, some or all of the functions may be realized as hardware modules using hardware circuits such as ASICs and FPGAs.

[0035] The in-vehicle device 10 includes a control unit 101 , a storage unit 102 , a communication unit 103 , a location information acquisition unit 104 , and an input / output unit 105 .

[0036] The control unit 101 is a computing unit that executes predetermined programs to realize various functions of the in-vehicle device 10. The control unit 101 can be realized by, for example, a hardware processor such as a CPU. The control unit 101 may also be configured to include a RAM (Random Access Memory), a ROM (Read Only Memory), a cache memory, etc.

[0037] The storage unit 102 is a means for storing information, and is configured with storage media such as RAM, a magnetic disk, a flash memory, etc. The storage unit 102 stores programs executed by the control unit 101, data used by the programs, etc.

[0038] The communication unit 103 is a communication interface for connecting the in-vehicle device 10 to a vehicle network. The communication unit 103 is, for example, a network such as a CAN (Controller Area Network). The communication system is configured to be capable of communicating with on-board components of the vehicle 1 via the network.

[0039] The position information acquisition unit 104 acquires the position information of the vehicle 1. The position information acquisition unit 104 includes a GPS antenna and a positioning module for determining the position information. The GPS antenna is an antenna that receives positioning signals transmitted from positioning satellites (also referred to as GNSS satellites). The positioning module is a module that calculates the position information based on the signals received by the GPS antenna. The position information acquisition unit 104 may determine the traveling direction of the vehicle 1 based on the transition of the position information.

[0040] The vehicle-mounted camera 11 is an optical unit including an image sensor for acquiring images. The vehicle-mounted camera 11 is mounted facing forward of the vehicle 1, for example.

[0041] The sensor group 12 is a collection of multiple sensors included in the vehicle 1. The multiple sensors may be, for example, sensors that acquire data related to the traveling of the vehicle, such as a speed sensor, an acceleration sensor, and a GPS module. The multiple sensors may also acquire data related to the traveling environment of the vehicle 1.

[0042] The input / output unit 105 is a unit that receives input from vehicle occupants and presents information to them. Specifically, the input / output unit 105 is composed of a touch panel and its control means, and a liquid crystal display and its control means. In this embodiment, the touch panel and the liquid crystal display are combined into one touch panel display.

[0043] [Software configuration] Next, the software configuration of each device constituting the system will be described. Fig. 3 is a diagram showing a schematic software configuration of the in-vehicle device 10 according to this embodiment.

[0044] In this embodiment, the control unit 101 of the in-vehicle device 10 includes a recognition unit 111, a generation unit 112, and a recognition unit 113. , a driving control unit 113, and a correction unit 114. Each software module may be realized by the control unit 101 (CPU, etc.) executing a program stored in the storage unit 102 (described later). The information processing executed by the software modules is synonymous with the information processing executed by the control unit 101 (CPU, etc.).

[0045] Each software module can be broadly divided into those that recognize the road environment and determine the vehicle's planned trajectory, and those that estimate the presence of factors that may interfere with recognition (i.e., other vehicles that may cause obstruction) and correct the vehicle's planned trajectory.

[0046] First, the former will be described. The recognition unit 111, the generation unit 112, and the driving control unit 113 recognize the road environment within the field of view using an on-board camera, and determine an appropriate trajectory (planned trajectory) for the host vehicle.

[0047] The recognition unit 111 acquires data from the on-board camera 11 and the sensor group 12, and recognizes the road environment within the field of view of the camera based on the data. Specifically, the recognition unit 111 inputs image data acquired from the on-board camera 11 and sensor data acquired from the sensor group 12 into a machine learning model stored in the storage unit 102. The image data captures the scenery ahead of the vehicle, and the sensor data includes information on the position and attitude of the vehicle.

[0048] In this embodiment, the recognition unit 111 uses, as machine learning models, a model (first model) for recognizing objects on a road and a model (second model) for recognizing the network topology of the road. The first model is a model that estimates the position of an object in space based on image data and sensor data. The object may be anything that is referenced when performing autonomous driving, such as road boundaries, lane boundaries, traffic lights, crosswalks, and stop lines. The first model outputs information about the position of the recognized object in space (hereinafter referred to as feature information).

[0049] The second model is a model that estimates the road network topology based on image data and sensor data. The road network topology is, for example, information that indicates the state of connection between multiple lanes. The road network topology may, for example, be a representation of the connection relationships between lanes using nodes and edges. This makes it possible to determine, for example, that "on a road with two parallel lanes, if you drive in the right lane, you will be able to turn right at the next intersection (you will be able to transition to the edge of the intersecting road)." The second model outputs the connection relationships of one or more road edges within the camera's field of view as information about the road network topology (hereinafter referred to as topology information). By using the second model, the vehicle-mounted device 10 can recognize the connection relationship between a plurality of road edges.

[0050] The generation unit 112 generates map data based on the results of the recognition performed by the recognition unit 111. The map data is two-dimensional or three-dimensional road map data that indicates the drivable area (road area) within the field of view captured by the on-board camera. The generation unit 112 maps the objects indicated by the feature information into space. This results in a road map on which the drivable road area, lanes, etc. are mapped. The current position and orientation of the vehicle may also be mapped onto the road map. Furthermore, the generation unit 112 adds information about the network topology between lanes to the obtained map. By using such a road map, for example, when two lanes of road intersect at grade, it becomes possible to make a judgment such as "If you proceed in the left lane, you will be able to merge into the left lane of the intersecting road," and it becomes possible to appropriately determine the planned trajectory during autonomous driving. It becomes like this.

[0051] The driving control unit 113 determines the trajectory of the vehicle based on the map data generated by the generation unit 112 and the guidance data created in advance, and causes the vehicle to drive. A known method can be adopted as a method for autonomously driving the vehicle. In this embodiment, the driving control unit 113 determines the trajectory of the vehicle according to the generated map data, detects waypoints indicated by the guidance data, and sets the course of the vehicle in an appropriate direction at the waypoints. In the following description, information indicating the planned trajectory of the vehicle determined by the driving control unit 113 will be referred to as a "driving plan." The driving control unit 113 drives the vehicle while determining a driving plan within the field of view as needed.

[0052] On the other hand, as mentioned above, if the view of the onboard camera is blocked by another vehicle, there is a risk that the route points will not be correctly recognized. This may also result in the vehicle not being able to proceed to the appropriate road edge at a branch point, intersection, etc. To address this, the correction unit 114 estimates the presence of another vehicle that may cause blocking, and based on the result, corrects the driving plan determined by the driving control unit 113.

[0053] In this embodiment, the correction unit 114 detects other vehicles near the host vehicle based on images captured by an on-board camera, and estimates changes in the relative positional relationship between the host vehicle and the other vehicles. Then, when the host vehicle passes near an object (e.g., a guide sign) for identifying a waypoint, the correction unit 114 calculates the probability (the "first probability" in the present invention) that the object will be occluded by the other vehicle, making it impossible to correctly recognize the object. The larger the calculated value, the more likely it is that the object to be recognized will not be correctly recognized. Therefore, when the probability exceeds a predetermined value, the correction unit 114 interacts with the driving control unit 113 and modifies the driving plan. For example, the correction unit 114 may correct the driving plan so that the vehicle maintains a certain distance from other vehicles when the vehicle passes near a target object. This makes it possible to reliably recognize objects that need to be recognized.

[0054] In the following description, an object that needs to be visually recognized in order to identify a waypoint will be referred to as a “first object.” The first object may be a structure that exists at the waypoint, or a sign that provides advance notice of the existence of the waypoint. In this embodiment, the vehicle acquires almost all information other than route information, including road edge connectivity, using cameras and sensors. Therefore, there are many objects to be recognized in order to detect waypoints, such as signs, white lines, and road markings. The correction unit 114 may treat each of these multiple objects as a first target.

[0055] The storage unit 102 is a means for storing information, and is configured with storage media such as RAM, a magnetic disk, a flash memory, etc. The storage unit 102 stores programs executed by the control unit 101, data used by the programs, etc.

[0056] The storage unit 102 stores the above-mentioned map data, guidance data, machine learning models (first model and second model), and the like.

[0057] The map data is generated by the generation unit 112 as needed, and represents a range corresponding to the field of view of the vehicle-mounted camera. The generated map data may be deleted after passing through an area, or may be stored for use in subsequent trips. The map data may also be transmitted to an external device to assist other vehicles in autonomous driving.

[0058] Guidance data is data used to provide route guidance during autonomous driving. The guidance data may be data that records the general positions of intersections, branches, etc., and the direction of travel. The guidance data is acquired from a predetermined device (for example, a device that performs route search) before starting driving.

[0059] An example of guidance data is shown in Fig. 4. In the illustrated example, the guidance data includes the type of waypoint (intersection, branch, interchange, etc.), its name, the direction of travel at the waypoint (right turn, left turn, etc.), and location information of the waypoint. Vehicle 1 needs to visually recognize structures or buildings (such as junctions or interchanges) at the waypoints, or visually recognize auxiliary objects (such as guide signs) that provide advance guidance to the waypoints.

[0060] For example, to uniquely identify an intersection, it is necessary to read a nameplate installed at the intersection. The guidance data may include information about these auxiliary objects for identifying intermediate points. The "auxiliary information" field is an example of such information. For example, when the vehicle needs to exit an interchange with a specific name, the in-vehicle device 10 can recognize that the target interchange is ahead by reading a guide sign installed in front. Information about the installation location of such objects may be stored in the auxiliary information field.

[0061] The first model is a machine learning model that receives as input image data acquired by the in-vehicle camera 11 and sensor data acquired by the sensor group 12, estimates the positions of objects on the road, and outputs the results. In this embodiment, the first model recognizes road boundaries, lane boundaries, traffic lights, pedestrian crossings, stop lines, etc. as objects. The first model outputs the recognition results as "function information."

[0062] The second model is a machine learning model that estimates and outputs a road network topology using image data acquired by the in-vehicle camera 11 and sensor data acquired by the sensor group 12 as input. The road network topology is information that indicates, for example, the state of connections between multiple lanes using nodes and edges. The second model outputs information such as "topology information" as follows: "There are three edges (lanes) within the field of view, and the rightmost edge is connected to the edge of the intersecting road by a right turn, and the leftmost edge is connected to the edge of the intersecting road by a left turn." Figure 5 shows an example of visualization of topology information. The topology information is associated with the position information of the nodes and edges.

[0063] The specific configuration of the in-vehicle device 10 may include omissions, substitutions, and additions of components as appropriate depending on the embodiment. For example, the control unit 101 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, GPU, etc. Furthermore, input / output devices other than those illustrated (for example, an optical drive, etc.) may be added. Furthermore, the in-vehicle device 10 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same.

[0064] [Processing details] Next, the process executed by the in-vehicle device 10 (controller 101) will be described in detail. The processing executed by the in-vehicle device 10 (control unit 101) can be broadly divided into a processing (first processing) that determines an appropriate trajectory for the vehicle based on the results of recognizing the road environment and controls autonomous driving, and a processing (second processing) that estimates the presence of factors (other vehicles) that may interfere with recognition and corrects the trajectory.

[0065] First, the first process will be described. Fig. 6 is a schematic diagram showing the flow of data transmitted and received by a plurality of software modules included in control unit 101. First, the camera image acquired by the on-board camera 11 and the sensor data acquired by the sensor group 12 are input to the recognition unit 111. In this embodiment, the sensor data includes position information of the vehicle 1 and information on the attitude of the vehicle 1 (for example, the heading, which is the direction of travel). The recognition unit 111 inputs these pieces of data to the first model and the second model, respectively, and acquires feature information and topology information as estimation results. The feature information is information that indicates the arrangement of objects (e.g., road boundaries, lane boundaries, traffic lights, pedestrian crossings, stop lines, etc.) in a space. The topology information is information that indicates the network topology of roads, as shown in FIG.

[0066] The feature information and topology information output by the recognition unit 111 are transmitted to the generation unit 112. The generation unit 112 generates a road map by arranging the objects indicated by the feature information in space. The feature information includes position information of road boundaries, and arranging this information allows for a road map showing drivable road areas to be obtained. The feature information also includes position information of lane boundaries, and arranging this information allows for a road map with lane information to be obtained. The generation unit 112 also arranges traffic-related objects, such as traffic lights, stop lines, crosswalks, and guide signs, on the road map. Note that names and the like may be associated with the objects. For example, if the object to be recognized is a guide sign, or if the object to be recognized is a traffic light with an intersection nameplate attached to the traffic light, the generation unit 112 may associate the read name with the object.

[0067] Furthermore, the generation unit 112 adds topology information to the generated road map. As described with reference to FIG. 5, topology information is information that indicates the state of connection between multiple lanes. Because the topology information is associated with location information, the generation unit 112 provides network topology information to each lane included in the road map based on this. This provides information such as "whether a transition from one lane (edge) to another lane (edge) is possible" to the road map. The generation unit 112 stores the road map obtained by the above processing as map data.

[0068] The driving control unit 113 generates a driving trajectory of the vehicle by referring to the map data generated by the generation unit 112. The driving control unit 113 also references the guidance data, detects waypoints defined in the guidance data, and sets an appropriate route. For example, if the guidance data includes information to "turn left at an intersection named X" and the generated map data includes an intersection named X, the driving control unit 113 generates a trajectory that turns left at the intersection.

[0069] Next, the second processing will be described. The second processing is executed by the correction unit 114. Fig. 7 is a schematic diagram showing the flow of data transmitted and received by the correction unit 114. The correction unit 114 performs the following two types of processing.

[0070] (1) A process for determining the timing of passing near the first object As described with reference to FIG. 4, the in-vehicle device 10 needs to visually recognize structures and buildings (such as junctions and interchanges) that exist at waypoints, or auxiliary objects (such as guide signs) that provide advance guidance to waypoints. For this reason, the correction unit 114 first estimates the timing at which the vehicle will pass near a target object (first target object). This estimation can be performed based on map data and guidance data. For example, the correction unit 114 may calculate the timing based on the position information of the waypoints defined in the guidance data and the position information of the vehicle. Based on this, it is determined that the host vehicle will pass near the first object within one minute.

[0071] (2) Acquiring data on the movements of other vehicles located near the vehicle The correction unit 114 acquires data relating to the movement of other vehicles traveling near the host vehicle based on image data acquired from the in-vehicle camera 11. The data relating to the movement of other vehicles may be, for example, data indicating a change over time in the relative position of the other vehicle relative to the host vehicle (hereinafter referred to as relative position data). The correction unit 114 may determine a change over time in the relative position based on, for example, a change over time in the position of the other vehicle in the image. The correction unit 114 may also determine a change over time in the relative position with respect to other vehicles by using sensor data acquired from the sensor group 12. Furthermore, the correction unit 114 may acquire a driving plan (planned trajectory) from the driving control unit 113, and determine a change over time in the relative position with respect to other vehicles based on the plan.

[0072] (3) Calculating a first probability based on the relative position data Based on the acquired relative position data, the correction unit 114 calculates the probability that the first object is occluded by another vehicle or the degree of occlusion, and based on this, calculates the probability (first probability) that the first object cannot be correctly recognized. The first probability may be calculated based on, for example, an estimated distance between the host vehicle and another vehicle at the time when the host vehicle passes near the first object. For example, the first probability may be set to be higher when it is estimated that the distance between the host vehicle and another vehicle is closer. Furthermore, the first probability may be calculated based on, for example, a positional relationship (arrangement) between the host vehicle, the other vehicle, and the first object at the time when the host vehicle passes near the first object. The first probability may also be calculated based on the results of simulating the field of view of the vehicle-mounted camera 11 in a three-dimensional space.

[0073] (4) Modifying the driving plan based on the calculated first probability. When the calculated first probability exceeds a threshold, the correction unit 114 interacts with the driving control unit 113 and corrects the driving plan so that occlusion by another vehicle does not occur. For example, the correction unit 114 instructs the driving control unit 113 to correct the driving plan so that the distance between the host vehicle and another vehicle becomes equal to or greater than a predetermined value (or so that the first object is visible) before the host vehicle passes near the first object. This allows you to perform control such as "temporarily slow down and increase the distance between your vehicle and other vehicles" or "change lanes and move to a position where the first object is not blocked by other vehicles."

[0074] [Processing flow] Next, a description will be given of the flow of processing executed by the in-vehicle device 10. Fig. 8 is a flowchart of processing (first processing) for determining an appropriate trajectory for the vehicle based on the result of recognizing the road environment and controlling autonomous driving. The illustrated processing is executed periodically while the vehicle 1 is traveling.

[0075] First, in step S11 , the recognition unit 111 acquires a camera image from the vehicle-mounted camera 11 and acquires sensor data from the sensor group 12 .

[0076] Next, in step S12, the recognition unit 111 acquires feature information based on the camera image and sensor data. In this step, the recognition unit 111 inputs the image data and sensor data into the first model and acquires the output feature information. The feature information includes spatial position information of road boundaries, lane boundaries, traffic lights, pedestrian crossings, stop lines, etc.

[0077] In step S13, the recognition unit 111 recognizes the target based on the camera image and the sensor data. In this step, the recognition unit 111 inputs the image data and sensor data into the second model and acquires the output topology information. The topology information is information that represents the network topology of the road for each lane.

[0078] Next, in step S14, the generation unit 112 generates map data based on the feature information and topology information. In this step, the generation unit 112 generates a road map by arranging the objects indicated by the feature information in space, and adds topology information to the generated road map. The road map obtained by this process is temporarily stored as map data.

[0079] Next, in step S15, the driving control unit 113 generates a planned trajectory of the vehicle within the field of view captured by the on-board camera based on the generated map data. The driving control unit 113 also references the guidance data, detects waypoints defined in the guidance data, and sets an appropriate route.

[0080] Next, a flow of a process (second process) in which the on-vehicle device 10 estimates the presence of other vehicles that may interfere with object recognition and corrects the trajectory will be described. Fig. 9 is a flowchart of the second process. The illustrated process is periodically executed while the vehicle 1 is traveling.

[0081] First, in step S21, the correction unit 114 determines whether there is a first object on the route that needs to be recognized. In this step, the correction unit 114 determines the first object that needs to be recognized next based on the guidance data and the position information of the vehicle.

[0082] Next, in step S22, the correction unit 114 determines whether the host vehicle has approached the first object. This determination can be made based on the map data and the guidance data. For example, the correction unit 114 determines that the host vehicle has approached within 500 m of the first object based on the position information of the waypoint defined in the guidance data and the position information of the host vehicle. If the determination in this step is positive, the process proceeds to step S23. If the determination in this step is negative, the process returns to step S21.

[0083] Next, in step S23, the correction unit 114 acquires data (relative position data) indicating a change over time in the relative position between the host vehicle and another vehicle traveling nearby the host vehicle. The relative position data may be generated by the correction unit 114 based on a change over time in the position of the other vehicle in the image, a planned trajectory generated by the traveling control unit 113, or the like.

[0084] Next, in step S24, the correction unit 114 estimates the first probability. As described above, the first probability can be calculated based on the relative positional relationship between the host vehicle and another vehicle at the time when the host vehicle passes near the first object.

[0085] FIG. 10 is a flowchart showing in more detail an example of the process executed in step S24. First, in step S241, the correction unit 114 estimates the relative positional relationship between the host vehicle and another vehicle at the time when the host vehicle passes near the first object. The relative positional relationship between the host vehicle and another vehicle can be determined, for example, based on the relative position data acquired in step S23 and the driving plan (planned trajectory, etc.) acquired from the driving control unit 113. The relative positional relationship may include distance information.

[0086] Next, in step S242, the correction unit 114 performs a simulation of the field of view of the vehicle-mounted camera 11. In this step, the simulation of the field of view may be performed based on the mounting position of the vehicle-mounted camera 11, the size of other vehicles, etc. In this case, the correction unit 11 4 may be performed multiple times while changing the parameters to measure the rate at which occlusion occurs. Then, the correction unit 114 calculates the first probability based on the simulation result (step S243).

[0087] In this example, the correction unit 114 simulates the field of view, but the first probability may be calculated by other methods. For example, a table may be prepared that defines the positional relationships and distances between the first object, the host vehicle, and other vehicles in association with the first probability, and the first probability may be calculated using the table. The first probability may also be calculated using a machine learning model. For example, a machine learning model that has learned the relationship between the positional relationships and distances between the first object, the host vehicle, and other vehicles and the first probability may be used.

[0088] In step S25, correction unit 114 determines whether the estimated first probability is equal to or greater than a predetermined value. If the estimated first probability is equal to or greater than the predetermined value, the process proceeds to step S26. If the estimated first probability is less than the predetermined value, the process ends.

[0089] In step S26, the correction unit 114 instructs the driving control unit 113 to modify the driving plan. The instruction may include information for identifying the other vehicle that is causing the obstruction. In response to the instruction, the driving control unit 113 moves the vehicle to a position where the view of the on-board camera is not obstructed by the other vehicle, for example.

[0090] As described above, the in-vehicle device 10 according to the first embodiment estimates that a first object that needs to be recognized by the in-vehicle camera while traveling may be blocked by another vehicle, and modifies the traveling plan of the vehicle in advance, thereby improving the reliability of autonomous traveling.

[0091] (Modification of the first embodiment) In the first embodiment, the relative positional relationship between the vehicle and the other vehicle is estimated using images acquired by an on-board camera in step S23. On the other hand, if the other vehicle is traveling autonomously or semi-autonomously, it may be possible to acquire data such as the planned trajectory of the other vehicle.

[0092] For example, another vehicle may broadcast its own speed, traveling direction, planned trajectory, and the like using a V2X message. In this case, the correction unit 114 can generate relative position data by receiving the V2X message. Also, there are cases where autonomous driving is controlled by an external server device. In this case, the correction unit 114 may acquire data related to the driving of the other vehicle from the server device and generate relative position data based on the data.

[0093] (Variation) The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate within the scope that does not deviate from the gist of the disclosure. For example, the processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0094] In addition, in the description of the embodiment, an example has been given in which the in-vehicle device 10 controls the autonomous driving of the vehicle 1, but the autonomous driving control may be performed by a server device installed in a location different from the vehicle. Also, the first process may be performed by the vehicle 1, and the second process may be performed by the server device.

[0095] In addition, in the description of the embodiment, the terms "own vehicle" and "other vehicle" are used. The moving bodies (first moving body and second moving body) are not limited to vehicles, and may be, for example, autonomously moving robots, drones, or other moving bodies.

[0096] In the description of the embodiment, the vehicle 1 has only route information (guidance data), and almost all information required for traveling along the route is acquired by cameras and sensors. However, the degree of dependency on cameras and sensors may be lower than that illustrated. For example, if the on-board device stores a road map in which the connection relationships between road edges are defined, it may be possible to travel along the route if only the lane in which the vehicle is traveling can be recognized. In such a case, it is sufficient to control the vehicle so that the objects required for recognizing the lane are not blocked as much as possible.

[0097] In the description of the embodiment, control is performed so that all first objects included in the guidance data are not occluded, but it is not necessary to avoid occlusion for all objects. For example, if one of multiple first objects can be recognized and the route can be appropriately determined based on that, it may not be a problem even if other objects are occluded. In such a case, the "probability of taking the correct route based on the currently available information" may be calculated. Furthermore, if the probability is sufficiently high, processing to avoid occlusion may not be performed.

[0098] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0099] The present disclosure can also be realized by providing a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer via a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0100] 1. Vehicle 10...In-vehicle equipment 101 Control unit 102...Storage section 103 Communications Department 104...Location information acquisition unit 105...Input / output section

Claims

1. Recognizing a predetermined object based on an image acquired by a camera of the first moving object; Estimating in advance a first probability that a first object to be recognized will be blocked by a second moving object and will therefore be unable to be recognized normally; modifying a travel plan of the first moving object based on the first probability; a control unit that executes the following: Information processing device.

2. the control unit estimates a relative positional relationship between the first moving body and the second moving body at a timing when the first moving body passes near the first object, and estimates the first probability based on a result of the estimation. The information processing device according to claim 1 .

3. the control unit corrects a travel plan of the first moving object so as to change the estimated relative positional relationship when the estimated first probability is equal to or greater than a predetermined value. The information processing device according to claim 2 .

4. When the estimated first probability is equal to or greater than a predetermined value, the control unit modifies a travel plan of the first moving body so that a distance between the first moving body and the second moving body exceeds a predetermined value at a timing when the first moving body passes near the first object. The information processing device according to claim 1 .

5. the first moving body is an autonomous vehicle that travels along a predetermined route, the control unit acquires in advance information about one or more of the first objects that need to be recognized on the predetermined route; The information processing device according to claim 1 .

6. the first object is an object for guiding the location of a road branch point; The information processing device according to claim 5 .

7. the control unit acquires travel data relating to travel of one or more of the second moving bodies traveling near the first moving body; The information processing device according to claim 1 .

8. The control unit acquires the traveling data included in a V2X message transmitted from the second moving object. The information processing device according to claim 7 .

9. the control unit generates the traveling data based on a result of sensing the second moving object. The information processing device according to claim 7 .

10. 1. A computer-implemented information processing method, comprising: a first step of recognizing a predetermined object based on an image acquired by a camera of a first moving object; a second step of estimating in advance a first probability that the first object to be recognized will be blocked by a second moving object and therefore will not be recognized normally; a third step of correcting a travel plan of the first moving object based on the first probability; An information processing method, including:

11. In the second step, a relative positional relationship between the first moving body and the second moving body is estimated at a timing when the first moving body passes near the first object, and the first probability is estimated based on a result of the estimation. The information processing method according to claim 10.

12. In the third step, when the estimated first probability is equal to or greater than a predetermined value, a travel plan of the first moving object is corrected so that the estimated relative positional relationship is changed. The information processing method according to claim 11.

13. In the third step, when the estimated first probability is equal to or greater than a predetermined value, a travel plan of the first moving body is corrected so that a distance between the first moving body and the second moving body exceeds a predetermined value at a timing when the first moving body passes near the first object. The information processing method according to claim 10.

14. the first moving body is an autonomous vehicle that travels along a predetermined route, acquiring information about one or more of the first objects that need to be recognized on the predetermined route in advance; The information processing method according to claim 10.

15. the first object is an object for guiding the location of a road branch point; The information processing method according to claim 14.

16. acquiring travel data relating to the travel of one or more second moving bodies traveling in the vicinity of the first moving body; The information processing method according to claim 10.

17. Acquire the traveling data included in the V2X message transmitted from the second moving object.

17. The information processing method according to claim 16.

18. generating the traveling data based on a result of sensing the second moving object; 17. The information processing method according to claim 16.

19. A program for causing a computer to execute the information processing method according to any one of claims 10 to 18.

Images