Information processing device, information processing method, and program

Abstract

To strike a balance between accuracy for recognizing a road environment and cost.SOLUTION: An information processing device performs first processing of recognizing a predetermined object on the basis of an image acquired by an on-vehicle camera included in a first vehicle, and performs second processing for improving recognition accuracy for a first region including at least a road region in which the first vehicle is predicted to travel included in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to vehicle technology. [Background technology]

[0002] There is a technology that generates road map data in real time while sensing the road environment. In this regard, for example, Patent Document 1 discloses a device that performs weighting correction on image recognition results according to the driving environment and recognizes road dividing lines based on the correction results. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-194815 [Patent Document 2] Japanese Patent Publication No. 2023-117563 [Patent Document 3] Japanese Patent Application Publication No. 2020-118890 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to achieve both high recognition accuracy of road environments and low costs. [Means for solving the problem]

[0005] One aspect of the present disclosure is The information processing device has a control unit that performs a first process to recognize a specified object based on an image acquired by an onboard camera of a first vehicle, and a second process to improve recognition accuracy for a first area included in the image that includes at least a road area along which the first vehicle is predicted to travel.

[0006] One aspect of the present disclosure is An information processing method executed by an information processing device capable of communicating with a first vehicle, comprising: This is an information processing method that includes performing a first process to recognize a specified object based on an image acquired by an onboard camera of the first vehicle, and performing a second process to improve recognition accuracy for a first area included in the image that includes at least a road area along which the first vehicle is predicted to travel.

[0007] Another aspect is a program for causing a computer to execute the above information processing method, or a computer-readable storage medium non-temporarily storing the program. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to achieve both high accuracy in recognizing road environments and low costs. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining a problem in the present disclosure. [Figure 2] FIG. 1 is a diagram for explaining a problem in the present disclosure. [Figure 3] FIG. 2 is a diagram illustrating the configuration of an in-vehicle device 10. [Figure 4] FIG. 4 is a diagram for explaining an outline of an addition process according to the first embodiment. [Figure 5] 2 is a diagram illustrating the flow of data in the in-vehicle device 10. [Figure 6] 3 is a flowchart of a process executed by the in-vehicle device 10 in the first embodiment. [Figure 7] FIG. 10 is a diagram for explaining an outline of processing in the second embodiment. [Figure 8] FIG. 10 is a diagram for explaining an outline of processing in the second embodiment. [Figure 9] 10 is a flowchart of a process executed by the in-vehicle device 10 in the second embodiment. [Figure 10] FIG. 10 is a diagram for explaining an outline of processing in a modified example of the second embodiment. [Figure 11]10 is a flowchart of a process executed by the in-vehicle device 10 in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] In recent years, there has been much research into autonomous driving systems in which a vehicle autonomously drives along a pre-set route. In autonomous driving, a vehicle determines its own position and attitude by comparing a pre-stored road map with the results of sensing the road environment.

[0011] However, such systems face the challenge of constantly maintaining up-to-date road maps. For example, if a building or structure along the road is demolished, an inconsistency occurs with the road map, which can cause the vehicle to be unable to correctly identify its own location. A similar problem occurs when some lanes are closed due to construction work or other reasons. While there are methods for updating road maps based on information collected by probe cars, this problem cannot be completely solved due to the time lag.

[0012] To address this issue, research is being conducted into technologies that allow vehicles to recognize road environments in real time without storing road maps on the vehicle side. For example, a vehicle stores only data for route guidance and determines the route to follow based on the results of real-time recognition of road areas (drivable areas). In this configuration, it is necessary to accurately determine road areas based on images captured by an on-board camera.

[0013] To accurately determine road areas, it is possible to increase resources allocated to image processing, for example by improving resolution or frame rate. However, in-vehicle devices have limited resources compared to stationary computers. Therefore, a more cost-effective resource allocation method is required. The information processing device according to the present disclosure solves such problems.

[0014] An information processing device according to one aspect of the present disclosure has a control unit that performs a first process for recognizing a specified object based on an image acquired by an onboard camera of a first vehicle, and a second process for improving recognition accuracy for a first area included in the image that includes at least a road area along which the first vehicle is predicted to travel.

[0015] The information processing device according to the present disclosure may be an on-board device mounted on a first vehicle, or may be a server device that performs processing based on images acquired by the first vehicle and provides information to the first vehicle. The control unit executes a first process of recognizing a predetermined object based on an image acquired by the on-board camera. The predetermined object may be, for example, a road on which the first vehicle is traveling. For example, the control unit can travel while recognizing an area (road area) in which the first vehicle can travel based on the image.

[0016] The control unit also performs a second process for improving recognition accuracy for a first region in the image. The first region is a region that includes at least a road region on which the host vehicle is predicted to travel. The road region on which the host vehicle is predicted to travel may be any road region on which the host vehicle is likely to travel. The second process is an additional process for improving the recognition accuracy. That is, the control unit allocates more resources to the area where the host vehicle is likely to travel. In other words, the control unit does not execute additional processing to improve the recognition accuracy in an area where the host vehicle is not likely to travel. Such an arrangement allows additional resources to be allocated to more important areas of the image.

[0017] The first area may include at least an area that is at a distance from the first vehicle that is equal to or greater than a predetermined value. The area that is farther away from the first vehicle than a predetermined distance has a lower resolution than other areas, and therefore the recognition accuracy is relatively low. Therefore, by performing additional processing on such areas, the overall recognition accuracy can be improved.

[0018] The control unit may also determine the first area in the image based on a planned route of the first vehicle. The planned route of the first vehicle may be determined based on information acquired from, for example, a navigation device or a control device that controls autonomous driving, which allows the direction of the vehicle to be determined in the image and the first area to be appropriately determined.

[0019] The control unit may also determine the first area in the image based on vehicle data acquired from the first vehicle. The vehicle data may include, for example, data indicating the state and behavior of the vehicle. The vehicle data may include data indicating the steering angle and the operation status of the turn signals. The control unit can estimate the planned route of the vehicle based on such data.

[0020] The control unit may also set the first area to include a second road area corresponding to a lane in which a second vehicle located near the first vehicle is traveling.

[0021] In some cases, it may be necessary to improve the recognition accuracy not only for the lane in which the host vehicle is traveling but also for the lane in which other vehicles located near the host vehicle are traveling. Therefore, the first area may be set to include the lane in which the second vehicle is traveling.

[0022] The second process may be, for example, a process in which resources for image recognition are increased compared to the first process, such as "performing recognition processing without downsampling the resolution" or "performing recognition processing without thinning out the frame rate."

[0023] Furthermore, if the first process is a process executed using a machine learning model, the second process may be a process of re-learning the machine learning model. For example, when image recognition is performed at each time step, as an object in the first region approaches the first vehicle, the object can be recognized more accurately. Therefore, the recognition results obtained at later time steps may be used as training data for input data at earlier time steps to retrain the machine learning model. This can improve the object recognition accuracy.

[0024] The second process may be a process of correcting the recognition result of the road area included in the first area by using an information source that is not used in the first process. For example, the second process may be a process of correcting the recognition results of the road area included in the first area using data output by a sensor other than an image sensor (e.g., radar, LiDAR, etc.) or data received from outside the vehicle.

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments.

[0026] (First embodiment) An overview of a vehicle system according to a first embodiment will be described. The vehicle system according to this embodiment includes a vehicle 1, an in-vehicle device 10 mounted on the vehicle 1, and a camera 20 mounted on the vehicle 1.

[0027] The problem to be solved by the system will be described with reference to FIGS. An in-vehicle device 10 mounted on a vehicle 1 recognizes the road area around the vehicle based on images captured by a camera 20, and executes control to make the vehicle 1 travel autonomously while utilizing the recognition results. The vehicle-mounted device 10 stores data (guidance data) for guiding the vehicle to the destination, and travels along the recognized road area in accordance with the guidance data.

[0028] A road area is typically an area in which the vehicle 1 can travel. A road area can be recognized by detecting road boundaries (road edges), but the objects of recognition are not limited to road edges. For example, lane boundary lines, lane centerlines, road centerlines, etc. may also be recognized.

[0029] In the example of Figure 1, assume that there are two road segments (segments A and B) that are made up of curves, and that vehicle 1 is traveling on these roads. Segment A is a curve with a radius r1, and segment B is a curve with a radius r2, where r2 is smaller than r1. In other words, segment B is a sharper curve than segment A.

[0030] 2 is an example of an image captured by the camera 20 of the vehicle 1 immediately before entering the illustrated curve. The image captures both segment A and segment B. However, since the image resolution is low in areas far from the vehicle, the recognition accuracy tends to be worse than in areas closer to the vehicle. For example, at the timing shown in the figure, the in-vehicle device 10 may recognize that "there is a series of curves with a radius r1 (both segments A and B have a radius r1)." In this case, the in-vehicle device 10 generates a driving trajectory based on the premise that "there is a series of curves with a radius r1," as indicated by reference numeral 1001 in FIG. 1.

[0031] However, in reality, the curve in segment B has a smaller radius than the curve in segment A. Since the image resolution is higher for closer road areas, as the vehicle 1 travels, the on-board device 10 can correctly recognize the curve radius of segment B. Therefore, if the travel trajectory indicated by reference numeral 1001 is corrected while the vehicle 1 is traveling, the turning rate may change suddenly in the middle of the curve, which is undesirable from the viewpoint of stable vehicle traveling.

[0032] Therefore, the in-vehicle device 10 according to this embodiment executes additional processing (hereinafter, additional processing) to improve the recognition accuracy for an area farther away from the vehicle (for example, area 2002 in FIG. 2). By executing the additional processing partially in this way, it is possible to improve the recognition accuracy while minimizing the increase in cost.

[0033] [Device configuration] FIG. 3 is a diagram showing an example of the configuration of the vehicle 1. The in-vehicle device 10 can be configured as a computer having a processor (CPU, GPU, etc.), a main memory device (RAM, ROM, etc.), and an auxiliary memory device (EPROM, hard disk drive, removable media, etc.). The auxiliary memory device stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that meet predetermined purposes, as described below, can be realized. However, some or all of the functions can be realized, for example, It may also be realized as a hardware module using a hardware circuit such as an ASIC or FPGA.

[0034] The in-vehicle device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and an input / output unit 14.

[0035] The control unit 11 is a computing unit that executes predetermined programs to realize various functions of the in-vehicle device 10. The control unit 11 can be realized by, for example, a hardware processor such as a CPU. The control unit 11 may also be configured to include RAM, ROM (Read Only Memory), cache memory, etc.

[0036] The control unit 11 is configured to have four software modules: a recognition unit 111, a generation unit 112, a correction unit 113, and a driving control unit 114. Each software module may be realized by the control unit 11 (such as a CPU) executing a program stored in the storage unit 12, which will be described later.

[0037] The recognition unit 111 acquires an image from the camera 20, which will be described later, and recognizes a road area included in the image. In this embodiment, the recognition unit 111 converts the acquired image into feature quantities and inputs the obtained feature quantities into a machine learning model stored in the storage unit 12. The machine learning model is a model specialized for recognizing road areas (hereinafter, referred to as a recognition model). This makes it possible to estimate the road area in the image. The road area recognition result is sent to the generation unit 112.

[0038] The generation unit 112 generates map data based on the road area recognition result performed by the recognition unit 111. The map data is a two-dimensional map or three-dimensional map that indicates a drivable area (road area) within the space in which the host vehicle is located. The generation unit 112 identifies the drivable area (road area) within the space based on the recognition result received from the recognition unit 111, and generates data (road area data) that indicates the identified road area. The generation unit 112 also generates map data that is a collection of road area data. The map data may be deleted each time it is used, or the generated map data may be stored and reused.

[0039] In this embodiment, additional processing may be performed on a road area that has already been recognized in order to improve accuracy, i.e., the map data that has been generated may be updated as the vehicle travels.

[0040] The correction unit 113 performs additional processing to improve the accuracy of a road area that has already been recognized. As described above, the resolution of a road area that is far away is low, while the resolution of a road area that is close is high. That is, the recognition accuracy of a road area that is far away is initially low, and as the vehicle 1 travels, the recognition accuracy gradually improves. In other words, the correct answer to the result of recognizing a road area in a certain time step may become clear in a later time step. Therefore, the correction unit 113 re-learns the machine learning model for recognizing the road area when the target road area approaches the vehicle.

[0041] FIG. 4 is a diagram illustrating this processing. At time t1 in the figure, it is assumed that there is a road area B near the vehicle and a road area A in the distance. The recognition model of the in-vehicle device 10 separately recognizes the road area A in the distance and the road area B in the vicinity. In this embodiment, the distant area refers to an area including a road area on which the host vehicle is predicted to travel, and is an area that is at a distance from the host vehicle that is equal to or greater than a predetermined value. Here, road area B is closer to the host vehicle. Therefore, the recognition accuracy is higher than that of road area A.

[0042] On the other hand, as the vehicle 1 travels and time t2 arrives, the in-vehicle device 10 becomes able to more accurately recognize the road area A. In this embodiment, at this timing, the correction unit 113 re-learns the recognition model. Specifically, the recognition result of road area A at time t2 (i.e., the highly accurate recognition result) is treated as the correct answer for the feature used for recognition at time t1. In other words, the recognition model is retrained using the feature corresponding to road area A at time t1 as input data and the recognition result at time t2 as training data.

[0043] The driving control unit 114 controls the autonomous driving of the vehicle based on the generated map data. The driving control unit 114 detects obstacles around the vehicle based on images captured by the camera 20, and controls the autonomous driving of the vehicle using data obtained as a result of the detection (hereinafter, environmental data) in combination with the generated map data. The environmental data indicates, for example, the number and positions of vehicles around the vehicle, and the number and positions of obstacles around the vehicle (e.g., pedestrians, bicycles, structures, buildings, etc.), but is not limited to these. Any object may be detected as long as it is necessary for autonomous driving. The environmental data is generated by a process different from the road area recognition process.

[0044] The driving control unit 114 drives the vehicle along the route indicated by the guidance data and in such a way that no obstacles enter a predetermined safety area centered on the vehicle. A known method can be adopted as a method for autonomously driving the vehicle. The guidance data is data for providing route guidance, and typically includes information for providing guidance on right and left turns that exist on the way to the destination, interchanges that should be used, and the like.

[0045] The storage unit 12 is a means for storing information, and is configured with storage media such as RAM, a magnetic disk, a flash memory, etc. The storage unit 12 stores programs executed by the control unit 11, data used by the programs, etc.

[0046] The storage unit 12 stores the map data, guidance data, recognition models, and the like described above.

[0047] The communication unit 13 is a wireless communication interface for connecting the in-vehicle device 10 to an in-vehicle network.

[0048] The input / output unit 14 is a unit that accepts input operations performed by a vehicle occupant and presents information to the occupant. In this embodiment, it is composed of a single touch panel display. That is, it is composed of a liquid crystal display and its control means, and a touch panel and its control means.

[0049] The specific hardware configuration of the in-vehicle device 10 may include omissions, substitutions, and additions of components as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, GPU, etc. Furthermore, input / output devices other than those illustrated (for example, an optical drive, etc.) may be added. Furthermore, the in-vehicle device 10 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same.

[0050] [Additional Processing Details] Next, the additional processing executed by the correction unit 113 will be described in detail.

[0051] FIG. 5 is a chart showing in more detail the flow of data when the in-vehicle device 10 re-learns the recognition model. First, at time t1, an image (referred to as a camera image) is acquired from the camera 20. The recognition unit 111 converts the camera image into feature quantities and recognizes a road area using a recognition model. In this case, the recognition unit 111 divides the feature quantities into a road area located near the host vehicle (hereinafter referred to as a near area) and a road area located at a distance from the host vehicle equal to or greater than a predetermined value (hereinafter referred to as a far area), and recognizes the road area for each area separately. The near area and the far area can be set at any position in the image, but both the near area and the far area include a road area where the host vehicle is predicted to travel, with the near area being located closer to the host vehicle. For example, the recognition unit 111 may recognize the lane on which the host vehicle is traveling and then set the near area and the far area on that lane. For example, in the example of FIG. 2, reference numeral 2001 can be set as the near area, and reference numeral 2002 can be set as the far area. As a result, data (road area data) including the recognition results is generated for each of the near area and the far area.

[0052] Next, consider a case where the vehicle continues traveling and time t2 arrives. In this example, time t2 is the timing when a road area that was located far from the vehicle at time t1 comes close to the vehicle. At time t2, the recognition unit 111 similarly recognizes road areas in the near area and the far area. Here, the recognition result for the near area at time t2 (reference numeral 502) and the recognition result for the far area at time t1 (reference numeral 501) are assumed to be the results of recognizing the same road area. However, in reality, the recognition accuracy of reference numeral 501 is lower. Therefore, the correction unit 113 compares reference numerals 501 and 502, and if the error is equal to or greater than a predetermined value, re-learns the recognition model.

[0053] The re-learning is performed using the feature quantity (reference numeral 503) corresponding to the distant region at time t1 as input data and the recognition result (reference numeral 502) at time t2 as training data, thereby improving the recognition model's ability to recognize the distant region at time t1.

[0054] Note that whether or not to perform re-learning may be dynamically determined based on the results of comparing the road area data indicated by reference numeral 501 with the road area data indicated by reference numeral 502. For example, if the difference in shape, area, curve radius, etc. between the road areas obtained as a result of recognition is equal to or greater than a predetermined value, it may be determined that re-learning is necessary. Even if the difference in the recognition results is large, re-learning may not be performed if it does not affect the safety of driving.

[0055] [Processing flow] Next, a description will be given of the flow of processing executed by the in-vehicle device 10. Fig. 6 is a flowchart of processing executed by the in-vehicle device 10. The processing shown in the figure is executed periodically while the vehicle 1 is traveling.

[0056] First, in step S11, the recognition unit 111 acquires an image from the camera 20.

[0057] Next, in step S12, the recognition unit 111 determines a road area in the image where the host vehicle is predicted to travel. The road area where the host vehicle is predicted to travel can be determined based on, for example, the following information. - The vehicle's driving lane determined by image recognition Location information obtained by the GPS module - Direction of travel obtained from gyro (heading direction) Steering angle Based on a combination of these pieces of information, the recognition unit 111 can determine the road area in the image toward which the vehicle is predicted to head.

[0058] The recognition unit 111 may also estimate that the vehicle will change course within a predetermined period of time based on other information. For example, if the turn signal of the vehicle is on, the recognition unit 111 may estimate that the vehicle will change lanes in the direction indicated by the turn signal within a predetermined period of time. Such a determination may be made based on, for example, CAN data flowing through the in-vehicle network. These data are examples of "vehicle data."

[0059] Next, in step S13, the recognition unit 111 sets a near area and a far area in the image acquired from the camera 20. Both the near area and the far area are set in a road area along which the host vehicle is predicted to travel. For example, both the near area and the far area may be areas including the lane along which the host vehicle is currently traveling. The recognition unit 111 may, for example, recognize the lane along which the host vehicle is traveling and then set the near area and the far area on the lane. For example, in the example of FIG. 2, reference numeral 2001 is the near area and reference numeral 2002 is the far area. The near area is an area where the road area is recognized by normal processing, and the far area is an area where additional processing is performed in addition to the normal processing. Therefore, it is preferable to set the far area as an area where recognition accuracy is expected to be low. For example, the far area can be an area that is farther than a predetermined distance from the vehicle and has a relatively low resolution. By setting the near area and the far area in this step, data including the recognition results (road area data) is generated for each of the near area and the far area.

[0060] Next, in step S14, the recognition unit 111 recognizes the road area for the area to be processed and generates road area data as a result. The road area data can be data that represents the area (road area) in which the vehicle can travel in a two-dimensional or three-dimensional space. If the area to be processed is divided into a near area and a far area, the recognition unit 111 generates road area data for each area.

[0061] Next, in step S15, the correction unit 113 determines whether or not there is an area to be reprocessed among the road areas processed in step S15. An area to be reprocessed is an area for which additional processing should be performed to improve recognition accuracy. For example, as described with reference to FIGS. 4 and 5, if the recognition accuracy of a road area performed on a distant area in a previous time step is low, it can be determined that reprocessing should be performed on that road area in the current time step. The low accuracy of a road area recognized in a previous time step can be determined by comparing road area data corresponding to the same road area between different time steps, for example, as shown by reference numerals 501 and 502 in FIG. 5. If there are no areas determined to have low recognition accuracy or areas where low recognition accuracy poses a safety problem, the correction unit 113 may determine that "there are no areas to be reprocessed."

[0062] If it is determined in step S15 that there is an area to be reprocessed, the process proceeds to step S16, and the correction unit 113 executes re-learning for the area. In step S16, for example, the recognition model is re-learned using the feature amount (reference numeral 503) used for estimation in the past time step as input data and the recognition result of the road area in the current time step (reference numeral 502) as training data.

[0063] As described above, the vehicle-mounted device 10 according to the first embodiment performs additional processing to improve recognition accuracy for an area included in an image captured by the vehicle-mounted camera, the area including at least a road area on which the vehicle is expected to travel. For road areas that are unlikely to be affected, processing to improve recognition accuracy is not performed. This makes it possible to improve recognition accuracy while minimizing increases in costs.

[0064] (Second embodiment) In the first embodiment, the recognition accuracy is improved by re-learning the recognition model in step S16. That is, the additional processing is a process of re-learning the recognition model using both data acquired in past time steps and data acquired in the current time step. On the other hand, the additional processing may be processing that improves recognition accuracy by using only the data obtained in the current time step.

[0065] The in-vehicle device 10 according to the second embodiment is an embodiment that determines, from a camera image, a road area on which the host vehicle is predicted to travel, and allocates more processing resources to the road area in real time.

[0066] Here, an additional example of a method for determining the road area on which the host vehicle is expected to travel will be described. The first method is to estimate the path of the host vehicle and then use the estimation result to determine the "area in which the host vehicle is predicted to travel." For example, if the host vehicle is traveling in a lane in which it is not possible to change lanes, it can be estimated that "the host vehicle will continue traveling in the same lane." In this case, for example, the area indicated by reference numeral 701 in FIG. 7 can be determined as the area in which the host vehicle is predicted to travel.

[0067] Furthermore, for example, when it is determined that the vehicle will change lanes within a predetermined period based on route information acquired in advance or vehicle information acquired in real time while traveling (for example, the operation status of the turn signals), the area corresponding to the adjacent lane may be set as the area in which the vehicle is predicted to travel. The route information may be acquired from a navigation device or the like, or from an ECU or the like that controls autonomous or semi-autonomous traveling.

[0068] The second method is to divide the area included in the camera image into areas where the vehicle may be traveling and areas where the vehicle may not be traveling, and treat the areas where the vehicle may be traveling as "areas where the vehicle is predicted to travel."

[0069] Fig. 8 is a diagram for explaining this method. In the illustrated example, an image captured by an on-board camera is divided into an area indicated by reference numeral 801 and an area indicated by reference numeral 802. Of these, the area indicated by reference numeral 802 is an area in which the host vehicle will not travel, so it does not matter if the recognition accuracy is low. In this way, the area included in the camera image may be divided into an area in which the host vehicle may travel and an area in which it is not, and the former may be treated as an area in which the host vehicle is predicted to travel.

[0070] In the second embodiment, the correction unit 113 executes the additional processing in real time. For example, if the frame rate of the image output by the camera is 60 frames per second, it is possible to say, "perform recognition processing for area 802 at 30 frames per second, and for area 801 at 60 frames per second." Also, if the resolution of the image output by the camera is downsampled, it is possible to say, "perform recognition processing for area 802 after downsampling, and for area 801, perform recognition processing without downsampling."

[0071] The correction unit 113 may also correct the recognition result of the road area using information other than the camera image. For example, the correction unit 113 may perform processing to correct the recognition result using a sensor other than the on-board image sensor.

[0072] 9 is a flowchart of the process executed by the in-vehicle device 10 in the second embodiment. The process shown in the figure is executed periodically while the vehicle 1 is traveling.

[0073] First, in step S21, the recognition unit 111 acquires an image from the camera 20. Next, in step S22, the recognition unit 111 determines a road area in the image where the host vehicle is predicted to travel. The road area where the host vehicle is predicted to travel can be determined by the same method as in step S12. The road area where the host vehicle is predicted to travel is a road area for which recognition accuracy is to be improved. In the following description, the road area where the host vehicle is expected to travel is referred to as the "planned travel area," and other areas are referred to as the "non-travel area."

[0074] Next, in step S23, it is determined whether or not to improve the recognition accuracy for the planned travel area. For example, when there is a lot of traffic and it is preferable to have a large safety margin, it is preferable to improve the recognition accuracy for the planned travel area. Also, when the likelihood (reliability) output by the recognition model is equal to or less than a predetermined value, it is preferable to improve the recognition accuracy. If the determination in this step is negative, the process proceeds to step S24, whereas if the determination in this step is positive, the process proceeds to step S25.

[0075] In step S24, the recognition unit 111 recognizes the road area for the area to be processed by the same process as in step S14, and generates road area data as a result. In this embodiment, the area to be processed is not divided into a near area and a far area, so the recognition unit 111 generates single road area data.

[0076] In step S25, in addition to the process described in step S24, additional processing is performed to improve the recognition accuracy of the intended travel area. The additional processing may be processing to recognize the road area using a method different from the process performed in step S24 and correct the recognition result based on the result.

[0077] The additional processing may also be processing to correct the road area recognition result using information sources not used in step S24. For example, if messages transmitted from surrounding vehicles or roadside devices are available, the processing to correct the recognition result may be performed using these messages as additional information sources. The processing to correct the recognition result may also be performed based on data output by sensors other than the on-board camera. In either case, additional resources are invested in recognizing road areas in the planned travel area compared to other areas.

[0078] As described above, in the second embodiment, road areas are divided into those on which the host vehicle is predicted to travel and those on which it is not predicted, and additional processing is performed to improve the recognition accuracy of the road areas on which the host vehicle is predicted to travel. This makes it possible to allocate resources for improving the recognition accuracy of road areas only to those areas on which the host vehicle is likely to travel.

[0079] In the example of FIG. 8, additional processing is performed on all areas (area 801) where the host vehicle may travel. On the other hand, additional processing may be performed only on areas where the recognition accuracy is relatively low. For example, as shown in FIG. 10, areas where the host vehicle may travel can be divided into an area (area 801A) that is closer to the host vehicle and an area (area 801B) that is farther away from the host vehicle. Area 801A is an area where recognition can be performed with a predetermined accuracy without additional processing. On the other hand, area 801B is an area where recognition can be performed with a predetermined accuracy without additional processing. In such a case, the in-vehicle device 10 may execute the additional process only for the distant area 801B. This makes it possible to save resources required for the additional process.

[0080] (Third embodiment) In the first and second embodiments, additional processing to improve recognition accuracy was performed only on the road area where the host vehicle is predicted to travel (planned travel area). However, the road area where it is beneficial to improve recognition accuracy is not necessarily the lane in which the host vehicle is traveling. For example, when another vehicle is traveling parallel to the host vehicle in an adjacent lane, additional processing to improve recognition accuracy may be performed on the lane in which the other vehicle is traveling. This makes it possible, for example, to accurately predict the behavior of the other vehicle.

[0081] In the third embodiment, in addition to the processes described in the first and second embodiments, the on-vehicle device 10 recognizes another vehicle located near the host vehicle and performs additional processing on a road area (second road area) corresponding to the lane on which the other vehicle is traveling. Fig. 11 is a flowchart of this processing. The illustrated flowchart is executed after the processing shown in Fig. 6 or after the processing shown in Fig. 9, or at any timing.

[0082] First, in step S31, the recognition unit 111 detects the presence of another vehicle traveling in a lane other than the lane the host vehicle is traveling in. The presence of the other vehicle may be detected via the camera 20, or may be detected using an on-board sensor other than a camera (for example, an ultrasonic sensor, LiDAR, etc.). Next, in step S32, the correction unit 113 determines whether to perform additional processing on the lane in which the detected other vehicle is traveling. For example, in cases where the in-vehicle device 10 has sufficient resources or where the other vehicle is close to the host vehicle and it is preferable to predict the behavior of the vehicle with higher accuracy, the determination in this step is affirmative.

[0083] If the determination in step S32 is affirmative, the process proceeds to step S33, where the recognition unit 111 executes additional processing on the area corresponding to the lane the detected other vehicle is traveling in. If the determination in step S32 is negative, the process ends.

[0084] According to the third embodiment, additional processing is performed to improve the recognition accuracy for the lane in which the other vehicle is traveling, which has the effect of enabling the behavior of the other vehicle to be predicted with high accuracy.

[0085] The vehicle-mounted device 10 may detect all other vehicles in the image captured by the camera 20, and then perform the additional process for all lanes in which other vehicles exist.

[0086] (Other variations) The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate within the scope that does not deviate from the gist of the disclosure. For example, the processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0087] Although the embodiment has been described with reference to an example in which the in-vehicle device 10 recognizes the road area, the recognition of the road area may be performed by a server device installed in a location different from the vehicle. In this case, a camera image may be transmitted from the in-vehicle device to the server device, and the server device may transmit the result of the recognition of the road area to the in-vehicle device.

[0088] In addition, in the description of the embodiment, the road itself is used as an example of the object of recognition. The recognition target does not necessarily have to be the road itself. For example, an object on the road or an object moving on the road may be recognized.

[0089] In the embodiment, the road area where the host vehicle is predicted to travel is determined, and additional processing is performed on that road area. However, the road area to be subjected to additional processing does not necessarily have to be an area where the host vehicle is predicted to travel, as long as it is an area where the host vehicle may travel. For example, additional processing may be performed on a lane adjacent to the lane in which the host vehicle is traveling, even if the turn signal is not activated.

[0090] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0091] The present disclosure can also be realized by providing a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer via a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Examples of non-transitory computer-readable storage media include any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0092] 1. Vehicle 10...In-vehicle equipment 11 Control section 12...Storage section 13. Communications Department 14...Input / output section

Claims

1. performing a first process of recognizing a predetermined object based on an image acquired by an on-board camera of the first vehicle; performing a second process for improving recognition accuracy on a first area, which includes at least a road area on which the first vehicle is predicted to travel, among areas included in the image; a control unit that executes the following: Information processing device.

2. The first area includes at least an area that is a distance from the first vehicle that is equal to or greater than a predetermined value. The information processing device according to claim 1 .

3. the control unit determines the first area in the image based on a planned route of the first vehicle. The information processing device according to claim 1 .

4. the control unit determines the first area in the image based on vehicle data acquired from the first vehicle. The information processing device according to claim 1 .

5. the control unit sets the first area to include a second road area corresponding to a lane on which a second vehicle located near the first vehicle is traveling; The information processing device according to claim 1 .

6. The second process is a process in which resources for image recognition are increased compared to the first process. The information processing device according to claim 1 .

7. the first processing is a processing executed using a machine learning model, the second process is a process of re-learning the machine learning model when an object included in the first area approaches the first vehicle. The information processing device according to claim 1 .

8. the first processing is processing for recognizing a road area, The second processing is processing for correcting the recognition result of the road area included in the first area by using an information source not used in the first processing. The information processing device according to claim 1 .

9. An information processing method executed by an information processing device capable of communicating with a first vehicle, performing a first process of recognizing a predetermined object based on an image acquired by an on-board camera of the first vehicle; performing a second process for improving recognition accuracy on a first area, which includes at least a road area on which the first vehicle is predicted to travel, among areas included in the image; An information processing method, including:

10. The first area includes at least an area that is a distance from the first vehicle that is equal to or greater than a predetermined value. The information processing method according to claim 9.

11. determining the first region in the image based on a planned route of the first vehicle; The information processing method according to claim 9.

12. determining the first region in the image based on vehicle data acquired from the first vehicle; The information processing method according to claim 9.

13. setting the first area to include a second road area corresponding to a lane in which a second vehicle located near the first vehicle is traveling; The information processing method according to claim 9.

14. The second process is a process in which resources for image recognition are increased compared to the first process. The information processing method according to claim 9.

15. the first processing is a processing executed using a machine learning model, the second process is a process of re-learning the machine learning model when an object included in the first area approaches the first vehicle. The information processing method according to claim 9.

16. the first processing is processing for recognizing a road area, The second processing is processing for correcting the recognition result of the road area included in the first area by using an information source not used in the first processing. The information processing method according to claim 9.

17. A program for causing a computer to execute the information processing method according to any one of claims 9 to 16.

Images