Information processing device, information processing method and storage medium
By calculating the confidence level of the lane boundary region in a machine learning model, a model for outputting the coordinates and confidence level of the lane boundary is generated, which solves the problem of insufficient lane boundary recognition accuracy in existing technologies and achieves higher-precision autonomous driving.
Patent Information
- Application Number
- CN202610615982.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies using machine learning models for lane boundary line recognition suffer from insufficient detection accuracy due to the complexity of road environments, especially when lane boundary lines are obscured or misidentified during vehicle movement, which affects the safety of autonomous driving.
By acquiring camera images, the area presumed to contain a target is identified, and the confidence scores of multiple coordinates are calculated to generate a machine learning model. This model outputs the coordinates along with the confidence scores to improve the accuracy of lane boundary line recognition.
This improves the accuracy of lane boundary line recognition by machine learning models in complex road environments, ensuring the safety and accuracy of autonomous driving.
Smart Images

Figure CN123071971A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to vehicle technology. Background Technology
[0002] There are technologies that simultaneously sense the road environment and identify lane boundary lines in real time (e.g., Patent Document 1).
[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2016-194815 Summary of the Invention
[0004] The problem that the invention aims to solve The purpose of this disclosure is to improve the accuracy of object detection using machine learning models.
[0005] Solution for solving the problem One aspect of this disclosure is an information processing apparatus having a control unit that performs the following actions: acquiring a camera image captured by a camera mounted on a moving body; determining a first region in the camera image presumed to contain a first target; calculating a confidence level of the existence of the first target for each of a plurality of coordinates corresponding to the first region; and performing learning by taking a set of the plurality of coordinates corresponding to the first region as input data, and taking the set of confidence levels corresponding to each of the plurality of coordinates and the set of coordinates of the first target actually existing in a specified space as correct answer data, thereby generating a machine learning model, wherein the machine learning model is a model that outputs the coordinates of the first target presumed to exist in the specified space along with the confidence levels.
[0006] One aspect of this disclosure is an information processing method executed by an information processing device. The information processing method includes: acquiring a camera image captured by a camera mounted on a moving body; determining a first region in the camera image presumed to contain a first target; calculating a confidence level of the existence of the first target for each of a plurality of coordinates corresponding to the first region; and using a set of the plurality of coordinates corresponding to the first region as input data, and the set of confidence levels corresponding to each of the plurality of coordinates and the set of coordinates of the first target actually existing within a specified space as correct answer data, thereby enabling a machine learning model to learn. The machine learning model is a model that outputs the coordinates within the specified space presumed to contain the first target along with the confidence levels.
[0007] In addition, as another option, one could provide a program for causing a computer to perform the aforementioned information processing method, or a computer-readable storage medium that non-temporarily stores the program.
[0008] Invention Effects According to this disclosure, the accuracy of object detection using machine learning models can be improved. Attached Figure Description
[0009] Figure 1 This is a diagram used to illustrate the subject matter of this disclosure.
[0010] Figure 2 This is a diagram used to illustrate the subject matter of this disclosure.
[0011] Figure 3 This is a schematic diagram of the vehicle system disclosed herein.
[0012] Figure 4 This diagram illustrates the configuration of vehicle 1 and onboard device 10.
[0013] Figure 5 This diagram illustrates the configuration of server device 30.
[0014] Figure 6 This is a diagram illustrating the data flow during the learning phase.
[0015] Figure 7 This diagram illustrates the data flow during the estimation phase.
[0016] Figure 8 This is a flowchart of the processes performed by the on-board unit 10 during the learning phase.
[0017] Figure 9 This is a flowchart of the processes performed by the server device 30 during the learning phase.
[0018] Figure 10 This is a flowchart of the process performed by the on-board unit 10 during the presumption phase.
[0019] Explanation of reference numerals in the attached figures: 1: Vehicle; 10: Vehicle-mounted devices; 11: Control Department; 12: Storage Department; 13: Ministry of Communications; 14: Input / output section; 15: Location Information Acquisition Department. Detailed Implementation
[0020] In recent years, research has been conducted on autonomous driving systems that enable vehicles to navigate along pre-defined routes. Autonomous vehicles determine their position and posture, for example, by comparing pre-stored road maps with results obtained from sensing the road environment.
[0021] For example, an autonomous vehicle identifies the position of a designated target (typically a lane boundary line) on the road based on images acquired by multiple onboard cameras (hereinafter referred to as camera images), and generates a predetermined trajectory based on the inferred position of the vehicle based on the identification results.
[0022] The location of targets such as lane boundary lines can be identified using image processing techniques. Figure 1 (A) is an example of a road view captured by a camera positioned in front of the vehicle. The road shown has three lanes, which are divided by four white lines (lane boundary lines). Figure 1 (B) is a diagram showing the accurate identification results of lane boundary lines. Autonomous vehicles, for example, can identify the existence of three lanes on a road by recognizing four white lines as lane boundary lines.
[0023] On the other hand, when attempting to identify lane boundary lines solely through image processing, misidentification may occur. For example, in cases where the illustrated road includes white objects such as guardrails, side walls, and signposts, these white objects may sometimes be incorrectly identified as lane boundary lines. Figure 1 (C) is an example of simply extracting white objects from an image. In this case, for example, it might be misidentified as having five lanes. When lane boundaries are misidentified, it might be interpreted as the presence of lanes that do not exist, thus compromising driving safety.
[0024] In addition, Figure 1 In the example, the lane boundary lines are depicted with straight lines. However, in reality, lane boundary lines are recognized as a collection of multiple pixels. Therefore, depending on the quality of the image, lanes may sometimes be difficult to identify clearly.
[0025] Therefore, attempts are underway to identify targets using machine learning.
[0026] For example, a machine learning model can be used to identify lane boundary lines with high accuracy based on real-time camera images (e.g., to obtain vector images representing the positions of lane boundary lines). This machine learning model is a model that learns by taking images acquired through an onboard camera as input data and the accurate position information of the lane boundary lines as the correct answer data.
[0027] However, this method also has problems. For example, ... Figure 2 As shown in the diagram, assume there are other vehicles on the road traveled by the vehicle acquiring the learning image. In this case, the lane boundary lines are obscured in the area shown by the dashed lines, but they are present in the correct answer data. This may cause inconsistencies between the input data (image) used for learning and the correct answer data.
[0028] Thus, due to the presence of various targets in a real road environment, it is possible that even using images collected in such an environment to train a machine learning model may not yield sufficient accuracy.
[0029] The information processing apparatus disclosed herein will solve such problems.
[0030] One information processing apparatus disclosed herein includes a control unit that performs the following actions: acquiring a camera image captured by a camera mounted on a moving body; determining a first region in the camera image where a first target is presumed to exist; calculating a confidence level of the existence of the first target for each of a plurality of coordinates corresponding to the first region; and generating a machine learning model by using a set of a plurality of coordinates corresponding to the first region as input data, and using a set of confidence levels corresponding to each of the plurality of coordinates and a set of coordinates of the first target actually existing in a specified space as correct answer data, wherein the machine learning model is a model that outputs the coordinates of the first target presumed to exist in the specified space along with the confidence levels.
[0031] The control unit determines a first region presumed to contain a first target based on images captured by a camera mounted on the moving body. For example, if the first target is a white lane boundary line, the control unit can extract a region consisting of a set of white pixels from the road surface reflected in the image as the first region.
[0032] Furthermore, the control unit calculates the confidence level of the existence of the first target for each of the multiple coordinates corresponding to the first region. The confidence level is a value representing the likelihood that the first target exists in the coordinates of the object.
[0033] Confidence levels can be calculated, for example, based on the results obtained by simply parsing the image. For instance, if the first target is a lane boundary line, and the first region consists of multiple bar-shaped regions extending toward the same coordinate (vanishing point), it can be said that the confidence level is high that a lane boundary line exists at multiple coordinates within that region. Conversely, if the first region is shaped in a way that cannot be presumed to be a lane boundary line (e.g., a square in the image), it can be said that the confidence level is low that a lane boundary line exists at multiple coordinates within that region.
[0034] Furthermore, the control unit takes a set of multiple coordinates corresponding to the first region as input data, and uses the set of confidence scores corresponding to each of these multiple coordinates and the set of coordinates of the first target that actually exists within the specified space as correct answer data to perform learning. Thus, a machine learning model can be generated, wherein this machine learning model outputs the coordinates within the specified space that are presumed to indicate the existence of the first target, along with the confidence scores.
[0035] It should be noted that the "multiple coordinates corresponding to the first region" can be coordinates on the camera image, or coordinates obtained by projecting the first region set on the camera image onto a specified space. Furthermore, the specified space can refer to a three-dimensional space with the moving body as the reference.
[0036] Alternatively, the first target could be the lane boundary line, and the control unit could calculate the confidence level for each of the plurality of coordinates based on the shape of the first region. Since the lane boundary line extends in a fixed direction on the image, the confidence level can be estimated by utilizing its shape.
[0037] Alternatively, the control unit may determine a second region in the camera image that is presumed to contain a second target different from the first target, and if the camera image contains coordinates belonging to both the first and second regions, calculate a relatively low confidence level for those coordinates compared to the case where the target belongs only to the first region.
[0038] For example, in an image, where a passenger vehicle traveling nearby overlaps with a lane boundary line, the area presumed to "have a lane boundary line" sometimes overlaps with the area presumed to "have a passenger vehicle." Furthermore, in such cases, the area where a passenger vehicle is present is sometimes learned as part of the lane boundary line.
[0039] Therefore, in such a case, it is also possible to calculate a relatively low confidence level for the coordinates of regions belonging to both parties compared to the case where the region belongs to only one party. The control unit can determine the second region through independent image analysis processing.
[0040] Alternatively, the control unit may determine a third region with defined visual characteristics in the camera image, and in the confidence calculation, if there are coordinates belonging to both the first region and the third region, calculate a relatively low confidence level for those coordinates compared to the case where the coordinates belong only to the first region.
[0041] The third region can be defined as an area where the average brightness exceeds (or falls below) a threshold, or an area where the average chromaticity is below a threshold, where the accuracy of image recognition may be reduced. For example, although the existence of lane boundary lines can be inferred, an area overexposed due to backlighting or other factors may become a third region. The control unit can determine the third region through independent image analysis processing. The third region can be defined as an area where the statistical value of at least one of brightness, chromaticity, and hue meets a specified threshold.
[0042] Hereinafter, embodiments of the present disclosure will be described based on the accompanying drawings. The configurations of the following embodiments are illustrative, and the present disclosure is not limited to the configurations of the embodiments.
[0043] (First Implementation) An overview of the vehicle system according to the first embodiment will be described. The vehicle system of this embodiment is configured to include a vehicle 1 and a server device 30.
[0044] Reference Figure 3 Provide an overview of the system.
[0045] Vehicle 1 is a networked vehicle capable of autonomous driving. Vehicle 1 has an onboard device 10 and multiple cameras 20. The onboard device 10 has the function of identifying the position of lane boundary lines using images acquired by the multiple cameras 20 and machine learning models stored in a storage device, and controlling the autonomous driving of vehicle 1 based on the identification results.
[0046] Server device 30 is a device that generates machine learning models for use by vehicle device 10 through machine learning.
[0047] The vehicle-mounted device 10 is configured to perform: a phase (learning phase) in which data is collected for the machine learning model to learn and provided to the server device 30; and a phase (prediction phase) in which the learned machine learning model generated by the server device 30 is used to perform autonomous driving.
[0048] During the learning phase, the vehicle-mounted device 10 collects images via multiple cameras 20 to enable the machine learning model to learn, generates a dataset containing the images, and sends it to the server device 30.
[0049] Server device 30 performs machine learning model learning based on the dataset and provides the obtained machine learning model to vehicle device 10.
[0050] The vehicle-mounted device 10 stores the machine learning model that has been learned from the server device 30, and autonomously drives by inputting images acquired by multiple cameras 20 into the machine learning model to identify the position of the lane boundary line.
[0051] In the following description, the images captured by the multiple cameras 20 of vehicle 1 are referred to as "camera images". Furthermore, the aforementioned machine learning model is referred to as the "prediction model". The prediction model is a model that predetermines the position of the lane boundary line based on the camera images (or based on the feature quantities of the camera images).
[0052] [Device Composition] Figure 4 This is a diagram illustrating an example of the configuration of vehicle 1. Vehicle 1 is configured to include an onboard unit 10 and multiple cameras 20.
[0053] The vehicle-mounted device 10 can be configured as a computer with a processor (CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc.), main storage (RAM (Random Access Memory), ROM (Read Only Memory), etc.), and auxiliary storage (EPROM (Erasable Programmable Read Only Memory), hard disk drive, removable media, etc.). The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and can perform various functions (software modules) for the specified purposes as described below by executing the programs stored in the auxiliary storage device. However, some or all of the functions can also be implemented as hardware modules using hardware circuits such as ASIC (Application Specific Integrated Circuit) and FPGA (Field-Programmable Gate Array).
[0054] The vehicle-mounted device 10 is configured to include a control unit 11, a storage unit 12, a communication unit 13, an input / output unit 14, and a location information acquisition unit 15.
[0055] The control unit 11 is a computing component that executes prescribed programs to implement various functions of the vehicle-mounted device 10. The control unit 11 can be implemented, for example, by a hardware processor such as a CPU. Furthermore, the control unit 11 may also be configured to include RAM, ROM (Read Only Memory), cache memory, etc.
[0056] The control unit 11 is configured to have three software modules: a collection unit 111, an estimation unit 112, and a driving control unit 113. Each software module can be implemented by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 12, which will be described later.
[0057] The collection unit 111 collects images for the inference model to learn, generates a dataset for learning based on the collected images, and sends it to the server device 30.
[0058] Furthermore, the learned estimation model is received from the server device 30 and stored in the storage unit 12, which will be described later. The method for generating the dataset for learning will be described later.
[0059] The estimation unit 112 estimates the position of the lane boundary line based on the estimation model stored in the storage unit 12 and the images acquired by the multiple cameras 20. In this embodiment, the estimation unit 112 converts the acquired images into feature quantities and inputs the obtained feature quantities into the estimation model stored in the storage unit 12. The estimation model is a model specifically designed for lane boundary line recognition, and the estimation model outputs the position information of the lane boundary line (specifically, the coordinates of the lane boundary line in a three-dimensional coordinate system based on the vehicle).
[0060] Therefore, the estimation unit 112 can identify the position of the lane boundary lines around the vehicle. The identification result of the lane boundary lines is sent to the driving control unit 113.
[0061] The driving control unit 113 controls the autonomous driving of the vehicle based on the lane boundary line recognition results obtained from the estimation unit 112. For example, the driving control unit 113 uses the lane boundary line recognition results and obstacle recognition results generated by the estimation unit 112 to control the autonomous driving of the vehicle. For instance, the driving control unit 113 causes the vehicle to drive along the lane in a manner that prevents obstacles from entering a predetermined safe area centered on the vehicle.
[0062] It should be noted that this embodiment focuses on the method for identifying lane boundary lines, but obstacle identification can also be performed by any method.
[0063] Storage unit 12 is a unit for storing information and is composed of storage media such as RAM, disk, and flash memory. Storage unit 12 stores programs executed by control unit 11, data used by those programs, etc. Furthermore, storage unit 12 stores the aforementioned estimated model.
[0064] An inference model is a machine learning model that infers the position of lane boundary lines based on feature quantities, where these feature quantities are obtained from multiple camera images. The learning method of the inference model will be described later.
[0065] The communication unit 13 is a communication interface for connecting the vehicle-mounted device 10 to an external network. The communication unit 13 can be configured to include a module for cellular communication, or it can provide a connection to an external network via a cellular communication network.
[0066] The input / output unit 14 is a component that receives input operations from the occupants of the vehicle and presents information to them. In this embodiment, the input / output unit 14 is composed of a touch panel display. That is, it consists of a liquid crystal display and its control unit, a touch panel and its control unit.
[0067] The location information acquisition unit 15 is a component that acquires the location information of the vehicle. The location information acquisition unit 15 may include, for example, a GPS (Global Positioning System) antenna and a positioning module for measuring the location information. The GPS antenna is an antenna that receives positioning signals transmitted from positioning satellites (also known as GNSS (Global Navigation Satellite System) satellites). The positioning module is a module that calculates the location information based on the signals received through the GPS antenna.
[0068] It should be noted that, regarding the specific hardware configuration of the vehicle-mounted device 10, constituent elements can be appropriately omitted, substituted, or added depending on the implementation method. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be composed of microprocessors, FPGAs, GPUs, etc. In addition, input / output devices other than those shown in the examples (such as optical drivers) may also be added. Furthermore, the vehicle-mounted device 10 may be composed of multiple computers. In this case, the hardware configuration of each computer may be the same or different.
[0069] Camera 20 is a camera installed in vehicle 1. Figure 4 The example shows a single camera 20, but the vehicle 1 is configured to have multiple cameras 20 mounted in different locations. The multiple cameras 20 can also be mounted in locations that allow them to capture images of the entire surroundings of the vehicle 1.
[0070] Figure 5 This diagram illustrates an example of the configuration of server device 30. Server device 30 is configured to include a control unit 31, a storage unit 32, and a communication unit 33.
[0071] The control unit 31 is a computing component that executes prescribed programs to implement various functions of the server device 30. The control unit 31 can be implemented, for example, by a hardware processor such as a CPU.
[0072] The control unit 31 is configured to have a learning unit 311 as a software module. This software module can be implemented by the control unit 31 (CPU, etc.) executing a program stored in the storage unit 32, which will be described later.
[0073] The learning unit 311 receives a dataset from the vehicle-mounted device 10 for learning the estimation model and uses the dataset to learn the estimation model. The server device 30 stores a three-dimensional map (lane map) that records the accurate location information of the lane boundary lines. The learning unit 311 uses the dataset received from the vehicle-mounted device 10 and the location information of the lane boundary lines recorded in the lane map to learn the estimation model (described later).
[0074] Storage unit 32 is a unit for storing information and is composed of storage media such as RAM, disk, and flash memory. Storage unit 32 stores programs executed by control unit 31, data used by those programs, etc. In addition, storage unit 32 stores the learned estimation model and lane map.
[0075] The communication unit 33 is a communication interface used to connect the server device 30 to an external network.
[0076] [Learning phase of the inference model] Next, the learning method of the inference model will be explained.
[0077] Figure 6 This diagram illustrates the data flow during the stage of generating the learned inference model (learning stage). It should be noted that the left side of the dashed line represents the processing performed by the vehicle-mounted device 10 (collection unit 111), and the right side of the dashed line represents the processing performed by the server device 30 (learning unit 311).
[0078] First, the vehicle-mounted device 10 acquires camera images for learning purposes from multiple cameras 20 (e.g., images taken by a vehicle 1 traveling on an actual road).
[0079] The vehicle-mounted device 10 performs prescribed processing on the acquired camera images, converting the images into feature quantities. In this embodiment, it is assumed that the feature quantity refers to the set of two-dimensional coordinates of pixels in the camera image that are presumed to represent lane boundary lines (pixels within the "first region" in this disclosure, such as white pixels). These coordinates are presented in a two-dimensional coordinate system based on the camera.
[0080] Furthermore, the collection unit 111 integrates the feature quantities corresponding to each camera. This integration can be achieved, for example, by projecting a set of two-dimensional coordinates representing the white line obtained from the images of each camera onto a common three-dimensional space. The projection destination space is a three-dimensional space with a coordinate system (vehicle coordinate system) based on vehicle 1. This space is called the feature quantity space. The projected feature quantities become a set of three-dimensional coordinates representing the location of the lane boundary line in the feature quantity space.
[0081] Here, the projected feature quantities (the set of three-dimensional coordinates in the feature quantity space) are referred to as the first data.
[0082] Unlike the processing described above, the collection unit 111 calculates the confidence level of the existence of the lane boundary line for each two-dimensional coordinate of the pixels contained in the first region.
[0083] Here, as examples of methods for calculating the confidence level of each coordinate, the following three methods are shown.
[0084] (A) Region-based shape method For example, if the first region is composed of a combination of multiple rod-shaped regions extending in a specific direction (vanishing point), it can be said that there is a high confidence that a lane boundary line exists among multiple coordinates within that region. In this case, the collection unit 111 can calculate a confidence level higher than a predetermined value for the coordinates within that region.
[0085] Conversely, if the first region is formed into a shape such as a square or a circle, which is difficult to represent a lane boundary line, it can be said that the confidence level of the existence of a lane boundary line at multiple coordinates within that region is low. In this case, the collection unit 111 can calculate a confidence level lower than a predetermined value for the coordinates within that region.
[0086] (B) Target-based overlapping method For example, the collection unit 111 determines, in the camera image of the object, an area (second area) presumed to contain a target different from the lane boundary line (e.g., other vehicles, pedestrians, bicycles, structures, puddles, snow, etc.).
[0087] If the camera image contains coordinates belonging to both the first and second regions, then regarding those coordinates, we can say that the confidence level of the lane boundary line's existence is lower compared to coordinates belonging only to the first region. Conversely, if the camera image contains coordinates belonging only to the first region, then regarding those coordinates, we can say that the confidence level of the lane boundary line's existence is higher compared to coordinates belonging to both the first and second regions.
[0088] Therefore, for example, if each of the multiple coordinates within the first region belongs only to the first region, the collection unit 111 can calculate a confidence level higher than a predetermined value for the coordinates of the object. Furthermore, if each of the multiple coordinates within the first region belongs to both the first and second regions, the collection unit 111 can calculate a confidence level lower than a predetermined value for the coordinates of the object.
[0089] (C) Methods based on visual features on images For example, the collection unit 111 determines a region (third region) with prescribed visual characteristics in the camera image of the object.
[0090] The third region can be defined as areas where the average brightness exceeds (or falls below) a threshold, or areas where the average chroma falls below a threshold, where the accuracy of image recognition may be reduced. In addition, regions where the statistical values for at least one of brightness, chroma, and hue meet specified conditions can also be designated as the third region.
[0091] If the camera image contains coordinates belonging to both the first and third regions, then regarding those coordinates, we can say that the confidence level of the lane boundary line's existence is lower compared to coordinates belonging only to the first region. Conversely, if the camera image contains coordinates belonging only to the first region, then regarding those coordinates, we can say that the confidence level of the lane boundary line's existence is higher compared to coordinates belonging to both the first and third regions.
[0092] Therefore, for example, if each of the multiple coordinates within the first region belongs only to the first region, the collection unit 111 can calculate a confidence level higher than a predetermined value for the coordinates of the object. Furthermore, if each of the multiple coordinates within the first region belongs to both the first and third regions, the collection unit 111 can calculate a confidence level lower than a predetermined value for the coordinates of the object.
[0093] It should be noted that the vehicle-mounted device 10 may have a unit (e.g., a software module) for detecting regions in an image where a specified target exists (e.g., regions reflecting other vehicles) or regions with specified features (e.g., regions that produce halos).
[0094] The confidence score for each coordinate, calculated as described above, is projected onto the feature space in the same way as the feature quantity. It should be noted that when multiple confidence scores are projected onto the same coordinate, an average or weighted average can be calculated. Thus, the confidence score information for each coordinate is integrated.
[0095] Here, the set of projected confidence scores (confidence scores for each three-dimensional coordinate in the feature space) is called the second data.
[0096] The vehicle-mounted device 10 sends the set of the first data and the second data as a learning dataset to the server device 30.
[0097] It should be noted that at this time, the on-board device 10 sends the location information of vehicle 1 (e.g., latitude and longitude in the World Geodetic System) to the server device 30. This is because the feature space is represented by a local coordinate system (a coordinate system based on vehicle 1). Therefore, in order to obtain the correct answer data (accurate location information of the lane boundary lines) for learning the inference model, it is necessary to teach the server device 30 the position of the vehicle in the global coordinate system (e.g., the World Geodetic System).
[0098] The server device 30, which receives the dataset for learning from the vehicle-mounted device 10, obtains the correct answer data corresponding to the feature space. For example, the server device 30 uses the location information received from the vehicle 1 to overlap the pre-stored lane map with the coordinates in the feature space and obtains the accurate location information of the lane boundary lines in the feature space (hereinafter referred to as third data). The third data is the set of coordinates of the lane boundary lines that actually exist in the feature space.
[0099] Then, the server device 30 performs a process that uses the first data as input data and the second and third data as correct answer data to enable the inference model to learn.
[0100] The inference model outputs a set of coordinates of lane boundary lines presumed to be within the feature space (corresponding to the third data) and a set of confidence scores for each coordinate (corresponding to the second data) as the inference results. During learning, the comparison results of these inference results with the correct answer data (the second and third data) are fed back into the inference model, and the parameters are updated to reduce the difference between the inference results and the correct answer data.
[0101] Through the processing described above, when a set of three-dimensional coordinates in the feature space is input, a machine learning model can be obtained that outputs the estimated lane boundary line position information (set of coordinates) in the vehicle coordinate system along with confidence information. The generated estimated model is distributed from server device 30 to vehicle device 10. The distribution of the estimated model can be performed online or offline.
[0102] [Presumption processing implemented by the presumption model] The process of estimating the position of lane boundary lines using the estimation model obtained through the processing described above will be explained.
[0103] Figure 7This diagram illustrates the method by which the on-board device 10 uses a learned estimation model to estimate the position of the lane boundary line. For example... Figure 7 As shown, the positional information of lane boundary lines can be obtained by inputting feature quantities obtained from multiple images acquired by an onboard camera into the estimation model. The method for generating the feature quantities is the same as in the learning phase. That is, in the estimation phase, the feature quantities are integrated in the same way as in the learning phase, and the results are input into the estimation model.
[0104] The estimation model takes a set of three-dimensional coordinates in the feature space as input and outputs a set of coordinates that are estimated to actually exist as lane boundary lines in the feature space, along with a set of confidence scores corresponding to each coordinate. By referencing these sets, the on-board device 10 can determine the confidence level of the estimated lane boundary lines.
[0105] The estimation unit 112 sends the lane boundary line position information (a set of coordinates) and confidence information (a set of confidence levels corresponding to each coordinate) output from the estimation model to the driving control unit 113. The driving control unit 113 uses this information to control the autonomous driving of the vehicle 1. For example, the driving control unit 113 can determine the level of confidence based on the estimated lane boundary line position information and use the result of the determination to control the autonomous driving. For example, for parts with high confidence, the probability of the lane boundary line existing is high, so it can be given greater weight in the predetermined trajectory generation process; for parts with low confidence, the probability of the lane boundary line existing is low, so it can be given less weight in the predetermined trajectory generation process.
[0106] [Processing Flowchart] Next, the process flow performed by each device will be explained.
[0107] Figure 8 This is a flowchart of the process of generating a dataset for learning, performed by the vehicle-mounted device 10. Figure 8 The process shown is performed periodically during the driving of vehicle 1.
[0108] First, in step S11, the collection unit 111 acquires images from multiple cameras 20.
[0109] The processing in steps S12 to S13 is performed on each of the multiple camera images obtained by the multiple cameras 20.
[0110] In step S12, the collection unit 111 extracts pixels from the image of the object that are presumed to represent lane boundary lines, and uses the set of their two-dimensional coordinates as a feature quantity.
[0111] Next, in step S13, the collection unit 111 calculates the confidence level of the existence of the lane boundary line for each of the multiple two-dimensional coordinates obtained in step S12. The confidence level of the existence of the lane boundary line at a specific coordinate on the image can be calculated, for example, using the methods illustrated in (A) to (C) above.
[0112] Next, in step S14, the collection unit 111 projects the feature quantities generated in step S12 onto the feature quantity space. The feature quantity space is a three-dimensional space represented by a coordinate system based on vehicle 1. Through this step, the positions of the lane boundary lines captured by each camera are projected onto the feature quantity space.
[0113] Next, in step S15, the collection unit 111 projects the confidence score of each coordinate generated in step S13 onto the feature space. This step projects the confidence score corresponding to each coordinate projected in step S14 onto the feature space.
[0114] Then, in step S16, the collection unit 111 sends the projection results (first data) from step S14 and the projection results (second data) from step S15 as a learning dataset to the server device 30. It should be noted that in this step, the collection unit 111 sends the vehicle's location information (e.g., latitude and longitude in the World Geodetic System) along with the dataset to the server device 30 simultaneously.
[0115] Figure 9 This is a flowchart of the process by which server device 30 performs learning of an inference model based on a dataset received from vehicle-mounted device 10. Figure 9 The process shown begins, for example, when there is a request to send a dataset from the vehicle-mounted device 10.
[0116] First, in step S21, the learning unit 311 receives a dataset (a set of first data and second data) for learning and the location information of the vehicle 1 from the vehicle-mounted device 10.
[0117] Next, in step S22, the learning unit 311 retrieves the correct answer data (lane boundary line location information) corresponding to the received dataset from the stored lane map. In this step, the learning unit 311 aligns the feature space with the lane map based on the vehicle 1's location information. This allows it to obtain the accurate coordinates of the lane boundary lines in the feature space.
[0118] In step S23, the learning unit 311 uses the first data (i.e., the set of coordinates in the feature space) as input data, and the second data (i.e., the set of confidence scores corresponding to the coordinates) and the third data (i.e., the set of coordinates representing the exact location of the lane boundary line) as correct answer data to perform the learning of the inference model.
[0119] The presumed model obtained through learning is distributed to the vehicle-mounted device 10.
[0120] Figure 10 This is a flowchart of the process by which the vehicle-mounted device 10 uses the estimated model obtained from the server device 30 to identify lane boundary lines. Figure 10 The process shown is performed periodically during the driving of vehicle 1.
[0121] First, in step S31, the estimation unit 112 acquires camera images from the plurality of cameras 20.
[0122] The processing in step S32 is performed for each of the multiple images obtained through multiple cameras.
[0123] In step S32, the estimation unit 112 extracts pixels that are estimated to represent lane boundary lines from the image of the object, and uses the set of two-dimensional coordinates of these pixels as feature quantities.
[0124] Next, in step S33, the estimation unit 112 projects the feature values of each camera generated in step S32 onto a predetermined feature value space.
[0125] In step S34, the estimation unit 112 inputs a set of three-dimensional coordinates representing the position of the lane boundary line in the feature space into the estimation model, and obtains a set of coordinates representing the estimated position of the lane boundary line and a set of confidence scores for each coordinate as output.
[0126] In step S35, the estimation unit 112 sends the set of coordinates obtained as the result of estimation, together with the confidence information, from the estimation unit 112 to the driving control unit 113, and uses it for autonomous driving control of the vehicle 1.
[0127] As explained above, the vehicle-mounted device 10 of the first embodiment generates a dataset for learning the inference model based on images acquired by the vehicle-mounted multiple cameras 20. Furthermore, at this time, a confidence level of the actual existence of the lane boundary line is calculated for each coordinate corresponding to the lane boundary line, and this confidence level is used as data for learning.
[0128] Based on this structure, a machine learning model can be obtained that outputs the confidence level of the existence of lane boundary lines, along with a set of coordinates representing the position of the lane boundary lines. Furthermore, more accurate autonomous driving can be achieved by utilizing such a machine learning model.
[0129] It should be noted that in the first embodiment, the camera images are converted into feature quantities, and the feature quantities are integrated before being input into the estimation model. However, the input to the estimation model can also be the camera images themselves. In this case, the input to the estimation model (first data) becomes multiple camera images. It should also be noted that, similarly to the first embodiment, the second data can be information obtained by projecting the confidence level of each coordinate onto the feature quantity space.
[0130] (Modified example) The above-described implementation is merely an example, and this disclosure may be implemented with appropriate modifications without departing from its spirit.
[0131] For example, as long as there is no technical contradiction, the processing and units described in this disclosure can be freely combined to implement the process.
[0132] Furthermore, in the description of the implementation, an example is given of the server device 30 learning the inference model based on the dataset generated by the vehicle device 10, but the learning of the inference model can also be performed by the vehicle device 10.
[0133] Furthermore, while lane boundary lines are shown as an example of the object to be identified in the description of the implementation method, the object to be identified does not necessarily have to be lane boundary lines. For example, any target on the road, a target moving on the road, etc., can also be identified as the object.
[0134] Furthermore, a process described as being performed by one device can also be executed by multiple devices. Alternatively, a process described as being performed by different devices can also be executed by one device. In a computer system, the hardware configuration (server configuration) used to implement each function can be flexibly changed.
[0135] This disclosure can also be implemented by supplying a computer program that implements the functions described in the above embodiments to a computer, which then reads and executes the program using one or more processors. Such a computer program can be provided to the computer via a non-transitory computer-readable storage medium that can be connected to the computer's system bus, or via a network. Non-transitory computer-readable storage media include, for example, any type of optical disk (such as a floppy disk, hard disk drive, etc.), optical disk (such as CD-ROM, DVD / Blu-ray disc, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM (Electrically Erasable Programmable Read Only Memory), magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions.
Claims
1. An information processing apparatus comprising a control unit, the control unit performing the following operations: Acquire camera images captured by a camera mounted on a moving body; In the camera image, a first region is identified as presumed to contain the first target; For each of the multiple coordinates corresponding to the first region, calculate the confidence level of the existence of the first target; as well as The set of multiple coordinates corresponding to the first region is used as input data, and the set of confidence scores corresponding to each of the multiple coordinates and the set of coordinates of the first target that actually exists in the specified space are used as correct answer data to perform learning, thereby generating a machine learning model, wherein the machine learning model is a model that outputs the coordinates of the first target that are presumed to exist in the specified space together with the confidence scores.
2. The information processing apparatus according to claim 1, wherein, The defined space is a three-dimensional space based on the moving body. The control unit obtains the plurality of coordinates corresponding to the first region by projecting the first region onto the three-dimensional space.
3. The information processing apparatus according to claim 1, wherein, The first target is the lane boundary line. The control unit calculates the confidence level for each of the plurality of coordinates based on the shape of the first region.
4. The information processing apparatus according to claim 1, wherein, The control unit is configured to: In the camera image, a second region is identified that is presumed to contain a second target different from the first target; as well as In the case where the camera image contains coordinates belonging to both the first region and the second region, a relatively low confidence level is calculated for those coordinates compared to the case where the coordinates belong only to the first region.
5. The information processing apparatus according to claim 1, wherein, The control unit is configured to: A third region with prescribed visual characteristics is identified in the camera image; as well as In the calculation of the confidence level, when there are coordinates belonging to both the first region and the third region, a relatively low confidence level is calculated for the coordinates compared to the case where the coordinates belong only to the first region.
6. The information processing apparatus according to claim 5, wherein, The third region is the region in which the statistical values of at least one of lightness, chroma, and hue meet the specified conditions.
7. An information processing method, which is executed by an information processing device. The information processing method includes: Acquire camera images captured by a camera mounted on a moving body; In the camera image, a first region is identified as presumed to contain the first target; For each of the multiple coordinates corresponding to the first region, calculate the confidence level of the existence of the first target; as well as The set of multiple coordinates corresponding to the first region is used as input data, and the set of confidence scores corresponding to each of the multiple coordinates and the set of coordinates of the first target that actually exists in the specified space are used as correct answer data to enable the machine learning model to learn. The machine learning model is a model that outputs the coordinates of the first target that are presumed to exist in the specified space together with the confidence scores.
8. The information processing method according to claim 7, wherein, The defined space is a three-dimensional space based on the moving body. The multiple coordinates corresponding to the first region are obtained by projecting the first region onto the three-dimensional space.
9. The information processing method according to claim 7, wherein, The first target is the lane boundary line. The confidence level is calculated for each of the plurality of coordinates based on the shape of the first region.
10. The information processing method according to claim 7, wherein, In the camera image, a second region is identified that is presumed to contain a second target different from the first target. In the case where the camera image contains coordinates belonging to both the first region and the second region, a relatively low confidence level is calculated for those coordinates compared to the case where the coordinates belong only to the first region.
11. The information processing method according to claim 7, wherein, In the camera image, a third region with prescribed visual characteristics is identified. In the calculation of the confidence level, when there are coordinates belonging to both the first region and the third region, a relatively low confidence level is calculated for the coordinates compared to the case where the coordinates belong only to the first region.
12. The information processing method according to claim 11, wherein, The third region is the region in which the statistical values of at least one of lightness, chroma, and hue meet the specified conditions.
13. A non-transitory storage medium having a program stored thereon for causing a computer to perform the information processing method as described in claim 7.
Citation Information
Patent Citations
Feature image recognition system, feature image recognition method, and computer program
JP2016194815A
