Signal lamp and lane matching method, device, equipment, medium and product
By combining signal light detection information and spatial position information, using pre-trained models and calibration parameters of image acquisition equipment, the accurate matching of signal lights and lanes is achieved, solving the problem of low accuracy of signal lights and lanes in autonomous driving environments, and improving Safety and reliability of autonomous vehicles.
Patent Information
- Application Number
- CN202411105411.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-12-03
AI Technical Summary
The matching accuracy of signal lights and lanes in autonomous driving environments is low, which makes it difficult for autonomous driving vehicles to accurately identify the relationship between signal lights and lanes when driving to the target intersection, affecting safety and reliability.
Based on the signal light detection information, the signal light position information under the image reference system is determined by combining the spatial position information of the signal light, and the signal light position information is realized based on the signal light lane mapping relationship. Accurate matching of lanes.
It improves the accuracy of the matching of signal lights to lanes when autonomous vehicles drive to target intersections, and enhances the safety and reliability of the vehicle.
Smart Images

Figure CN119068457A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, electronic device, storage medium and program product for matching a signal light with a lane. Background Art
[0002] In the autonomous driving scenario, the detection and recognition of traffic lights is an essential part of the autonomous driving vehicle. Autonomous driving vehicles can usually collect driving environment images through cameras, and based on models such as target detection and target recognition, they can extract features such as the color and shape of traffic lights from the driving environment images. However, it is difficult to obtain the accurate matching relationship between traffic lights and lanes through driving environment images. Summary of the invention
[0003] The present invention provides a method, device, electronic device, storage medium and program product for matching signal lights and lanes to solve the problem of low accuracy in matching signal lights and lanes in an autonomous driving environment. By combining the spatial position information of the signal lights on the basis of signal light detection information, accurate matching of signal lights and lanes is achieved, which is beneficial to improving the safety and reliability of autonomous driving vehicles.
[0004] According to one aspect of the present invention, a method for matching a signal light with a lane is provided, the method being performed by an autonomous driving vehicle, wherein an image acquisition device is configured in the autonomous driving vehicle; the method comprising:
[0005] If the autonomous driving vehicle is detected to be driving to the target intersection, the driving environment image of the autonomous driving vehicle is acquired through the image acquisition device;
[0006] Determine signal light detection information based on the driving environment image and a pre-trained signal light detection model; wherein the signal light detection information includes planar position information of the signal light;
[0007] Determine the signal light position information in the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection, and the calibration parameters of the image acquisition device;
[0008] The matching result of the lane and the signal light in the driving environment image is determined according to the signal light position information in the image reference system, the planar position information of the signal light in the signal light detection information and the pre-acquired signal light lane mapping relationship.
[0009] According to another aspect of the present invention, a device for matching a signal light with a lane is provided, wherein the device is configured in an autonomous driving vehicle, wherein an image acquisition device is configured in the autonomous driving vehicle; the device comprises:
[0010] An image acquisition module is used to acquire an image of the driving environment of the autonomous driving vehicle through an image acquisition device if it is detected that the autonomous driving vehicle has driven to a target intersection;
[0011] A detection information determination module, used to determine the signal light detection information according to the driving environment image and based on a pre-trained signal light detection model; wherein the signal light detection information includes the planar position information of the signal light;
[0012] A position information determination module, used to determine the position information of the signal light in the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection, and the calibration parameters of the image acquisition device;
[0013] The matching result determination module is used to determine the matching result of the lane and the signal light in the driving environment image based on the signal light position information in the image reference system, the planar position information of the signal light in the signal light detection information, and the pre-acquired signal light lane mapping relationship.
[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0015] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for matching a signal light with a lane as described in any embodiment of the present invention.
[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for matching a signal light with a lane as described in any embodiment of the present invention when executed.
[0017] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method for matching a signal light with a lane according to any embodiment of the present invention is implemented.
[0018] The technical solution of the embodiment of the present invention is to obtain the driving environment image of the autonomous driving vehicle through the image acquisition device when the autonomous driving vehicle is detected to be driving to the target intersection; determine the signal light detection information based on the driving environment image and the pre-trained signal light detection model; determine the signal light position information under the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection and the calibration parameters of the image acquisition device; determine the matching result between the lane and the signal light in the driving environment image according to the signal light position information under the image reference system, the planar position information of the signal light in the signal light detection information and the pre-acquired signal light lane mapping relationship. This technical solution solves the problem of low accuracy in matching signal lights and lanes in the autonomous driving environment, and realizes accurate matching between signal lights and lanes by combining the spatial position information of signal lights on the basis of signal light detection information, which is conducive to improving the safety and reliability of autonomous driving vehicles.
[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 is a flow chart of a method for matching a signal light and a lane according to Embodiment 1 of the present invention;
[0022] Figure 2A is a flow chart of a method for matching a signal light and a lane according to a second embodiment of the present invention;
[0023] Figure 2B is a schematic diagram of projection and detection frame distribution provided according to Embodiment 2 of the present invention;
[0024] Figure 3 is a schematic diagram of the structure of a signal light and lane matching device provided according to Embodiment 3 of the present invention;
[0025] Figure 4 The present invention is a schematic diagram of the structure of an electronic device for implementing the method for matching a signal light with a lane according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.
[0028] Embodiment 1
[0029] Figure 1 A flowchart of a signal light and lane matching method is provided for the first embodiment of the present invention. This embodiment can be applied to the signal light and lane matching situation in an autonomous driving scenario. The method can be executed by a signal light and lane matching device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:
[0030] S110: If it is detected that the autonomous driving vehicle has reached a target intersection, an image of the driving environment of the autonomous driving vehicle is acquired through an image acquisition device.
[0031] This solution can be executed by an autonomous driving vehicle, which can be equipped with a navigation device inside the autonomous driving vehicle to realize the positioning of the autonomous driving vehicle, so as to obtain the spatial position information of the autonomous driving vehicle in real time. According to the spatial position information of the autonomous driving vehicle and the high-precision map provided by the navigation device, the autonomous driving vehicle can determine whether the current driving section is an intersection section. If the current driving section is an intersection section, it is determined that the autonomous driving vehicle is detected to have driven to the intersection.
[0032] An image acquisition device may be configured in an autonomous vehicle to acquire images of the driving environment while the autonomous vehicle is driving. The image acquisition device may be a visible light imaging device, such as a camera, or an infrared thermal imaging device, such as an infrared thermal imager. The number of image acquisition devices deployed in an autonomous vehicle may be one or more. If there is one image acquisition device, it may be deployed at the front end of the autonomous vehicle. If there are more than one image acquisition device, they may be deployed around the body of the autonomous vehicle.
[0033] It is understandable that the target intersection may be an intersection that the autonomous driving vehicle is about to arrive at, for example, an intersection that the autonomous driving vehicle has not passed and is closest to the autonomous driving vehicle in the driving direction of the autonomous driving vehicle. When the autonomous driving vehicle is detected to be driving to the target intersection, the autonomous driving vehicle may collect an image of the driving environment of the autonomous driving vehicle through an image acquisition device.
[0034] S120. Determine signal light detection information according to the driving environment image and based on a pre-trained signal light detection model; wherein the signal light detection information includes planar position information of the signal light.
[0035] After obtaining the driving environment image of the autonomous driving vehicle at the target intersection, the autonomous driving vehicle can use the pre-trained signal light detection model to perform signal light detection on the driving environment image and output signal light detection information. The signal light detection model can be obtained by training a pre-built target detection network using image samples containing signal lights. The target detection network can be a convolutional neural network built based on a deep learning algorithm, a one-stage target detection network based on YOLO, SSD, etc., or a two-stage target detection network based on Faster R-CNN, etc. The signal light detection model can determine the planar position information of the signal light based on the driving environment image.
[0036] It is understandable that the planar position information of the signal light can be used to characterize the distribution position of the signal light in the driving environment image, for example, it can include the edge coordinates of the signal light. Based on the planar position information of the signal light, the autonomous driving vehicle can determine the signal light detection frame in the driving environment image. The signal light detection frame can be a regular shape such as a rectangle, an ellipse, or an irregular shape. In a preferred embodiment, the signal light detection model can also perform feature extraction on the signal light image in the signal light detection frame after obtaining the signal light detection frame to obtain the color, shape and other features of the signal light.
[0037] S130. Determine the signal light position information in the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection, and the calibration parameters of the image acquisition device.
[0038] It is easy to understand that the spatial position information of the autonomous driving vehicle can be the position description information of the autonomous driving vehicle in three-dimensional space, for example, it can include the longitude, latitude and altitude of the autonomous driving vehicle. In the high-precision map provided by the navigation device, the autonomous driving vehicle can obtain the spatial position information of the traffic light in the target intersection. Among them, the spatial position information of the traffic light can be the position description information of the traffic light in three-dimensional space, for example, it can include the longitude, latitude and altitude of the traffic light. The autonomous driving vehicle can project the spatial position information of the traffic light in the target intersection into the driving environment image through the conversion between the reference systems such as the vehicle, the image acquisition device and the image, and obtain the traffic light position information under the image reference system. Specifically, the autonomous driving vehicle can first convert the spatial position information of the traffic light in the target intersection to the traffic light position information under the vehicle reference system according to the spatial position information of the autonomous driving vehicle and the spatial position information of the traffic light in the target intersection, and then convert the traffic light position information under the vehicle reference system to the traffic light position information under the image reference system according to the calibration parameters of the image acquisition device. It can be understood that the traffic light position information under the image reference system should be the planar position information of the traffic light in the driving environment image.
[0039] S140, determining a matching result between the lane and the signal light in the driving environment image according to the signal light position information in the image reference system, the planar position information of the signal light in the signal light detection information, and a pre-acquired signal light lane mapping relationship.
[0040] It can be understood that the plane position information of the signal light in the signal light detection information is the distribution position of the signal light obtained directly through the analysis of the driving environment image, and the signal light position information under the image reference system is the distribution position of the signal light in the driving environment image obtained by projecting the spatial position information of the signal light. The autonomous driving vehicle can compare the plane position information of the signal light in the signal light detection information with the signal light position information under the image reference system, calculate the degree of fit between the two, and then determine the matching of the two based on the degree of fit, so as to realize the mapping of the signal light in space and the signal light in the driving environment image.
[0041] The autonomous driving vehicle can obtain the signal light lane mapping relationship of the target intersection from the high-precision map, or determine the signal light lane mapping relationship in the target intersection based on the spatial position information of the signal light and the spatial position information of the lane. Specifically, the signal light lane mapping relationship can be the indication relationship between the signal light and the lane, that is, which lane the signal light is used to indicate. One lane can be mapped to multiple signal lights, and one signal light can also indicate multiple lanes.
[0042] Based on the signal light lane mapping relationship, combined with the matching of the planar position information of the signal light in the signal light detection information and the signal light position information in the image reference system, the autonomous driving vehicle can obtain the matching results of the lane and the signal light in the driving environment image, so as to realize the mapping of the lane in space and the signal light in the driving environment image.
[0043] The technical solution of the embodiment of the present invention is to obtain the driving environment image of the autonomous driving vehicle through the image acquisition device when the autonomous driving vehicle is detected to be driving to the target intersection; determine the signal light detection information based on the driving environment image and the pre-trained signal light detection model; determine the signal light position information under the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection and the calibration parameters of the image acquisition device; determine the matching result between the lane and the signal light in the driving environment image according to the signal light position information under the image reference system, the planar position information of the signal light in the signal light detection information and the pre-acquired signal light lane mapping relationship. This technical solution solves the problem of low accuracy in matching signal lights and lanes in the autonomous driving environment, and realizes accurate matching between signal lights and lanes by combining the spatial position information of signal lights on the basis of signal light detection information, which is conducive to improving the safety and reliability of autonomous driving vehicles.
[0044] Embodiment 2
[0045] Figure 2A This is a flow chart of a method for matching a signal light with a lane provided in the second embodiment of the present invention. This embodiment refines the above embodiment. Figure 2A As shown, the method includes:
[0046] S201. If it is detected that the autonomous driving vehicle has reached a target intersection, an image of the driving environment of the autonomous driving vehicle is acquired through an image acquisition device.
[0047] S202. Determine signal light detection information according to the driving environment image and based on a pre-trained signal light detection model; wherein the signal light detection information includes planar position information of the signal light.
[0048] S203. Determine the signal light position information in the vehicle reference system according to the spatial position information of the autonomous driving vehicle and the pre-acquired spatial position information of the signal light at the target intersection.
[0049] In this solution, the spatial location information can be expressed in the form of coordinates, such as (longitude i ,latitude i ,altitude i ), where i represents the index of traffic targets such as autonomous vehicles or traffic lights, and longitudei Indicates the longitude of traffic target i, latitude i Indicates the latitude of traffic target i, altitude i Indicates the height of traffic target i. The autonomous driving vehicle can establish a spatial coordinate system with the autonomous driving vehicle as the center, convert the spatial position information of the traffic light at the target intersection into the spatial coordinate system, and obtain the traffic light position information in the vehicle reference system. The traffic light position information in the vehicle reference system should be three-dimensional position information.
[0050] S204: Determine the signal light position information in the image acquisition device reference system according to the signal light position information in the vehicle reference system and external parameters of the image acquisition device.
[0051] It can be understood that the calibration parameters of the image acquisition device include external parameters and internal parameters. Among them, the external parameters can be used to describe the parameters of the position and posture of the image acquisition device in the world coordinate system, usually including a rotation matrix and a translation vector. The rotation matrix is used to describe the rotation of the image acquisition device coordinate system relative to the world coordinate system, and the translation vector is used to describe the translation of the image acquisition device coordinate system relative to the world coordinate system. According to the external parameters of the image acquisition device, the autonomous driving vehicle can convert the signal light position information in the vehicle reference system to the signal light position information in the image acquisition device reference system, that is, the signal light position information in the image acquisition device coordinate system. The signal light position information in the image acquisition device reference system should be three-dimensional position information.
[0052] S205: Determine the traffic light position information in the image reference system according to the traffic light position information in the image acquisition device reference system and internal parameters of the image acquisition device.
[0053] The internal parameters of the image acquisition device can be used to describe the transformation relationship from the image acquisition device reference system to the image reference system, which usually include parameters such as focal length, optical center coordinates, and distortion coefficients. Based on the internal parameters of the image acquisition device, the autonomous driving vehicle can convert the signal light position information in the image acquisition device reference system to the signal light position information in the image reference system. The signal light position information in the image acquisition device reference system should be two-dimensional position information.
[0054] S206: Determine the projection of the signal light according to the position information of the signal light in the image reference system.
[0055] It can be understood that the signal light position information under the image reference system is the mapping of the spatial position information of the signal light in the driving environment image. Therefore, the autonomous driving vehicle can generate a signal light projection in the driving environment image based on the signal light position information under the image reference system. Among them, the signal light position information under the image reference system can be the description information of the outline of the signal light; the signal light projection can be a description box of the distribution position of the signal light, which can be a regular shape such as a rectangle or a circle, or an irregular shape.
[0056] S207: Determine a signal light detection frame according to the planar position information of the signal light in the signal light detection information.
[0057] It is easy to understand that the planar position information of the signal light can be used to characterize the distribution position of the signal light in the driving environment image. According to the planar position information of the signal light in the signal light detection information, the autonomous driving vehicle can generate a signal light detection frame. It should be noted that, for ease of comparison, the signal light projection generated by S206 and the signal light detection frame determined by S207 can be the same shape.
[0058] S208. Determine a cost evaluation result of the signal light projection and the signal light detection frame according to the signal light projection and the signal light detection frame.
[0059] After obtaining the signal light projection and the signal light detection frame, the autonomous driving vehicle can determine the degree of fit between the signal light projection and the signal light detection frame based on information such as the distance, overlapping area, and area size between the two, and then determine the cost evaluation result of the signal light projection and the signal light detection frame based on the degree of fit between the signal light projection and the signal light detection frame.
[0060] Figure 2B is a schematic diagram of the distribution of projections and detection frames according to the second embodiment of the present invention. Figure 2B As shown in the figure, the solid line frame represents the signal light detection frame obtained by performing signal light detection on the driving environment image, and the dotted line frame represents the signal light projection obtained by converting the spatial position information of the target intersection signal light. Figure 2B In the driving environment image generated from the perspective of the autonomous driving vehicle, signal lights 1-3 can be observed. Therefore, according to the driving environment image, the autonomous driving vehicle can detect signal lights 1-3 and obtain the following: Figure 2B The signal light detection frames shown. Signal lights 4-6 are not within the field of view of the autonomous driving vehicle and cannot be imaged in the driving environment image. Therefore, the autonomous driving vehicle cannot detect signal lights 4-6 from the driving environment image.
[0061] In this solution, optionally, determining the cost evaluation result of the signal light projection and the signal light detection frame according to the signal light projection and the signal light detection frame includes:
[0062] Calculate the distance from the center of the signal light projection to the center of the signal light detection frame;
[0063] Determine the degree of fit between the signal light projection and the signal light detection frame according to the distance between the signal light projection center and the signal light detection frame center and the area of the signal light detection frame;
[0064] According to the degree of fit between the signal light projection and the signal light detection frame, a cost evaluation result of the signal light projection and the signal light detection frame is determined.
[0065] refer to Figure 2B , the autonomous driving vehicle can determine the center position of the signal light projection and the center position of the signal light detection frame, and calculate the distance from the center of the signal light projection to the center of the signal light detection frame. At the same time, the autonomous driving vehicle can calculate the area of the signal light detection frame. According to the distance from the center of the signal light projection to the center of the signal light detection frame, and the area of the signal light detection frame, the degree of fit between the signal light projection and the signal light detection frame is calculated. The autonomous driving vehicle can directly use the degree of fit between the signal light projection and the signal light detection frame as the cost evaluation result. In order to speed up the matching speed, the autonomous driving vehicle can also set a fit threshold, by comparing the fit between the signal light projection and the signal light detection frame with the fit threshold, and determining the cost evaluation result of the signal light projection and the signal light detection frame according to the comparison result.
[0066] Based on the above scheme, the calculation formula of the degree of fit is expressed as:
[0067] f(bbox det ,P prj ) = d 2 (bbox det ,P prj ) / area(bbox det );
[0068] Among them, bbox det Indicates the traffic light detection frame, P prj Indicates the signal light projection, d(bbox det ,P prj ) represents the distance from the center of the signal light projection to the center of the signal light detection frame, area(bbox det ) represents the area of the traffic light detection box, f(bbox det ,P prj ) indicates the degree of fit between the signal light projection and the signal light detection frame;
[0069] The expression of the cost evaluation result is:
[0070]
[0071] Among them, cost_value represents the cost evaluation result, and threshold represents the preset fit threshold.
[0072] This solution calculates the degree of fit between the signal light detection frame and the signal light projection by calculating d 2 (bbox det ,P prj ) can eliminate the image scale effect caused by the distance of the target and obtain a more accurate fit value. det ,P prj ) is less than or equal to the fit threshold, f(bbox det ,P prj ) as the cost evaluation result of the signal light projection and the signal light detection frame, in f(bbox det ,P prj ) is greater than the fit threshold, the cost evaluation result of the signal light projection and the signal light detection frame is set to positive infinity. The above scheme is conducive to finely distinguishing the signal light projection and the signal light detection frame when the fit is low, and speeding up the matching process when the fit is high.
[0073] S209: According to the cost evaluation result of the signal light projection and the signal light detection frame, based on the Hungarian model, determine the matching result of the signal light projection and the signal light detection frame.
[0074] It can be understood that the Hungarian model is a combinatorial optimization algorithm for solving task allocation problems in polynomial time, and it has promoted the subsequent primal dual method. In this embodiment, the autonomous driving vehicle can construct a cost evaluation matrix with the signal light projection as one dimension and the signal light detection frame as another dimension, for example, with the signal light projection as the row and the signal light detection frame as the column to construct the cost evaluation matrix. There can be multiple signal light projections and signal light detection frames. The row where each element in the cost evaluation matrix is located corresponds to a signal light projection, and the column where it is located corresponds to a signal light detection frame. The element value is the cost evaluation result of the signal light projection and the signal light detection frame. According to the cost evaluation matrix, the Hungarian model is used to determine the matching result of the signal light projection and the signal light detection frame.
[0075] In this embodiment, the number of the signal light projections is at least two, and the number of the signal light detection frames is at least two;
[0076] The step of determining the matching result of the signal light projection and the signal light detection frame based on the Hungarian model according to the cost evaluation result of the signal light projection and the signal light detection frame includes:
[0077] Generate a cost evaluation matrix based on the cost evaluation results of each signal light projection and signal light detection frame;
[0078] The Hungarian model is used to perform at least one operation on the cost evaluation matrix, and the matching result between the signal light projection and the signal light detection frame is determined according to the cost evaluation matrix output by the last operation.
[0079] After obtaining the cost evaluation matrix, the autonomous driving vehicle can calculate the cost evaluation matrix according to the following steps:
[0080] Step 1: Determine the minimum value of each row of elements in the cost evaluation matrix, and subtract the minimum value of the row from the elements of the row;
[0081] Step 2: Determine the minimum value of each column of the cost evaluation matrix and subtract the minimum value of the column from the elements of the column;
[0082] Step 3: Use the minimum number of straight lines to cover all zero elements in the cost evaluation matrix, where the straight lines include horizontal lines and / or vertical lines;
[0083] Step 4: Determine whether the number of lines is equal to the dimension of the cost evaluation matrix;
[0084] Step 5: If the number of straight lines is less than the dimension of the cost evaluation matrix, determine the minimum element among the uncovered elements of the cost evaluation matrix, subtract the minimum element from all the uncovered elements of the cost evaluation matrix, and return to execute step 3; if the number of straight lines is equal to the dimension of the cost evaluation matrix, determine the matching result between the traffic light projection and the traffic light detection frame based on the 0 elements covered by the straight lines.
[0085] S210: Determine a matching result between the lane and the signal light in the driving environment image according to a matching result between the signal light projection and the signal light detection frame and a pre-acquired signal light lane mapping relationship.
[0086] After obtaining the matching result of the signal light projection and the signal light detection frame, the signal light projection and the signal light detection frame can correspond one to one. Among them, each signal light projection corresponds to a spatial position information in the high-precision map. The autonomous driving vehicle can obtain the mapping relationship between the signal light and the lane in the high-precision map in advance. It can be understood that the signal light lane mapping relationship can characterize the corresponding relationship between the signal light projection and the lane. Based on the signal light lane mapping relationship and the matching result of the signal light projection and the signal light detection frame, the autonomous driving vehicle can obtain the matching result of the lane and the signal light in the driving environment image.
[0087] In this solution, the signal light detection information may also include signal light color information and signal light shape information. After obtaining the matching results of the signal lights in the lane and driving environment image, the autonomous driving vehicle may generate driving decisions based on the signal light color information and signal light shape information to ensure safe and orderly driving.
[0088] The technical solution of the embodiment of the present invention is to obtain the driving environment image of the autonomous driving vehicle through the image acquisition device when the autonomous driving vehicle is detected to be driving to the target intersection; determine the signal light detection information based on the driving environment image and the pre-trained signal light detection model; determine the signal light position information under the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection and the calibration parameters of the image acquisition device; determine the matching result between the lane and the signal light in the driving environment image according to the signal light position information under the image reference system, the planar position information of the signal light in the signal light detection information and the pre-acquired signal light lane mapping relationship. This technical solution solves the problem of low accuracy in matching signal lights and lanes in the autonomous driving environment, and realizes accurate matching between signal lights and lanes by combining the spatial position information of signal lights on the basis of signal light detection information, which is conducive to improving the safety and reliability of autonomous driving vehicles.
[0089] Embodiment 3
[0090] Figure 3 The schematic diagram of the structure of a signal light and lane matching device provided in the third embodiment of the present invention. The device is configured in an autonomous driving vehicle, and the autonomous driving vehicle is equipped with an image acquisition device. Figure 3 As shown, the device comprises:
[0091] The image acquisition module 310 is used to acquire the driving environment image of the autonomous driving vehicle through an image acquisition device if it is detected that the autonomous driving vehicle has driven to a target intersection;
[0092] The detection information determination module 320 is used to determine the signal light detection information according to the driving environment image based on the pre-trained signal light detection model; wherein the signal light detection information includes the planar position information of the signal light;
[0093] A position information determination module 330, for determining the position information of the signal light in the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection, and the calibration parameters of the image acquisition device;
[0094] The matching result determination module 340 is used to determine the matching result of the lane and the signal light in the driving environment image according to the signal light position information in the image reference system, the planar position information of the signal light in the signal light detection information, and the pre-acquired signal light lane mapping relationship.
[0095] In this solution, optionally, the calibration parameters include external parameters and internal parameters;
[0096] The location information determination module 330 is specifically used to:
[0097] Determine the signal light position information in the vehicle reference system according to the spatial position information of the autonomous driving vehicle and the pre-acquired spatial position information of the signal light at the target intersection;
[0098] Determine the signal light position information in the image acquisition device reference system according to the signal light position information in the vehicle reference system and the external parameters of the image acquisition device;
[0099] The position information of the traffic light in the image reference system is determined according to the position information of the traffic light in the image acquisition device reference system and the internal parameters of the image acquisition device.
[0100] In a feasible solution, the matching result determination module 340 includes:
[0101] A projection determination unit, used to determine the projection of the signal light according to the position information of the signal light in the image reference system;
[0102] A detection frame determination unit, used to determine a signal light detection frame according to the planar position information of the signal light in the signal light detection information;
[0103] A cost evaluation unit, used to determine a cost evaluation result of the signal light projection and the signal light detection frame according to the signal light projection and the signal light detection frame;
[0104] A first matching unit, configured to determine a matching result of the signal light projection and the signal light detection frame based on a Hungarian model according to a cost evaluation result of the signal light projection and the signal light detection frame;
[0105] The second matching unit is used to determine the matching result of the lane and the signal light in the driving environment image according to the matching result of the signal light projection and the signal light detection frame and the pre-acquired signal light lane mapping relationship.
[0106] Based on the above solution, the cost evaluation unit is specifically used to:
[0107] Calculate the distance from the center of the signal light projection to the center of the signal light detection frame;
[0108] Determine the degree of fit between the signal light projection and the signal light detection frame according to the distance between the signal light projection center and the signal light detection frame center and the area of the signal light detection frame;
[0109] According to the degree of fit between the signal light projection and the signal light detection frame, a cost evaluation result of the signal light projection and the signal light detection frame is determined.
[0110] In a feasible solution, the calculation formula of the degree of fit is expressed as:
[0111] f(bbox det ,Pprj ) = d 2 (bbox det ,P prj ) / area(bbox det );
[0112] Among them, bbox det Indicates the traffic light detection frame, P prj Indicates the signal light projection, d(bbox det ,P prj ) represents the distance from the center of the signal light projection to the center of the signal light detection frame, area(bbox det ) represents the area of the traffic light detection box, f(bbox det ,P prj ) indicates the degree of fit between the signal light projection and the signal light detection frame;
[0113] The expression of the cost evaluation result is:
[0114]
[0115] Among them, cost_value represents the cost evaluation result, and threshold represents the preset fit threshold.
[0116] Optionally, the number of the signal light projections is at least two, and the number of the signal light detection frames is at least two;
[0117] The first matching unit is specifically used for:
[0118] Generate a cost evaluation matrix based on the cost evaluation results of each signal light projection and signal light detection frame;
[0119] The Hungarian model is used to perform at least one operation on the cost evaluation matrix, and the matching result between the signal light projection and the signal light detection frame is determined according to the cost evaluation matrix output by the last operation.
[0120] The signal light and lane matching device provided in the embodiment of the present invention can execute the signal light and lane matching method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0121] Embodiment 4
[0122] Figure 4A schematic diagram of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0123] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.
[0124] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0125] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 executes the various methods and processes described above, such as a method for matching a signal light with a lane.
[0126] In some embodiments, the signal light and lane matching method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the signal light and lane matching method described above can be performed. Alternatively, in other embodiments, the processor 411 can be configured to perform the signal light and lane matching method in any other appropriate manner (for example, by means of firmware).
[0127] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0128] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable signal light and lane matching device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed completely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.
[0129] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0130] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0131] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0132] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0133] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0134] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for matching a signal light with a lane, characterized in that: The method is performed by an autonomous driving vehicle, wherein an image acquisition device is configured in the autonomous driving vehicle; the method comprises: If the autonomous driving vehicle is detected to be driving to the target intersection, the driving environment image of the autonomous driving vehicle is acquired through the image acquisition device; Determine signal light detection information based on the driving environment image and a pre-trained signal light detection model; wherein the signal light detection information includes planar position information of the signal light; Determine the signal light position information in the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection, and the calibration parameters of the image acquisition device; The matching result of the lane and the signal light in the driving environment image is determined according to the signal light position information in the image reference system, the planar position information of the signal light in the signal light detection information and the pre-acquired signal light lane mapping relationship.
2. The method according to claim 1, characterized in that The calibration parameters include external parameters and internal parameters; The determining of the signal light position information in the image reference system according to the spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection, and the calibration parameters of the image acquisition device includes: Determine the signal light position information in the vehicle reference system according to the spatial position information of the autonomous driving vehicle and the pre-acquired spatial position information of the signal light at the target intersection; Determine the signal light position information in the image acquisition device reference system according to the signal light position information in the vehicle reference system and the external parameters of the image acquisition device; The position information of the traffic light in the image reference system is determined according to the position information of the traffic light in the image acquisition device reference system and the internal parameters of the image acquisition device.
3. The method according to claim 2, characterized in that The determining of the matching result between the lane and the signal light in the driving environment image according to the signal light position information in the image reference system, the planar position information of the signal light in the signal light detection information, and the pre-acquired signal light lane mapping relationship includes: Determine the projection of the signal light according to the position information of the signal light in the image reference system; Determine a signal light detection frame according to the plane position information of the signal light in the signal light detection information; Determine the cost evaluation result of the signal light projection and the signal light detection frame according to the signal light projection and the signal light detection frame; According to the cost evaluation results of the signal light projection and the signal light detection frame, the matching result of the signal light projection and the signal light detection frame is determined based on the Hungarian model; According to the matching result of the signal light projection and the signal light detection frame, and the pre-acquired signal light lane mapping relationship, the matching result of the lane and the signal light in the driving environment image is determined.
4. The method according to claim 3, characterized in that The determining, according to the signal light projection and the signal light detection frame, a cost evaluation result of the signal light projection and the signal light detection frame comprises: Calculate the distance from the center of the signal light projection to the center of the signal light detection frame; Determine the degree of fit between the signal light projection and the signal light detection frame according to the distance between the signal light projection center and the signal light detection frame center and the area of the signal light detection frame; According to the degree of fit between the signal light projection and the signal light detection frame, a cost evaluation result of the signal light projection and the signal light detection frame is determined.
5. The method according to claim 4, characterized in that The calculation formula of the degree of fit is expressed as: f(bbox det ,P prj )=d 2 (bbox det ,P prj ) / area(bbox det ); Among them, bbox det Indicates the traffic light detection frame, P prj Indicates the signal light projection, d(bbox det ,P prj ) represents the distance from the center of the signal light projection to the center of the signal light detection frame, area(bbox det ) represents the area of the traffic light detection box, f(bbox det ,P prj ) indicates the degree of fit between the signal light projection and the signal light detection frame; The expression of the cost evaluation result is: Among them, cost_value represents the cost evaluation result, and threshold represents the preset fit threshold.
6. The method according to claim 5, characterized in that The number of the signal light projections is at least two, and the number of the signal light detection frames is at least two; The step of determining the matching result of the signal light projection and the signal light detection frame based on the Hungarian model according to the cost evaluation result of the signal light projection and the signal light detection frame includes: Generate a cost evaluation matrix based on the cost evaluation results of each signal light projection and signal light detection frame; The Hungarian model is used to perform at least one operation on the cost evaluation matrix, and the matching result between the signal light projection and the signal light detection frame is determined according to the cost evaluation matrix output by the last operation.
7. A signal light and lane matching device, characterized in that: The device is configured in an autonomous driving vehicle, and an image acquisition device is configured in the autonomous driving vehicle; the device comprises: An image acquisition module is used to acquire an image of the driving environment of the autonomous driving vehicle through an image acquisition device if it is detected that the autonomous driving vehicle has driven to a target intersection; A detection information determination module, used to determine the signal light detection information according to the driving environment image and based on a pre-trained signal light detection model; wherein the signal light detection information includes the planar position information of the signal light; A position information determination module, used to determine the position information of the signal light in the image reference system according to the pre-acquired spatial position information of the autonomous driving vehicle, the pre-acquired spatial position information of the signal light at the target intersection, and the calibration parameters of the image acquisition device; The matching result determination module is used to determine the matching result of the lane and the signal light in the driving environment image based on the signal light position information in the image reference system, the planar position information of the signal light in the signal light detection information, and the pre-acquired signal light lane mapping relationship.
8. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for matching a signal light with a lane according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for matching a signal light and a lane according to any one of claims 1 to 6 when executed.
10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for matching a signal light with a lane according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Input adjustment method and apparatus for intelligent-driving sensing network, and device and medium
WO2026157745A1
