Traffic signal lamp identification method and device, and storage medium
By using object detection network and standard map binding technology in autonomous driving cars, accurate identification and position determination of traffic lights in complex road environments are achieved, and the limitations of relying on high-precision maps in the prior art are solved, and the limitations of relying on high-precision maps are improved. The safety and cost-effectiveness of autonomous driving.
Patent Information
- Application Number
- CN202310627532.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
AI Technical Summary
It is difficult for existing autonomous vehicles to identify traffic lights without relying on high-precision maps to achieve accurate identification, especially in complex road environments, where traffic lights and lane lines cannot be effectively bound, resulting in limited identification accuracy and safety.
By acquiring the intersection image sequence, the preset object detection network is used to identify the traffic lights and lane positions in the BEV coordinate system, and project them to the global coordinate system, and bind them in combination with standard maps to obtain the area of interest images of the traffic lights. Conduct detection of symbols, colors and display status to realize the identification and position determination of traffic lights.
It realizes the accurate identification of the symbols, colors and display status of traffic lights without relying on high-precision maps, and binds traffic lights and lane lines in the global coordinate system, improving the safety and flexibility of autonomous driving and reducing The cost of high-precision map updates.
Smart Images

Figure CN119068447A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle-mounted sensing technology, and in particular to a traffic light recognition method and device, and a storage medium. Background Art
[0002] Traffic light recognition is an important function for autonomous vehicles in urban scenarios and is related to the safety of traffic participants. Urban traffic intersections vary in complexity, and the layout of traffic lights is even more complicated, which requires autonomous vehicles to accurately recognize traffic lights when passing through intersections in order to pass safely.
[0003] Currently, there are two main visual recognition schemes for autonomous vehicles to recognize traffic lights: a visual recognition scheme that relies on high-precision maps and a visual recognition scheme that does not rely on high-precision maps. High-precision maps have extremely high requirements for map information, are costly, and have poor solution flexibility. Among them, the visual recognition scheme that relies on high-precision maps is difficult to use on roads without high-precision maps. The visual recognition scheme that does not rely on high-precision maps only recognizes images, and it is difficult to provide the correct position of traffic lights, and it is impossible to bind traffic lights and lane lines. It can only be used for intersections with relatively simple traffic conditions. Summary of the invention
[0004] The purpose of this application is to propose a traffic light recognition method and device, and a computer-readable storage medium, which can recognize traffic light signals without relying on high-precision maps, bind traffic lights and lane lines, and provide the correct position of traffic lights.
[0005] To achieve the above objectives, the present application provides a method for identifying a traffic light, including:
[0006] Acquire a sequence of intersection images, and process the sequence of intersection images using a preset target detection network to obtain a position and posture of at least one traffic light and a position and posture of at least one lane in the BEV coordinate system;
[0007] Acquire the current posture of the vehicle, and project the at least one traffic light and the at least one lane into the global coordinate system according to the current posture of the vehicle, the posture of at least one traffic light in the BEV coordinate system, and the posture of at least one lane in the BEV coordinate system;
[0008] In the global coordinate system, the current lane of the vehicle is determined according to the current position of the vehicle and the coordinates of the at least one lane in the global coordinate system, and the current lane and the traffic light matching it are bound according to the standard map;
[0009] Acquire a region of interest of the bound traffic light in a global coordinate system, and project the region of interest onto a visual sensor image to obtain a region of interest image;
[0010] The image of the region of interest is detected to obtain at least one of a symbol, a color, and a display state of the traffic light.
[0011] In some embodiments, the target detection network includes:
[0012] An input terminal, used for inputting the intersection image sequence;
[0013] A feature extraction network, used for extracting image features of the intersection image sequence;
[0014] A feature encoding network, used for encoding the image features as spatial features or temporal features;
[0015] A feature decoding network, used for decoding the encoding result output by the feature encoding network;
[0016] A first detection sub-network, used to detect traffic lights based on the decoding results output by the feature decoding network;
[0017] The second detection sub-network is used to detect lane lines based on the decoding results output by the feature decoding network.
[0018] In some embodiments, binding the current lane and the traffic light matching the current lane includes:
[0019] Acquire a lane attribute of the current lane according to a standard map; the lane attribute is left turn, straight ahead or right turn;
[0020] acquiring at least one candidate traffic light corresponding to the lane attribute from the at least one traffic light;
[0021] A traffic light matching the current lane is determined according to the position and posture of the current lane in the global coordinate system and the position and posture of the at least one candidate traffic light in the global coordinate system.
[0022] In some embodiments, projecting the region of interest onto a visual sensor image to obtain an image of the region of interest includes:
[0023] The region of interest is projected onto the visual sensor image to obtain the area of the bound traffic light on the image, and it is determined whether the area on the image exceeds the preset image range. If it exceeds, the region of interest is determined to be an invalid area. If it does not exceed, the region of interest is determined to be a valid area, and the corresponding region of interest image is output.
[0024] In some embodiments, the detecting the image of the region of interest to obtain at least one of a symbol, a color, and a display state of a traffic light includes:
[0025] Detecting the lamp group and the lamp head in the image of the region of interest, and obtaining the position information of the lamp group in the image of the region of interest and the position information of the lamp head in the image of the region of interest;
[0026] Extracting a lamp head image from the image of the region of interest according to the position information of the lamp head in the image, and inputting the lamp head image into a preset classification network for classification to obtain the symbol and color of the lamp head;
[0027] The position information of the lamp head in the image and / or the position and posture of the traffic light in the global coordinate system are input into a preset tracker for tracking to obtain the display status of the lamp head.
[0028] The embodiment of the present application also provides a traffic light recognition device, comprising:
[0029] A first detection module is used to obtain a sequence of intersection images, and process the sequence of intersection images using a preset target detection network to obtain a position and posture of at least one traffic light in a BEV coordinate system and a position and posture of at least one lane in the BEV coordinate system;
[0030] A first projection module is used to obtain the current posture of the vehicle, and project the at least one traffic light and the at least one lane into the global coordinate system according to the current posture of the vehicle, the posture of at least one traffic light in the BEV coordinate system, and the posture of at least one lane in the BEV coordinate system;
[0031] A binding module, used to determine the current lane where the vehicle is located in the global coordinate system according to the current position of the vehicle and the coordinates of the at least one lane in the global coordinate system, and bind the current lane and the traffic light matching it according to the standard map;
[0032] A second projection module is used to obtain the interest region of the bound traffic light in the global coordinate system, and project the interest region onto the visual sensor image to obtain an interest region image;
[0033] The second detection module is used to detect the image of the area of interest to obtain at least one of the symbol, color, and display status of the traffic light.
[0034] In some embodiments, the binding module is used to:
[0035] Acquire a lane attribute of the current lane according to a standard map; the lane attribute is left turn, straight ahead or right turn;
[0036] acquiring at least one candidate traffic light corresponding to the lane attribute from the at least one traffic light;
[0037] A traffic light matching the current lane is determined according to the position and posture of the current lane in the global coordinate system and the position and posture of the at least one candidate traffic light in the global coordinate system.
[0038] In some embodiments, the second projection module is used to:
[0039] The region of interest is projected onto the visual sensor image to obtain the area of the bound traffic light on the image, and it is determined whether the area on the image exceeds the preset image range. If it exceeds, the region of interest is determined to be an invalid area. If it does not exceed, the region of interest is determined to be a valid area, and the corresponding region of interest image is output.
[0040] In some embodiments, the second detection module is used to:
[0041] Detecting the lamp group and the lamp head in the image of the region of interest, and obtaining the position information of the lamp group in the image of the region of interest and the position information of the lamp head in the image of the region of interest;
[0042] Extracting a lamp head image from the image of the region of interest according to the position information of the lamp head in the image, and inputting the lamp head image into a preset classification network for classification to obtain the symbol and color of the lamp head;
[0043] The position information of the lamp head in the image and / or the position and posture of the traffic light in the global coordinate system are input into a preset tracker for tracking to obtain the display status of the lamp head.
[0044] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the traffic light recognition device as described above is implemented.
[0045] The embodiments of the present application provide a traffic light recognition device and its device, and a computer-readable storage medium, which can recognize traffic light signals without relying on high-precision maps, obtain at least one of the symbols, colors, and display states of traffic lights, and match and bind the traffic lights and the current lane where the vehicle is located in the global coordinate system. At the same time, it can also provide the correct posture of the traffic light (that is, the posture of the traffic light in the global coordinate system); the embodiments of the present application only require general standard maps to be implemented, which can greatly reduce the problems caused by high-precision map updates and reduce the cost of popularizing autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 This is a flow chart of a traffic light recognition device in one embodiment of the present application.
[0048] Figure 2 A schematic diagram of a complex intersection scene in one embodiment of the present application.
[0049] Figure 3 A schematic diagram of a target detection network in one embodiment of the present application.
[0050] Figure 4 A schematic diagram of a scene at an intersection in one embodiment of the present application.
[0051] Figure 5 This is a schematic diagram of a traffic light recognition device in one embodiment of the present application. DETAILED DESCRIPTION
[0052] The detailed description of the drawings is intended as an illustration of some embodiments of the present application, and is not intended to represent the only form in which the present application can be implemented. It should be understood that the same or equivalent functions can be accomplished by different embodiments intended to be included in the spirit and scope of the present application.
[0053] An embodiment of the present application provides a method for identifying a traffic light. Figure 1 , the method of this embodiment includes the following steps:
[0054] Step S100, acquiring a sequence of intersection images, and processing the sequence of intersection images using a preset target detection network to obtain a position and posture of at least one traffic light and a position and posture of at least one lane in the BEV coordinate system.
[0055] Specifically, the intersection image sequence includes images at multiple consecutive moments. For example, if the current moment is t, then in step S100, the intersection images at moments t, t-1, and t-2 are obtained as the intersection image sequence. This is only an example, and does not necessarily mean the intersection images at 3 moments, but may also be intersection images at 4 moments, 5 moments, etc.; the preset target detection network recognizes traffic lights and lanes (or lane lines, each lane includes two left and right lane lines) based on the BEV perspective. The BEV perspective refers to the bird's eye view (Bird's Eye View). Correspondingly, the result of target detection based on BEV perspective is based on BEV coordinate system, which refers to bird's-eye view coordinate system, that is, top view coordinate system, which is a coordinate system that looks down on the target scene from above the perspective, and is often used in environmental perception and decision-making in the field of autonomous driving; in the BEV coordinate system, the position of the vehicle is regarded as the origin of the coordinate system, the front of the vehicle is the positive direction of the x-axis, and the left side of the vehicle is the positive direction of the y-axis; the obstacle information obtained by sensors such as cameras will be represented and processed in the BEV coordinate system, so as to realize the perception and decision-making of the environment around the vehicle, when passing through complex intersections, for example Figure 2 As shown, multiple traffic lights will be captured and need to be identified.
[0056] Among them, the position and posture of the traffic light in the BEV coordinate system includes the coordinates and orientation of the traffic light in the BEV coordinate system, the position and posture of each lane in the BEV coordinate system includes the coordinates and orientation of each lane in the BEV coordinate system, and the coordinates of each lane are specifically the coordinates of the points on the left and right lane lines of each lane.
[0057] Step S200, obtaining the current posture of the vehicle, and projecting the at least one traffic light and the at least one lane into the global coordinate system according to the current posture of the vehicle, the posture of at least one traffic light in the BEV coordinate system, and the posture of at least one lane in the BEV coordinate system.
[0058] Specifically, the current position of the vehicle refers to the coordinates and orientation of the vehicle in the global coordinate system, which can be obtained through sensors such as the vehicle-mounted GPS and IMU; the coordinates of the origin of the BEV coordinate system (the location of the vehicle) in the global coordinate system are determined, and then the at least one traffic light and the at least one lane can be projected into the global coordinate system by rotating according to the orientation of the vehicle;
[0059] For example, assuming that the coordinates of a point A in the BEV coordinate system are (x_bev, y_bev), the position of the vehicle in the global coordinate system is (x_vehicle, y_vehicle), and the orientation (heading angle) of the vehicle is theta, then the coordinates of the point A in the global coordinate system are (x_global, y_global):
[0060] x_global=x_bev*cos(theta)-y_bev*sin(theta)+x_vehicle;
[0061] y_global=x_bev*sin(theta)+y_bev*cos(theta)+y_vehicle;
[0062] Based on the above principles, at least one traffic light and at least one lane in the BEV coordinate system can be projected into the global coordinate system to achieve environmental perception and decision-making; at the same time, the orientation of the lane and the traffic light in the global coordinate system can also be determined according to the heading angle of the vehicle and the orientation of the lane and the traffic light in the BEV coordinate system.
[0063] Step S300, in the global coordinate system, determine the current lane where the vehicle is located according to the current position of the vehicle and the coordinates of the at least one lane in the global coordinate system, and bind the current lane and the traffic light matching it according to the standard map.
[0064] Specifically, in the global coordinate system, the coordinates (x_vehicle, y_vehicle) of the vehicle in the global coordinate system can be determined according to the current position of the vehicle, and the lane range within which the coordinates (x_vehicle, y_vehicle) fall can be found, that is, the current lane where the vehicle is located can be found; further, according to the lane attributes provided by the standard map (SD map), the lane attributes of the current lane are determined, and the lane attributes include left turn, right turn, straight going, etc. After determining the lane attributes, the current lane is matched with at least one detected traffic light according to the lane attributes of the current lane, and a traffic light indicating driving in the current lane is matched, and the traffic light is bound to the current lane; each traffic light can be assigned a traffic light id, and each lane is also assigned a lane id, and the traffic light id and lane id are bound.
[0065] Step S400: obtaining a region of interest of a bound traffic light in a global coordinate system, and projecting the region of interest onto a visual sensor image to obtain a region of interest image.
[0066] Specifically, the region of interest refers to ROI (Region of Interest), which refers to an area of interest in an image or video, also known as a region of interest. In computer vision, it is often necessary to perform special processing or recognition operations on a certain area in an image, and ROI is used to specify this area. In the field of autonomous driving, ROI may refer to an area that needs to be focused on in a driving assistance system, such as the scene in front of the vehicle; the region of interest in step S400 is specifically the area where a traffic light that has been bound in step S300 is located.
[0067] Furthermore, the region of interest in the global coordinate system is three-dimensional information, and the three-dimensional information can be projected onto a two-dimensional visual sensor image according to the internal and external parameters of the visual sensor to obtain a corresponding region of interest image. The projection principle is well known to those skilled in the art, so it will not be introduced here.
[0068] Step S500: Detect the image of the region of interest to obtain at least one of the symbol, color, and display state of the traffic light.
[0069] Specifically, the symbol of the traffic light is, for example, a left arrow, a right arrow, a straight arrow, or a circle; the color of the traffic light is, for example, red, green, or yellow; and the display state of the traffic light is, for example, flashing, always on, or always off.
[0070] In summary, the method of this embodiment can recognize traffic light signals without relying on high-precision maps, obtain at least one of the symbol, color, and display status of the traffic light, and match and bind the traffic light and the current lane where the vehicle is located in the global coordinate system. At the same time, it can also provide the correct position of the traffic light (that is, the position of the traffic light in the global coordinate system); the method of this embodiment only requires a general standard map to be implemented, which can greatly reduce the problems caused by high-precision map updates and reduce the cost of popularizing autonomous driving.
[0071] In some embodiments, see Figure 3 , the target detection network includes an input end, a feature extraction network, a feature encoding network, a feature decoding network, and a first detection sub-network and a second detection sub-network connected to the feature decoding network in sequence; it should be noted that, compared with the multi-model method of one network each for object detection and lane line detection, the target detection network of this embodiment adopts a multi-task single network method, which not only avoids the resource consumption caused by multiple networks and improves computing efficiency, but also realizes the identification of defined traffic participants under a unified BEV perspective. Multiple tasks can promote each other and reduce the deviation caused by the merging of multiple networks.
[0072] The input end is used to input the intersection image sequence.
[0073] The feature extraction network is used to extract image features of the intersection image sequence; specifically, the feature extraction network mainly extracts basic feature information, and can adopt a classic CNN structure, such as resnet, vovnet, GpuNet, etc., and then forms spatial multi-scale features through FPN / BiFPN and other methods, and sends the spatial multi-scale features to the feature encoding network.
[0074] The feature encoding network is used to encode the image features by spatial features or temporal features; specifically, the feature encoding can be spatial feature encoding or temporal feature encoding, and the spatial feature encoding and temporal feature encoding are illustrated below;
[0075] In order to establish the relationship between the 2D image and the 3D spatial position, the camera frustum is discretized to form a three-dimensional grid with a size of (W F ,H F ,D), where W F , H F Represent the width and height of the image feature respectively, and D represents the discrete number of depths. For points in the discrete three-dimensional grid, it can be expressed as Among them, u j ,v j is the point on the image coordinates, d j Represents the depth value perpendicular to the image plane, then the discrete grid is mapped to 3D space in the following way:
[0076]
[0077] Among them, K -1 Represents the transformation matrix from the 3D space in the camera coordinate system to the discrete three-dimensional grid;
[0078] The spatial position encoding is described as follows:
[0079] F 3d1 =ψ(ξ(F image ),ζ(p 3D ))
[0080] Among them, F 3d1 represents the feature after spatial feature encoding, F image represents image features, ψ() represents the fusion network, ξ() represents the feature adaptation network, and ζ() represents the 3D position embedding network.
[0081] The specific description of temporal feature encoding is as follows:
[0082] make Represents the transformation matrix from coordinate system A to coordinate system B, then the temporal coordinate alignment can be described as:
[0083]
[0084]
[0085]
[0086] Among them, (t) represents time t, (t-1) represents time t-1, c represents the camera coordinate system, e represents the vehicle coordinate system, d represents a virtual coordinate system, and g represents the global coordinate system global;
[0087] The temporal features are encoded as follows:
[0088]
[0089] Among them, F 3d2 represents the feature after temporal feature encoding, χ() represents feature concatenation, is a neural network used to encode image features and 3D positions, F image (t) represents the image features at time t, F image (t-1) represents the image features at time t-1;
[0090] The output of the feature encoding network is used as the input of the feature decoding network. In addition to the spatial feature encoding and temporal feature encoding listed above, the feature encoding method can also adopt the bevformer feature encoding method.
[0091] The feature decoding network is used to decode the encoding result output by the feature encoding network; specifically, the feature decoding network accepts the output of the feature encoding network as the key and value, and the query is obtained during the training process. In the attention mechanism, the key and the value are one-to-one corresponding. When calculating the attention weight, the key is used to calculate the similarity with the query vector, and the obtained similarity value is the weight. Then, this weight value is multiplied by the corresponding value, and finally all the products are added to obtain the output feature vector.
[0092] Q l =Φ l (F 3d ,Q l-1 ),l=1,...,L
[0093] Among them, Q l is the object query after the update of the decoding network of the first layer. Object query refers to the use of specific syntax in the program to query the object through the object's attributes, methods and other information.l is the lth layer decoding network, F 3d is the output of the encoding network;
[0094] The decoding result after decoding by the feature decoding network is sent to the first detection sub-network and the second detection sub-network to obtain corresponding information output.
[0095] The first detection sub-network is used to detect traffic lights based on the decoding results output by the feature decoding network;
[0096] The second detection sub-network is used to detect lane lines based on the decoding results output by the feature decoding network;
[0097] As an extension, the target detection network in this embodiment may further add other detection sub-networks for detecting other traffic targets, such as other vehicles, pedestrians, road traffic signs, and the like.
[0098] In some embodiments, the step S300 of binding the current lane and the traffic light matching the current lane according to the standard map specifically includes:
[0099] Step S301, obtaining the lane attribute of the current lane according to the standard map; the lane attribute is left turn, straight or right turn;
[0100] Step S302, obtaining at least one candidate traffic light corresponding to the lane attribute from the at least one traffic light;
[0101] Specifically, at a complex intersection, the image in step S100 may capture multiple groups of traffic lights, such as Figure 4 At the complex intersection shown, multiple groups of traffic lights are captured. If the lane attribute of the current lane is left turn, and the multiple groups of traffic lights all have lamp heads indicating left turn, then the multiple groups of traffic lights are all used as candidate traffic lights for screening in step S303.
[0102] Step S303, determining a traffic light matching the current lane according to the orientation of the current lane in the global coordinate system and the position of the at least one candidate traffic light in the global coordinate system;
[0103] Specifically, if Figure 4The figure shows a scene of an intersection. The lane id of the current lane where the vehicle is located is 2. The traffic light facing the vehicle is determined based on the direction of the current lane and the direction of the traffic light in the global coordinate system. The candidate traffic light closest to the vehicle is determined based on the position of the traffic light in the global coordinate system. Based on the above two factors, the traffic light with traffic light id = 3 can be bound to lane id = 2, thereby obtaining the binding relationship between the current lane and the traffic light.
[0104] For example, a neural network model can be pre-trained, and the neural network model may include a lane information extraction network, multiple traffic light information extraction networks and a matching network. The vehicle coordinates, lane posture and traffic light posture are used as inputs of the neural network model for training, and the output is the matching result of the lane and the traffic light. The lane information extraction network is used to extract lane feature information, the traffic light information extraction network is used to extract traffic light feature information, and the matching network is used to match the lane feature information with multiple traffic light feature information, and output the matching result; in the application process, the posture of the current lane in the global coordinate system is input into the lane information extraction network, and the posture of the at least one traffic light in the global coordinate system is respectively input into at least one traffic light information extraction network (for example, if three traffic lights are detected, the postures of the three traffic lights are respectively input into the three traffic light information extraction networks).
[0105] In some embodiments, projecting the region of interest onto the visual sensor image to obtain the region of interest image in step S400 specifically includes:
[0106] The region of interest is projected onto the visual sensor image to obtain the area of the bound traffic light on the image, and it is determined whether the area on the image exceeds the preset image range. If it exceeds, the region of interest is determined to be an invalid area. If it does not exceed, the region of interest is determined to be a valid area, and the corresponding region of interest image is output.
[0107] Specifically, the region of interest P global-roi Project it onto the narrow-angle vision sensor image to obtain the bound traffic light area P on the image roi-l , if P roi-l If the image exceeds the narrow-angle image range, it is projected onto the normal-angle visual sensor image to obtain the normal-angle area P roi-r , if P roi-l and P roi-r If both exceed the corresponding image range, then we can judge that the current P global-roi is an invalid area, where area P roi-l and P roi-rThe method of judging whether it is effective is:
[0108] 0≤P roi-lx <P roi-rx ≤w
[0109] 0≤P roi-ly <P roi-ry ≤h
[0110] Among them, P roi-lx , P roi-ly They are region P roi-l The x-axis coordinate and y-axis coordinate, P roi-rx , P roi-ry They are region P roi-r The x-axis coordinate and y-axis coordinate of .
[0111] In some embodiments, the step S500 includes:
[0112] Step S501, detecting the lamp group and the lamp head in the image of the region of interest, and obtaining the position information of the lamp group in the image of the region of interest and the position information of the lamp head in the image of the region of interest;
[0113] Specifically, the traffic light image block I in the interest area image can be extracted. roi , the image block I roi A pre-trained signal light detection network is input to detect light groups and lamp heads, and the position information of the light groups in the image of the area of interest and the position information of the lamp heads in the image of the area of interest are obtained; the light group refers to a group of multiple lighting fixtures, and their parameters such as switch, brightness and color temperature can be controlled at the same time, and the lamp head refers to a single lamp, for example, a light group includes 3 lamp heads, and the 3 lamp heads respectively indicate left turn, straight go, and right turn.
[0114] The signal light detection network is a 2D target detection network, such as the common SSD, faster-rcnn, yolo series, fcos series, etc., which are not specifically limited here. In this embodiment, the signal light detection network is modified to detect two categories, namely, light groups and lamp heads. During training, the position constraint relationship between the light groups and lamp heads is added as supervision information, which can be expressed as:
[0115] Loss = L rec +λL loc +ηL in
[0116]
[0117] Among them, obj light-group For the light group, obj light-head For the lamp holder, Lrec is the loss function for classification, L loc is the loss function of position regression, L in is the loss function of the position constraints of the lamp group and the lamp head, δ is the preset parameter value, λ and η are the weighted harmonic coefficients respectively. During reasoning, the lamp heads that are not in the lamp group are filtered out according to the relationship rules between the lamp group and the lamp head.
[0118] Step S502: extracting a lamp head image I from the image of the region of interest according to the position information of the lamp head in the image. roi-head , and the lamp head image I roi-head Input the preset classification network to classify and obtain the symbol and color of the lamp head;
[0119] Specifically, after obtaining the symbol and color of the lamp holder, a corresponding relationship can be established between the lamp holder and the lamp group, and the symbol and color can be assigned to the lamp group.
[0120] Step S503: input the position information of the lamp head in the image and / or the position and posture of the traffic light in the global coordinate system into a preset tracker for tracking to obtain the display status of the lamp head.
[0121] Specifically, the tracker is, for example, a Kalman filter tracker, and one or more of the position information of the lamp head in the image obtained in step S501 and / or the position and posture of the traffic light in the global coordinate system obtained in step S200 are input into the tracker. The tracker can be composed of a common Kalman tracker to establish the spatiotemporal relationship of the traffic light, and the relationship is constructed as follows:
[0122] Kalman tracking is used to obtain the trajectory of each lamp head, and then the state of the traffic light is determined based on the trajectory. For example, for determining the flashing state, assuming that the current frame rate of the visual sensor is 25fps and the flashing frequency of the traffic light is 1Hz, for the trajectory of lamp head A, its state changes from bright-dark-bright, then it can be determined that the current lamp head is in a flashing state.
[0123] Furthermore, for example, since the neural network is affected by various factors, it may have false detections.
[0124] If there is a missed detection, the output of the current frame can be corrected through the tracker. That is, assuming that the trajectory of lamp head A has been established and is valid, if lamp head A is missing in the current frame, the tracker can still output the status of lamp head A in the current frame. If the trajectory symbol of lamp head A is straight at this time, and due to the false detection of the neural network, the current lamp head A is mistakenly identified as a left turn, then according to the trajectory of lamp head A, it can be corrected to straight. For another example, possible false detection information can be further filtered according to prior rules. For example, the yellow light state after the red light can be considered as an error state, and the yellow light needs to be forcibly set to a red light. This also needs to be achieved by establishing a tracking trajectory.
[0125] Corresponding to the method of the above embodiment, another embodiment of the present application further provides a traffic light recognition device, including a module that can be used to execute the steps of the method of the above embodiment, and participating in Figure 5 , the module includes:
[0126] A first detection module 1 is used to obtain a sequence of intersection images, and to process the sequence of intersection images using a preset target detection network to obtain a position and posture of at least one traffic light and a position and posture of at least one lane in the BEV coordinate system;
[0127] A first projection module 2 is used to obtain the current posture of the vehicle, and project the at least one traffic light and the at least one lane into the global coordinate system according to the current posture of the vehicle, the posture of at least one traffic light in the BEV coordinate system, and the posture of at least one lane in the BEV coordinate system;
[0128] A binding module 3 is used to determine the current lane where the vehicle is located in the global coordinate system according to the current position of the vehicle and the coordinates of the at least one lane in the global coordinate system, and bind the current lane and the traffic light matching it;
[0129] The second projection module 4 is used to obtain the interest region of the bound traffic light in the global coordinate system, and project the interest region onto the visual sensor image to obtain an interest region image;
[0130] The second detection module 5 is used to detect the image of the region of interest to obtain at least one of the symbol, color and display status of the traffic light.
[0131] In some embodiments, the binding module 3 is specifically used to:
[0132] Acquire a lane attribute of the current lane according to a standard map; the lane attribute is left turn, straight ahead or right turn;
[0133] acquiring at least one candidate traffic light corresponding to the lane attribute from the at least one traffic light;
[0134] A traffic light matching the current lane is determined according to the position and posture of the current lane in the global coordinate system and the position and posture of the at least one candidate traffic light in the global coordinate system.
[0135] In some embodiments, the second projection module 4 is specifically used for:
[0136] The region of interest is projected onto the visual sensor image to obtain the area of the bound traffic light on the image, and it is determined whether the area on the image exceeds the preset image range. If it exceeds, the region of interest is determined to be an invalid area. If it does not exceed, the region of interest is determined to be a valid area, and the corresponding region of interest image is output.
[0137] In some embodiments, the second detection module 5 is used to:
[0138] Detecting the lamp group and the lamp head in the image of the region of interest, and obtaining the position information of the lamp group in the image of the region of interest and the position information of the lamp head in the image of the region of interest;
[0139] Extracting a lamp head image from the image of the region of interest according to the position information of the lamp head in the image, and inputting the lamp head image into a preset classification network for classification to obtain the symbol and color of the lamp head;
[0140] The position information of the lamp head in the image and / or the position and posture of the traffic light in the global coordinate system are input into a preset tracker for tracking to obtain the display status of the lamp head.
[0141] The traffic light recognition device of the above-described embodiment is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components of the modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the traffic light recognition device of the embodiment.
[0142] It should be noted that the traffic light recognition device of the above-mentioned embodiment corresponds to the traffic light recognition method of the above-mentioned embodiment. Therefore, the parts of the traffic light recognition device of the above-mentioned embodiment that are not described in detail can be obtained by referring to the contents of the traffic light recognition method of the above-mentioned embodiment, that is, the specific steps recorded in the traffic light recognition method of the above-mentioned embodiment can be understood as the functions that can be achieved by the traffic light recognition device of the above-mentioned embodiment, and will not be repeated here.
[0143] Furthermore, if the traffic light recognition device of the above embodiment is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0144] Another embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the traffic light recognition method as described in the above embodiment is implemented.
[0145] Specifically, the computer-readable storage medium may include: any entity or recording medium that can carry the computer program instructions, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0146] Another embodiment of the present application provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the traffic light recognition method described in the above embodiment when executing the program.
[0147] Wherein, the electronic device may also include a bus connecting different components (including memory and processor). The memory may include a computer-readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The memory may also include at least one program product, which has a set of (e.g., at least one) program modules, which are configured to perform the functions of the various embodiments of the present application. The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device, and / or communicate with any device (e.g., a network card) that enables the electronic device to communicate with one or more other computing devices, such communication may be performed through an input / output (I / O) interface, and the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter.
[0148] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for identifying a traffic light, characterized in that: include: Acquire a sequence of intersection images, and process the sequence of intersection images using a preset target detection network to obtain a position and posture of at least one traffic light and a position and posture of at least one lane in the BEV coordinate system; Acquire the current posture of the vehicle, and project the at least one traffic light and the at least one lane into the global coordinate system according to the current posture of the vehicle, the posture of at least one traffic light in the BEV coordinate system, and the posture of at least one lane in the BEV coordinate system; In the global coordinate system, the current lane of the vehicle is determined according to the current position of the vehicle and the coordinates of the at least one lane in the global coordinate system, and the current lane and the traffic light matching it are bound according to the standard map; Acquire a region of interest of the bound traffic light in a global coordinate system, and project the region of interest onto a visual sensor image to obtain a region of interest image; The image of the region of interest is detected to obtain at least one of a symbol, a color, and a display state of the traffic light.
2. The method according to claim 1, characterized in that The target detection network includes: An input terminal, used for inputting the intersection image sequence; A feature extraction network, used for extracting image features of the intersection image sequence; A feature encoding network, used for encoding the image features as spatial features or temporal features; A feature decoding network, used for decoding the encoding result output by the feature encoding network; A first detection sub-network, used to detect traffic lights based on the decoding results output by the feature decoding network; The second detection sub-network is used to detect lane lines based on the decoding results output by the feature decoding network.
3. The method according to claim 1, characterized in that Binding the current lane and the traffic light matching the current lane according to the standard map includes: Acquire a lane attribute of the current lane according to a standard map; the lane attribute is left turn, straight ahead or right turn; acquiring at least one candidate traffic light corresponding to the lane attribute from the at least one traffic light; A traffic light matching the current lane is determined according to the position and posture of the current lane in the global coordinate system and the position and posture of the at least one candidate traffic light in the global coordinate system.
4. The method according to claim 1, characterized in that: The step of projecting the region of interest onto a visual sensor image to obtain an image of the region of interest includes: The region of interest is projected onto the visual sensor image to obtain the area of the bound traffic light on the image, and it is determined whether the area on the image exceeds the preset image range. If it exceeds, the region of interest is determined to be an invalid area. If it does not exceed, the region of interest is determined to be a valid area, and the corresponding region of interest image is output.
5. The method according to claim 1, characterized in that The detecting the interest area image to obtain traffic light information includes: Detecting the lamp group and the lamp head in the image of the region of interest, and obtaining the position information of the lamp group in the image of the region of interest and the position information of the lamp head in the image of the region of interest; Extracting a lamp head image from the image of the region of interest according to the position information of the lamp head in the image, and inputting the lamp head image into a preset classification network for classification to obtain the symbol and color of the lamp head; The position information of the lamp head in the image and / or the position and posture of the traffic light in the global coordinate system are input into a preset tracker for tracking to obtain the display status of the lamp head.
6. A traffic light recognition device, characterized in that: include: A first detection module is used to obtain a sequence of intersection images, and process the sequence of intersection images using a preset target detection network to obtain a position and posture of at least one traffic light in a BEV coordinate system and a position and posture of at least one lane in the BEV coordinate system; A first projection module is used to obtain the current posture of the vehicle, and project the at least one traffic light and the at least one lane into the global coordinate system according to the current posture of the vehicle, the posture of at least one traffic light in the BEV coordinate system, and the posture of at least one lane in the BEV coordinate system; A binding module, used to determine the current lane where the vehicle is located in the global coordinate system according to the current position of the vehicle and the coordinates of the at least one lane in the global coordinate system, and bind the current lane and the traffic light matching it according to the standard map; A second projection module is used to obtain the interest region of the bound traffic light in the global coordinate system, and project the interest region onto the visual sensor image to obtain an interest region image; The second detection module is used to detect the image of the area of interest to obtain at least one of the symbol, color, and display status of the traffic light.
7. The device according to claim 6, characterized in that The binding module is used to: Acquire a lane attribute of the current lane according to a standard map; the lane attribute is left turn, straight ahead or right turn; acquiring at least one candidate traffic light corresponding to the lane attribute from the at least one traffic light; A traffic light matching the current lane is determined according to the position and posture of the current lane in the global coordinate system and the position and posture of the at least one candidate traffic light in the global coordinate system.
8. The device according to claim 6, characterized in that The second projection module is used to: The region of interest is projected onto the visual sensor image to obtain the area of the bound traffic light on the image, and it is determined whether the area on the image exceeds the preset image range. If it exceeds, the region of interest is determined to be an invalid area. If it does not exceed, the region of interest is determined to be a valid area, and the corresponding region of interest image is output.
9. The device according to claim 6, characterized in that The second detection module is used to: Detecting the lamp group and the lamp head in the image of the region of interest, and obtaining the position information of the lamp group in the image of the region of interest and the position information of the lamp head in the image of the region of interest; Extracting a lamp head image from the image of the region of interest according to the position information of the lamp head in the image, and inputting the lamp head image into a preset classification network for classification to obtain the symbol and color of the lamp head; The position information of the lamp head in the image and / or the position and posture of the traffic light in the global coordinate system are input into a preset tracker for tracking to obtain the display status of the lamp head.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the traffic light recognition method according to any one of claims 1 to 5 is implemented.
Citation Information
Cited By
Lane and traffic light association method and device, storage medium and electronic device
CN119905003A
