Traffic light detection method and device and storage medium

Abstract

The invention discloses a traffic light detection method comprising the following steps: obtaining a first detection result of a traffic light in a current frame image, the first detection result comprising a prediction coordinate of the traffic light in the current frame image; projecting the prediction coordinates of the traffic lights to a high-precision map coordinate system, and generating three-dimensional prediction coordinates of the corresponding traffic lights; constructing a bipartite graph containing three-dimensional space information based on the first detection result, the three-dimensional prediction coordinates and a high-precision map; and performing maximum matching calculation on the bipartite graph containing the three-dimensional space information, and determining a second detection result of the traffic light of the current frame image according to a maximum matching calculation result. According to the method provided by the embodiment of the invention, the traffic light is not identified by purely depending on the image detection result, but the image detection result is verified by introducing the high-precision map data on the basis of the image detection result, so that the finally obtained second detection result is matched with the position of the traffic light in the high-precision map; and the accuracy of a traffic light identification result is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a traffic light detection method, device, and storage medium. Background Technology

[0002] Intelligent driving technology is currently receiving unprecedented attention and is developing at an accelerated pace. Traffic light recognition is an essential and crucial component of intelligent driving systems, and accurate recognition of traffic light signals is of great significance to downstream modules such as path planning and safe driving control. Currently, common traffic light recognition and detection schemes mainly fall into three categories: those based on object detection networks, those based on image processing, and those based on binocular cameras. These detection and recognition schemes all rely solely on image information for traffic light detection. For example, when object detection networks are applied to traffic light detection, images captured by a camera are directly input into a pre-trained network, which then directly performs traffic light recognition and detection on the input image. Image processing-based techniques identify color density and other parameters in images acquired by the camera for traffic light detection. The binocular camera scheme performs image detection on images acquired from two cameras separately, and then correlates the detection results from the two cameras.

[0003] It is evident that the accuracy of current traffic light recognition and detection schemes is highly limited by the image information acquired by the camera itself. Since the pixel count and clarity of the image information are limited by the camera's imaging resolution and shooting distance, existing traffic light recognition and detection schemes suffer from limitations in accuracy due to factors such as long detection distances and low image resolutions. Summary of the Invention

[0004] Against this backdrop, a traffic light detection method is developed to effectively improve the accuracy of traffic light recognition and detection, especially to address the problem of low accuracy in detection results caused by practical issues such as detection distance and image resolution.

[0005] In a first aspect, embodiments of the present invention provide a traffic light detection method, comprising: Obtain a first detection result for traffic lights in the current frame image, wherein the first detection result includes the predicted coordinates of traffic lights in the current frame image; The predicted coordinates of the traffic lights are projected onto the high-precision map coordinate system to generate the corresponding three-dimensional predicted coordinates of the traffic lights. Based on the first detection result, the three-dimensional predicted coordinates, and the high-precision map, a bipartite graph containing three-dimensional spatial information is constructed. The maximum matching calculation is performed on the bipartite graph containing three-dimensional spatial information, and the second detection result of the traffic light in the current frame image is determined based on the maximum matching calculation result.

[0006] In a second aspect, embodiments of the present invention provide a traffic light detection device, comprising: The image detection result acquisition module is used to acquire the first detection result of the traffic lights in the current frame image, wherein the first detection result includes the predicted coordinates of the traffic lights in the current frame image; The 3D conversion module is used to project the predicted coordinates of traffic lights onto a high-precision map coordinate system to generate the corresponding 3D predicted coordinates of the traffic lights. The bipartite graph construction module is used to construct a bipartite graph containing three-dimensional spatial information based on the first detection result, the three-dimensional predicted coordinates, and the high-precision map. The matching module is used to perform maximum matching calculation on the bipartite graph containing three-dimensional spatial information, and determine the second detection result of the traffic light in the current frame image based on the maximum matching calculation result.

[0007] Thirdly, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the aforementioned traffic light detection method.

[0008] Fourthly, embodiments of the present invention provide an autonomous driving vehicle, including the aforementioned electronic equipment.

[0009] Fifthly, embodiments of the present invention provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned traffic light detection method.

[0010] The traffic light detection method provided by this invention constructs a bipartite graph in three-dimensional space using the image detection results of the current frame (i.e., the first detection result) and a high-precision map. This achieves data association between the image detection results of the current frame and the information in the high-precision map, transforming the traffic light recognition and detection problem into a maximum matching problem of the bipartite graph in three-dimensional space. Based on this, the method of this invention further performs maximum matching calculation on the bipartite graph in three-dimensional space based on the image detection results, and further determines the final traffic light detection result (i.e., the second detection result) based on the matching result. It is evident that the solution provided by this invention no longer relies solely on image detection results to identify traffic lights, but uses image detection results as a foundation and verifies them by introducing high-precision map data. This ensures that the final second detection result matches the traffic light position in the high-precision map, thus significantly improving the accuracy of traffic light recognition. This method effectively avoids excessive reliance on image resolution and clarity, enabling accurate detection and recognition of traffic light signals even when the image pixel count, resolution, or clarity is poor. It has a wider range of applicable scenarios and more accurate results. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a traffic light detection method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of a method for generating a first detection result using an adaptive traffic light detection method according to an embodiment of the present invention. Figure 3 The diagram illustrates a method flow chart of an embodiment of the present invention for real-time determination of a target region of interest in a current frame image based on a current frame image and a high-precision map. Figure 4 The diagram illustrates the principle and effect of determining the target projection area in a high-precision map in conjunction with the current driving area according to one embodiment of the present invention. Figure 5 The diagram illustrates a method flow chart for determining a target region of interest in a current frame image according to another embodiment of the present invention. Figure 6 This diagram schematically illustrates the effect of determining the region of interest according to an embodiment of the present invention. Figure 7 The diagram illustrates a method flow chart for determining a target region of interest in a current frame image according to another embodiment of the present invention. Figure 8 The diagram schematically illustrates the effect of determining the target region of interest according to another embodiment of the present invention; Figure 9 The diagram schematically illustrates the effect of determining the region of interest according to another embodiment of the present invention; Figure 10 The flowchart of a method for determining a target region of interest in a current frame image according to another embodiment of the present invention is shown schematically. Figure 11 The flowchart illustrates a method for determining the three-dimensional predicted coordinates of a traffic light in a high-precision map coordinate system based on the predicted coordinates of the traffic light, according to one embodiment of the present invention. Figure 12 This is a schematic diagram illustrating the processing effect of operation S23 according to one embodiment of the present invention; Figure 13 A flowchart illustrating a method for constructing a bipartite graph containing three-dimensional spatial information according to an embodiment of the present invention is shown schematically. Figure 14 A schematic block diagram of a traffic light detection device according to one embodiment of the present invention is shown. Figure 15 A schematic block diagram of a computing device according to an embodiment of the present invention is shown. Figure 16 A schematic block diagram of an autonomous vehicle according to one embodiment of the present invention is shown. Figure 17 This is a schematic diagram of an embodiment of the traffic light detection device of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0015] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, elements, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0016] In this invention, terms such as "module," "device," and "system" refer to relevant entities applied to a computer, such as hardware, combinations of hardware and software, software, or software in execution. More specifically, for example, an element can be, but is not limited to, a process running on a processor, a processor, an object, an executable element, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be an element. One or more elements may be in an execution process and / or thread, and elements may be localized on a single computer and / or distributed across two or more computers, and may be run on various computer-readable media. Elements can also communicate via local and / or remote processes based on signals having one or more data packets, for example, signals from data interacting with another element in a local system, a distributed system, and / or interacting with other systems via signals over a network of the Internet.

[0017] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0018] The traffic light detection method in this invention can be applied to any intelligent product that requires autonomous or semi-autonomous driving technology. This allows these intelligent products to more accurately detect and identify traffic light information using the solution provided in this invention, which can then be used for downstream path planning, driving strategy adjustment, and attitude repositioning. These intelligent products requiring autonomous or semi-autonomous driving technology include, but are not limited to, mobile tools capable of automatic or semi-automatic driving, such as drones, autonomous vehicles (passenger cars, buses, coaches, trucks, etc.), unmanned sweeping robots, unmanned cleaning vehicles, robots, unmanned sweeping trucks, sweeping robots, and other intelligent mobile tools. This invention does not limit these to any particular type of intelligent mobile tool.

[0019] Figure 1 The illustration schematically depicts a traffic light detection method according to an embodiment of the present invention. The executing entity of this method can be, for example, a sensing module, planning module, or controller on an autonomous or semi-autonomous vehicle; it can also be a processor of a traffic light detection device such as a smart tablet, personal PC, computer, or cloud server; or it can be a processor of an intelligent mobile tool such as an unmanned cleaning vehicle, unmanned sweeping vehicle, sweeping robot, autonomous vehicle, or semi-autonomous vehicle. The embodiments of the present invention do not limit this. Figure 1 As shown, the method of this embodiment of the invention includes: Operation S1: Obtain the first detection result of the traffic lights in the current frame image, wherein the first detection result includes the predicted coordinates of the traffic lights in the current frame image; Operation S2 determines the three-dimensional predicted coordinates of the corresponding traffic light in the high-precision map coordinate system based on the predicted coordinates of the traffic light. Operation S3: Construct a bipartite graph containing three-dimensional spatial information based on the first detection result, the three-dimensional predicted coordinates, and the high-precision map; Operation S4 involves performing a maximum matching calculation on the bipartite graph containing three-dimensional spatial information, and determining the second detection result of the traffic light in the current frame image based on the maximum matching calculation result.

[0020] Intelligent mobile tools, such as autonomous or semi-autonomous vehicles, acquire images of each frame from the real-world scene during operation using sensing devices such as monocular cameras, binocular cameras, and radar. These images serve as environmental input information perceived from the outside world, enabling the intelligent mobile tool to perform corresponding analysis and decisions based on the perceived image information. In this embodiment of the invention, the images used for traffic light detection preferably refer to RGB images acquired from the real-world scene by the camera devices on the intelligent mobile tool, such as monocular or binocular cameras. The camera devices acquire RGB images from the real-world scene frame by frame, and the current frame image refers to the frame image for which the intelligent mobile tool is currently performing traffic light recognition and detection. This frame can be the latest frame image acquired in real time or a historical frame image acquired in advance at a certain moment; this embodiment of the invention does not impose any limitations on this. In the embodiments of the present invention, the first detection result refers to the traffic light prediction result obtained by performing a first round of traffic light detection on the current frame image. This first round of traffic light detection differs from the traffic light detection method described in operations S2 to S4 of the embodiments of the present invention. That is, the traffic light detection method described in operations S2 to S4 can be regarded as a second round of detection result verification based on the first round of traffic light detection results. Therefore, compared with the traditional traffic light detection method of the prior art, the embodiments of the present invention achieve a significant improvement in the accuracy of traffic light detection and recognition results. The first round of traffic light detection can be obtained using existing traffic light detection methods, such as using a pre-trained target detection network model to perform traffic light detection and recognition on the current frame image, or using traditional image recognition processing methods to perform RGB image color density recognition on the current frame image. Alternatively, traffic light detection can be performed on two current frame images acquired by a binocular camera at the same time, and the detection results of the two images can be correlated to obtain the first detection result, etc. The embodiments of the present invention do not limit this. In this case, in operation S1, the embodiments of the present invention only need to obtain the first detection result output by other existing traffic light detection and recognition modules to be used in the traffic detection scheme of the embodiments of the present invention.

[0021] As a preferred implementation, the first detection result can also be generated using the adaptive traffic light detection method of this invention. In this embodiment, the adaptive traffic light detection method means that the target area of ​​interest (ROI) in the current frame image is dynamically determined in real time. That is, for each current frame image, as the current driving area changes, the surrounding environment and traffic light conditions corresponding to the current driving area change dynamically, resulting in different ROIs. The first detection result is obtained by detecting and recognizing traffic lights based on the dynamically determined ROI. Since the current driving area of ​​a smart mobile tool in a real-world application scenario changes dynamically, the perceived information about the surrounding environment also changes. Therefore, if a more effective ROI can be dynamically determined by combining the current driving area and the actual surrounding environment information, and traffic light detection is performed on the ROI, the recognition accuracy and detection efficiency of the first detection result can be effectively improved, thereby further assisting in improving the recognition accuracy and detection efficiency of the traffic light detection method of this invention. Figure 2 The diagram schematically illustrates a method flow for generating a first detection result using an adaptive traffic light detection method according to one embodiment of the present invention, such as... Figure 2 As shown, the method flow includes: Operation S11: Based on the current frame image and the high-precision map, determine the target area of ​​interest in the current frame image in real time; Operation S12: Extract the first target image from the current frame image based on the target region of interest; Operation S13 performs image preprocessing on the extracted first target image to generate the second target image; In operation S14, the second target image is input into a pre-trained target detection algorithm, and the output of the target detection algorithm is obtained as the first detection result.

[0022] High-definition maps (HD maps) are primarily used as prior data for autonomous driving services. HD maps clearly represent map elements in the surrounding environment, such as roads, lanes, roadside traffic signs, ground markings, the number of traffic lights, their locations (represented by three-dimensional coordinates in a spatial coordinate system), and their actual dimensions. In other words, the information provided by HD maps allows for the identification of the three-dimensional information of all spatial objects within the current driving area of ​​the intelligent mobile vehicle. Therefore, in operation S11, the target area of ​​interest (ROI) in the current frame image can be determined based on the traffic light locations and actual dimensions provided in the HD map. In this embodiment, when determining the ROI in the current frame image based on the HD map, the corresponding target projection area in the HD map is first determined by combining the current driving area. Then, the ROI of the current frame image is determined based on the traffic light locations and actual dimensions within the target projection area, ensuring that the determined ROI dynamically adapts to the current driving area. Determining the target projection area in a high-precision map means selecting only the portion of the high-precision map that is relevant to the current driving area as the target projection area. As a result, the amount of high-precision map data used to determine the target area of ​​interest is relatively small, thus resulting in high data processing efficiency and helping to determine the target area of ​​interest more accurately.

[0023] Preferably, Figure 3 This schematically describes a possible implementation for determining the region of interest in the current frame image in real time based on the current frame image and a high-precision map, such as... Figure 3 As shown, it can be implemented by including the following methods and procedures: Operate S111 to determine the target projection area on the high-precision map based on the current driving area; Operate S112 to obtain the high-precision map coordinates of traffic lights within the target projection area of ​​the high-precision map; Operation S113 projects the high-precision map coordinates of the traffic lights within the target projection area onto the image coordinate system of the current frame image, thus obtaining the image coordinates of the traffic lights corresponding to the high-precision map coordinates. Operation S114 determines the target region of interest in the current frame image based on the image coordinates corresponding to the traffic lights within the target projection area.

[0024] In operation S111, the high-precision map portion covered by the viewing cone range of a camera device (such as a camera) mounted on the intelligent mobile tool can be used as the target projection area in the high-precision map. When determining the target projection area, the pose of the intelligent mobile tool and the mounting position of the camera device on it can be obtained first. Then, combining the pose of the intelligent mobile tool and the mounting position of the camera device, the desired viewing cone range is determined based on a pre-set distance S. Subsequently, the desired viewing cone range of the camera device is calculated using existing technology. Taking the current frame image acquired by a camera positioned directly in front of the intelligent mobile tool as an example, based on the desired viewing cone range, after knowing that the camera's mounting position is directly in front of the intelligent mobile tool and the pose of the intelligent mobile tool, the high-precision map portion covered by the viewing cone range of the camera device at a certain distance S from the camera device acquiring the current frame image can be used as the target projection area in the high-precision map. The value of this certain distance S can be set based on experience and the desired viewing cone range, such as 100m. The camera's field of view (FOV) range can be determined based on the camera's focal length, field of view, the position of the clipping plane, and the camera coordinate system. The algorithm for determining this range can be found in existing technologies, which are known technologies for imaging devices such as cameras, and therefore will not be elaborated upon here. It should be noted that in this embodiment of the invention, the position of the clipping plane is determined based on a pre-set distance S, which defines the distance between the nearest clipping plane (also known as the imaging plane) and the farthest clipping plane (also known as the detection plane). The desired FVOV range refers to the spatial range covered by the imaging plane and the detection plane at a distance S from the imaging plane. Therefore, based on the pose of the intelligent mobile tool, the installation position of the camera device on the intelligent mobile tool, and the preset distance S, the desired FVOV range can be determined by referring to existing technologies. For example... Figure 4 The illustration demonstrates the principle and effect of determining the target projection area in a high-precision map based on the current driving area, such as... Figure 4As shown, taking a monocular camera on a smart mobile tool as an example, the spatial region defined between the detection plane and the imaging plane at a certain distance S from the camera's imaging plane can be used as the target projection area in the high-precision map. In this example, the certain distance S is set as the camera's farthest detection distance s. Correspondingly, the detection plane at a certain distance S from the camera's imaging plane is the camera's farthest detection plane in this case. Based on the camera's focal length, field of view, the position of the clipping plane, and the camera coordinate system, the coordinate range of this farthest detection plane in the camera coordinate system can be determined. Then, based on the camera coordinate system and the high-precision map coordinate system... The established transformation matrix allows the coordinate range of the farthest detection plane in the camera coordinate system and its distance S from the camera's imaging plane to be projected into the high-precision map coordinate system. This yields the spatial region of the farthest detection plane in the three-dimensional spatial coordinate system, i.e., the high-precision map coordinate system (the coordinate transformation method between the camera coordinate system and the high-precision map coordinate system can refer to existing technologies, which are also described later in this embodiment, so they will not be repeated here). The high-precision map area covered by this spatial region can be used as the target projection area, thus enabling the determination of the target projection area in the high-precision map based on the current driving area. Alternatively, after obtaining the camera's installation position, the camera's position coordinates can be directly transformed into the high-precision map coordinate system. Then, based on the predetermined distance S and other parameters used to calculate the camera's frustum range, the coordinate range of the detection plane at a distance S from the imaging plane can be directly calculated, and the spatial region covered by this coordinate range and its distance S from the imaging plane can be used as the target projection area. In cases where the camera, such as a video recording device, is installed in another location on the smart mobile tool, the position of the smart mobile tool, the installation location of the camera on the smart mobile tool, and a preset distance S can be used as a reference. Figure 4 The principle and related existing technology shown are used to determine the field of view of the camera device, and the high-precision map area covered by the field of view is used as the target projection area in the high-precision map.

[0025] In operation S112, the positions and actual sizes of the traffic lights included in the target projection area can be determined based on the traffic light locations and actual sizes in the high-precision map, as well as the coordinate range covered by the target projection area in the high-precision map. The traffic light locations in the high-precision map are identified by the high-precision map coordinates of the center point of the traffic light; therefore, the obtained traffic light locations are the high-precision map coordinates of the traffic lights. As some possible implementations, the traffic lights included in the target projection area can be determined based on their locations, i.e., their high-precision map coordinates. Specifically, it can be determined whether the traffic light locations fall within the spatial range covered by the target projection area, and traffic lights whose locations fall within this spatial range are identified as traffic lights included in the target projection area. In other possible implementations, the traffic lights included in the target projection area can be determined simultaneously based on the traffic light locations (i.e., the high-precision map coordinates of the traffic lights) and the actual size of the traffic lights. For example, by combining the traffic light locations and actual sizes, the coordinate range covered by the entire traffic light area or the coordinate range covered by a traffic light area larger than a preset area can be determined. Then, it is determined whether the entire traffic light area or the traffic light area larger than the preset area falls within the spatial range covered by the target projection area. Traffic lights whose entire traffic light area or the traffic light area larger than the preset area falls within the spatial range covered by the target projection area are identified as traffic lights included in the target projection area. The preset area can be determined based on the traffic light locations, actual sizes, and empirical values ​​(or requirements).

[0026] In operation S113, the high-precision map coordinates of the traffic lights within the target projection area obtained in operation S112 can be transformed into the image coordinate system one by one according to the known transformation method from the camera coordinate system to the high-precision map coordinate system. Then, the coordinates in the transformed image coordinate system are normalized to obtain the image coordinates of each traffic light within the target projection area corresponding to the high-precision map coordinates. Taking the traffic light detection using only one camera (the current frame image with only one viewpoint) as an example, the transformation method from points in the high-precision map to points in the camera image coordinate system can be determined by the following formula: Where point P(x,y,z) is a point in the high-precision map, point p(x*s,y*s,s) is a point in the image coordinate system, and R... wc T is the extrinsic parameter for the transition from the camera coordinate system to the high-precision map coordinate system. wc K is the transformation matrix from the camera coordinate system to the high-precision map coordinate system, and K is the camera intrinsic parameter.

[0027] After transforming the high-precision map coordinates of the traffic lights in the target projection area to the image coordinate system using the above formula, and then normalizing p, such as by performing p=p / p[2] normalization, the image coordinates corresponding to the high-precision map coordinates of the traffic lights can be obtained. It should be noted that the extrinsic parameters, transformation matrix and camera intrinsic parameters of the camera coordinate system to the high-precision map coordinate system can all be determined according to existing technology. For example, the extrinsic parameters and transformation matrix of the camera coordinate system to the high-precision map coordinate system can be obtained through calibration, and the camera intrinsic parameters can be obtained through camera calibration and distortion removal. Therefore, they will not be described in more detail here. It should be noted that in other possible implementations, when projecting the traffic lights in the target projection area to the image coordinate system, certain strategies can also be adopted to reduce the amount of computation and reduce noise. For example, only traffic lights of the type of interest can be selected for projection according to the type of traffic lights in the high-precision map, or only traffic lights at a certain distance away from the intelligent mobile tool, such as more than one meter, can be selected for projection.

[0028] In operation S114, as one possible implementation, the target region of interest in the current frame image can be determined based on the image coordinates corresponding to the traffic lights within the target projection area and the coordinate region where the current frame image is located; or, as another possible implementation, the target region of interest in the current frame image can also be determined based on the image coordinates corresponding to the traffic lights within the target projection area, the grid coordinates corresponding to the current frame image, and the grid coordinates corresponding to the previous frame image. The specific implementation schemes of these possible implementations will be described below through four concrete examples.

[0029] Taking the determination of the target region of interest (PGI) in the current frame image based on the image coordinates of traffic lights within the target projection area and the coordinate region where the current frame image is located as an example, the PPI in the current frame image can be determined by the maximum x-axis value, maximum y-axis value, minimum x-axis value, minimum y-axis value in the image coordinates of traffic lights within the target projection area, and the coordinate range of the current frame image. Specifically, assuming there are n traffic lights in the target projection area of ​​the high-precision map determined by the aforementioned steps, and the image coordinates corresponding to the high-precision map coordinates of the i-th traffic light are p... i Given (x, y), by performing operations S111 to S113, we can obtain the set of image coordinates of all traffic lights that meet the requirements (i.e., within the target projection area). Based on this, the boundaries of all image coordinates can be calculated first using the following algorithm as the initial target region of interest (x). min y min x max y max ): Where, x min That is, the minimum x-axis value in the image coordinates corresponding to the traffic lights within the target projection area, x max That is, the maximum x-axis value in the image coordinates corresponding to the traffic lights within the target projection area, and the maximum y-axis value. min That is, the minimum y-axis value in the image coordinates corresponding to the traffic lights within the target projection area. max This refers to the maximum y-axis value in the image coordinates corresponding to the traffic lights within the target projection area.

[0030] After obtaining the initial region of interest (ROI), to ensure that the final ROI is within the image range of the current frame, boundary preprocessing is required. Assuming the image size range of the current frame is (0, 0, width, height), the following algorithm can be used to further preprocess the initial ROI to obtain the final ROI (x...). min y min x max y max ): Continuing with the example of determining the region of interest in the current frame image based on the image coordinates of the traffic lights within the target projection area and the coordinate region where the current frame image is located, for instance, it can also be done by... Figure 5 The method shown determines the region of interest in the current frame image, such as Figure 5 As shown, it includes: Operation S1141 divides the current frame image into multiple grids of preset size and determines the grid coordinates corresponding to each grid. Operation S1143: Determine the grid into which the traffic light falls based on the image coordinates of the traffic light within the target projection area and the grid coordinates of each grid. Operation S1145 determines the region of interest in the current frame image based on the grid coordinates of the grid into which the traffic light falls.

[0031] The preset grid size can be set according to needs and experience, and the grid coordinates can include the grid center point coordinates, grid length, and width. Since both the traffic light image coordinates and grid coordinates are based on the image coordinate system, the grid to which each traffic light falls can be directly determined by judging whether the image coordinates of each traffic light fall within the coordinate range covered by the grid coordinates. In operation S1145, the area commonly covered by all the grids to which the traffic lights fall within the target projection area is taken as the target area of ​​interest (ROI) of the current frame image; that is, the coordinate area defined by the outer boundaries of all the grids to which the traffic lights fall is taken as the ROI. Figure 6 Taking the effect shown in the image as an example, as in Figure 6 As shown, when the traffic light falls into only one grid, the area covered by the grid coordinates of the grid M into which the traffic light falls is taken as the target region of interest in the current frame image. Similarly, in other possible embodiments, when there are two grids into which the traffic light falls, such as grid N and grid P, the area covered by both grids N and P is taken as the target region of interest in the current frame image.

[0032] Taking the determination of the target region of interest in the current frame image based on the image coordinates of the traffic lights within the target projection area, the raster coordinates of the current frame image, and the raster coordinates of the previous frame image as an example, it can be exemplarily achieved by... Figure 7 The method shown determines the region of interest in the current frame image, such as Figure 7 As shown, it includes: Operation S1140 divides the current frame image and the image of the previous frame image into multiple grids of preset size, and determines the grid coordinates corresponding to each grid. Operation S1142: Determine the target grid in the current frame where the traffic light falls based on the image coordinates of the traffic light in the target projection area and the grid coordinates of each grid in the current frame image. Operation S1144: Determine the target grid of the historical frame into which the traffic light falls based on the image coordinates of the traffic light in the target projection area and the grid coordinates of each grid in the previous frame of the current frame image. Operation S1146 determines the region of interest in the current frame image based on the grid coordinates of the target grid in the current frame and the grid coordinates of the target grid in the historical frames.

[0033] exist Figure 7In the embodiment shown, the traffic lights in the target projection area are simultaneously referenced in the current frame image and the previous frame image, where the traffic lights fall into the grid. In operation S1146, the target grid of the current frame and the target grid of the historical frame are merged, and the area occupied by the merged grid is taken as the target region of interest of the current frame image. That is, the area included by the outer boundary coordinates of the merged grid is selected as the target region of interest of the current frame image. Figure 8 and 9 The process and its effects are shown, such as Figure 8 The image shown is the result obtained after performing operation S1144 on the (n-1)th frame image. Specifically, in the (n-1)th frame image, the traffic light occupies one grid H, while... Figure 9 After performing operation S1142 on the current frame image, i.e. the nth frame image, the result is that it occupies three grids: grid J, grid K, and grid L. In operation S1146, the region after merging grids H, J, K, and L is taken as the target region of interest of the current frame image.

[0034] Continuing with the example of determining the region of interest in the current frame image based on the image coordinates of the traffic lights within the target projection area, the raster coordinates of the current frame image, and the raster coordinates of the previous frame image, it can also be exemplified by... Figure 10 The method shown determines the region of interest in the current frame image, such as Figure 10 As shown, it includes: Operation S1147 divides the current frame image into multiple grids of preset size and determines the grid coordinates corresponding to each grid. Operation S1142: Determine the target grid into which the traffic light falls based on the image coordinates of the traffic light within the target projection area and the grid coordinates of each grid in the current frame image. Operation S1148: Obtain the historical target grid in which the traffic light falls from the image of the previous frame of the current frame; Operation S1149 determines the region of interest in the current frame image based on the grid coordinates of the target grid and the grid coordinates of the historical target grid.

[0035] exist Figure 10 In the illustrated implementation, the traffic light falling into the grid is considered simultaneously with the current frame and the previous frame. This is because the previous frame has already been processed for the current frame image. Figure 10All the processing in operations S1147 to S1149 in the current frame image means that the processing result of operation S1142 of the previous frame image, i.e., the target grid into which the traffic light falls, is known prior information. Therefore, in operation S1148, the processing result of operation S1142 of the previous frame image can be directly obtained and used as the historical target grid in the current frame image. In operation S1149, the target grid of the current frame and the historical target grid are merged, and the area occupied by the merged grid is used as the target region of interest of the current frame image. Figure 8 and 9 It can also be used to represent the process and effect, such as Figure 8 The image shown is the result obtained after performing operation S1142 on the (n-1)th frame image. Specifically, in the (n-1)th frame image, the traffic light occupies one grid H, while... Figure 9 After performing operation S1142 on the current frame image (i.e., the nth frame image), the result is that it occupies three grids: grid J, grid K, and grid L. In operation S1149, the region after merging grids H, J, K, and L is specifically taken as the target area of ​​interest (ROI) for the current frame image. Since the positions of traffic lights in two consecutive frames are usually not too far apart, combining the projected positions of traffic lights in two consecutive frames can better reduce the problem of inaccurate ROI selection that may be caused by considering only the current frame image. Especially when high-precision maps are not updated in a timely manner, it can further reduce the problem of inaccurate ROI selection caused by changes in traffic light size, etc., thereby helping to further improve the accuracy of traffic light detection results.

[0036] After determining the region of interest (ROI) corresponding to the current frame image, in operation S12, existing image processing techniques such as cropping can be used to extract the ROI from the current frame image, and the extracted image is used as the first target image. Then, in operation S13, image preprocessing of the extracted first target image can involve adjusting its size and resolution, thereby generating a second target image that meets the desired specifications. In one embodiment, the image preprocessing in operation S13 involves setting the size of the second target image to be the same as the size of the current frame image, and setting the resolution of the second target image to be no greater than the resolution of the current frame image. Therefore, in this embodiment of the invention, the image size of the second target image generated after cropping and preprocessing based on the determined ROI is enlarged to the original size of the current frame image, and the resolution is set to be the same as or clearer than that of the current frame image. Therefore, when performing target detection based on the second target image in operation S14, it is easier to detect small objects, thereby enabling a longer detection distance and a larger minimum detected pixel in traffic light detection.

[0037] Preferably, in operation S14, the pre-trained target detection algorithm is a CNN-based target detection algorithm model, such as Faster R-CNN, which takes the second target image as input and the first detection result as output. In other embodiments, the pre-trained target detection algorithm can also be a traditional image processing algorithm model or other existing algorithm models that detect traffic lights based on RGB images. Since the adaptive traffic light detection method for obtaining the first detection result in this embodiment of the invention actually only performs traffic light detection on the image region within the target area of ​​interest, compared to performing traffic light detection on the entire current frame image, the image size input to the target detection algorithm is appropriately enlarged to the size of the current frame image. Therefore, the size of the traffic light in the two-dimensional image is also enlarged, making the target more obvious, thus achieving better detection results. Furthermore, since the second target image input to the target detection algorithm model is obtained based on the target area of ​​interest determined dynamically in real time, it can better adapt to the surrounding environment of the current driving area, which helps to improve the accuracy of the detection results.

[0038] In other embodiments, the image preprocessing performed in operation S13 may further involve setting the size of the second target image to be smaller than the size of the current frame image. Thus, the adaptive traffic light detection method for obtaining the first detection result in this embodiment of the invention actually only performs traffic light detection on the image region within the target area of ​​interest. Compared to performing traffic light detection on the entire current frame image, the image size input to the target detection algorithm is smaller, allowing the entire target detection algorithm to run faster. In other embodiments, the image preprocessing performed in operation S13 may further involve setting the size of the second target image to be larger than the size of the first target image but smaller than the size of the current frame image. In this case, the image size input to the target detection algorithm is smaller, and since the size of the traffic light in the two-dimensional image is also magnified (i.e., the size of the second target image is larger than the size of the first target image), not only can the entire target detection algorithm run faster, but the target is also more obvious, making it easier to detect small objects. Therefore, better detection results can be obtained, and the detection distance during traffic light detection can be further, with a larger minimum detected pixel. Furthermore, since the second target image input to the target detection algorithm model is based on the target region of interest determined dynamically in real time, it can better adapt to the surrounding environment of the current driving area, which helps to improve the accuracy of the detection results.

[0039] In a preferred embodiment, after obtaining the output result of the target detection algorithm, the coordinate data in the output result can be further projected onto the image coordinate system of the current frame image, and a first detection result is formed based on the projection result. Projecting the coordinate data onto the current frame image coordinate system mainly involves proportionally transforming the coordinate position and size range of the traffic light output by the target detection algorithm to the size of the current frame image, based on the size of the second target image and the size of the current frame image, thereby forming the final first detection result. Since the specific implementation process of the corresponding coordinate transformation based on the proportional image size can be achieved using existing technology, its process will not be described in detail here.

[0040] In this embodiment of the invention, the first detection result of traffic lights is correlated and verified based on a high-precision map to improve the accuracy of the traffic light detection results. Since the predicted coordinates of the traffic lights determined in the first detection result are two-dimensional information in an image coordinate system, while the high-precision map is three-dimensional information in a spatial coordinate system, to verify the first detection result of traffic lights using a high-precision map—that is, to associate and match the traffic light data in the first detection result with the traffic light data in the high-precision map—it is necessary to unify the coordinate systems. As a possible implementation, this embodiment of the invention transforms the predicted coordinates of the traffic lights to the high-precision map coordinate system, thereby achieving further detection and verification of traffic lights in three-dimensional space, thus improving the accuracy of traffic light detection using the data information from the high-precision map. Based on this, in operation S2, this embodiment of the invention first needs to determine the three-dimensional predicted coordinates of the corresponding traffic lights in the high-precision map coordinate system based on the predicted coordinates of the traffic lights. As a possible implementation... Figure 11 The illustration schematically depicts a method for determining the three-dimensional predicted coordinates of a traffic light in a high-precision map coordinate system based on the predicted coordinates of the traffic light, according to one embodiment of the present invention. Figure 11 As shown, it can be implemented as including: Operation S21 assigns a hypothetical depth to the traffic light and forms the first hypothetical coordinates of the traffic light in the image coordinate system of the current frame image based on the assigned hypothetical depth and the predicted coordinates of the traffic light. Operation S22 projects the first hypothetical coordinates of the traffic light onto the high-precision map coordinate system, forming the second hypothetical coordinates of the traffic light in the high-precision map coordinate system; Operation S23: Obtain the camera coordinates of the camera center point in the high-precision map coordinate system, and determine the actual depth corresponding to the traffic light based on the camera coordinates, the second assumed coordinates of the traffic light, and the point cloud data in the high-precision map. Operation S24 determines the three-dimensional predicted coordinates of the corresponding traffic light based on the actual depth and the second assumed coordinates.

[0041] Because the predicted coordinates of traffic lights in the current frame image acquired by a monocular camera lack depth information in the image coordinate system, obtaining the depth information of the detected traffic lights in the current frame image is a key challenge in converting the predicted coordinates of traffic lights in the current frame image into the corresponding 3D predicted coordinates of the traffic lights in a high-precision map coordinate system, especially in scenarios where the camera device is a monocular camera. To address the lack of depth information for traffic lights in the current frame image in monocular camera and similar scenarios, in... Figure 11 In the illustrated implementation, point cloud data from a high-precision map is used to determine the depth information of traffic lights. Specifically, in operation S21, for example, a hypothetical depth is assigned to each of the n traffic lights in the current frame image determined based on the first detection result, thereby giving the i-th traffic light a hypothetical depth h. i Accordingly, the first hypothetical coordinates (px) of the i-th traffic light in the image coordinate system of the current frame image can be formed. i py i h i ), where (px i py i Let (Px) be the predicted coordinates of the i-th traffic light. Optionally, the value of the assumed depth assigned to the traffic light can be any value. In a preferred embodiment, considering that the traffic lights that need to be detected are usually a certain distance away from the vehicle, the assumed depth can be set to an empirical value close to the usual traffic light distance or a value close to the preset distance S when determining the view frustum range. Then, in operation S22, the first assumed coordinates can be directly transformed to project the first assumed coordinates of the n traffic lights in the current frame image determined based on the first detection result onto the high-precision map coordinate system, thereby obtaining the second assumed coordinates of the projected high-precision map coordinate system. For example, the second assumed coordinates of the i-th traffic light in the high-precision map coordinate system after projection are (Px) i Py i H iThe projection of the first hypothetical coordinates in the image coordinate system onto the high-precision map coordinate system can be implemented based on camera parameters (including extrinsic parameters, transformation matrix, and camera intrinsic parameters from the camera coordinate system to the high-precision map coordinate system) with reference to existing technologies, and will not be elaborated further in this embodiment of the invention. After determining the second hypothetical coordinates of each traffic light in the high-precision map coordinate system, in operation S23, the actual depth corresponding to each traffic light can be calculated. Specifically, for the i-th traffic light, a straight line equation passing through both the camera center point and the i-th traffic light can be determined based on the camera coordinates and the second hypothetical coordinates of the i-th traffic light. Then, the point cloud data in the high-precision map is traversed to determine the point cloud passing through the straight line equation, and the depth data corresponding to the point cloud passing through the straight line equation is taken as the actual depth corresponding to the i-th traffic light. It should be noted that in operation S23, when determining the equation of the straight line that simultaneously passes through the camera center point and the i-th traffic light, the camera coordinates used are the coordinates of the camera center point in the high-precision map coordinate system. As one possible implementation, the current pose of the camera can be calculated based on the positioning results of the intelligent mobile tool carrying the camera and the camera calibration parameters. Then, based on the current pose of the camera and the transformation relationship between the camera coordinate system and the high-precision map coordinate system, the camera coordinates of the camera center point in the high-precision map coordinate system can be determined. The calculation process can refer to relevant existing technologies and will not be elaborated here. For example, Figure 12 The schematic diagram illustrates the processing effect of operation S23, such as... Figure 12As shown, a straight line equation is determined by using the camera coordinates in the high-precision map coordinate system corresponding to the camera center point and the second assumed coordinates of the i-th traffic light P_pix. Then, point cloud data corresponding to the traffic lights in the target projection area of ​​the high-precision map is acquired. The coordinates of the point cloud data corresponding to the traffic lights in the target projection area are substituted into the straight line equation to determine whether the corresponding point cloud also passes through the straight line equation. When a point cloud passing through the straight line equation is found, the depth data corresponding to the point cloud passing through the straight line equation is taken as the actual depth of the i-th traffic light P_pix. In a preferred embodiment, when substituting the coordinates of the point cloud data corresponding to the traffic lights in the target projection area into the straight line equation to determine whether the corresponding point cloud also passes through the straight line equation, a certain deviation is allowed. That is, if the Euclidean distance between the point cloud and its projection point on the straight line equation is less than a preset threshold, the point cloud can also be considered to have passed through the straight line equation. It should be noted that when multiple point clouds in the point cloud data corresponding to traffic lights in the target projection area pass through the straight line equation corresponding to the i-th traffic light P_pix, the average depth of these multiple point clouds can be used as the actual depth of the i-th traffic light P_pix, or the minimum depth value among these multiple point clouds can be used as the actual depth of the i-th traffic light P_pix. After determining the actual depth of the traffic light, in operation S24, the actual depth of the traffic light determined in operation S23 is used to replace the depth data in the second assumed coordinates to obtain the three-dimensional predicted coordinates of the corresponding traffic light in the high-precision map coordinate system. Since traffic lights are generally located at relatively high positions in actual map scenarios and are not easily obscured by other objects, this embodiment of the invention determines a straight line equation using the camera center point and the traffic light, and uses the point cloud data of the traffic light in the target projection area of ​​the high-precision map to infer the actual depth of the traffic light in the first detection result. This effectively solves the drawback of missing depth data in scenarios such as monocular cameras, which makes it impossible to detect traffic lights in three-dimensional space. This allows for the detection of traffic lights in three-dimensional space using image data, improving the accuracy of traffic light detection results.

[0042] In other embodiments, binocular stereo matching can be used to determine the depth information of traffic lights in the first detection result. In this embodiment, a binocular camera, including a left and a right camera, can be installed on a smart mobile device. Then, using the principle of depth measurement by binocular cameras in existing technology, the actual depth of each traffic light can be determined. The distance measurement principle of binocular cameras is based on the observation method of human eyes. It calculates the disparity through images from two perspectives, thereby inferring the depth information of the object. In existing technology, its working principle includes several steps such as camera calibration, binocular correction, binocular matching, and depth information calculation. Camera calibration is used to calibrate the binocular cameras to obtain the intrinsic and extrinsic parameters of each camera, the homography matrix, and the relative positional relationship between the two cameras (rotation matrix and translation vector). Binocular correction is used to correct the original images based on the calibration results, ensuring that the two images are on the same plane. Furthermore, they are parallel to each other to facilitate subsequent matching and calculation. Binocular matching is used to match pixels of the two corrected images, that is, to find corresponding points of the same scene in the two views. Calculating depth information is used to calculate the depth of each pixel based on the matching results, thereby obtaining a depth map. This process utilizes the principle of similar triangles. By using known disparity (the distance between two imaging points) and camera parameters (such as focal length, baseline length, etc.), the depth information of a point in space can be calculated. The principle of calculating depth information using a binocular camera is based on existing technologies and will not be elaborated here. In this embodiment of the invention, the only difference from the prior art is that operation S2 of this embodiment is to obtain the three-dimensional predicted coordinates of the traffic lights detected in the current frame image under the high-precision map coordinate system. That is, in this embodiment of the invention, the depth information of the predicted coordinates of the traffic lights in the first detection result is determined respectively. Therefore, in this embodiment of the invention, the pixel points of each traffic light in the binocular camera will be found first based on the predicted coordinates of each traffic light, and then the depth data corresponding to the corresponding pixel points will be calculated according to the principle of calculating depth information using a binocular camera, and used as the actual depth of the corresponding traffic light. Then, based on the predicted coordinates and actual depth of the traffic lights, the three-dimensional coordinates of each traffic light in the image coordinate system are formed, and the three-dimensional coordinates of each traffic light in the image coordinate system are projected and transformed to obtain the three-dimensional predicted coordinates of each traffic light in the high-precision map coordinate system.

[0043] After converting the traffic light predicted coordinates in the image coordinate system of the first detection result to the high-precision map coordinate system, in operation S3 of this embodiment, the invention will further construct a bipartite graph containing three-dimensional spatial information based on the first detection result, the three-dimensional predicted coordinates, and the high-precision map. This transforms the association and matching problem between the traffic light data in the first detection result and the traffic light data in the high-precision map into a maximum matching problem of a bipartite graph in three-dimensional space. Through this transformation, this embodiment can utilize existing algorithms for solving the maximum matching problem of bipartite graphs to achieve the association and matching of traffic light data in the first detection result and the high-precision map, thereby efficiently verifying the first detection result and ensuring that the final determined second detection result of the traffic lights is more objective and accurate. As one possible implementation method, Figure 13 The diagram schematically illustrates a method for constructing a bipartite graph containing three-dimensional spatial information according to an embodiment of the present invention, such as... Figure 13 As shown, it can be implemented as including: Operation S31: The traffic light in the current frame image determined based on the first detection result is taken as the left vertex of the bipartite graph, and the traffic light in the target projection area in the high-precision map is taken as the right vertex of the bipartite graph. Operation S32 determines the matching degree between the traffic light at the left vertex and the traffic light at the right vertex based on the first detection result and the three-dimensional predicted coordinates of the traffic light corresponding to the left vertex, as well as the information data of the traffic light corresponding to the right vertex in the high-precision map. Operation S33 constructs the edges of the bipartite graph based on the matching degree between the traffic lights at the left and right vertices.

[0044] In operation S32, the matching degree can be designed according to requirements. It can be used to limit the cost value of the bipartite graph. The maximum matching problem of the bipartite graph is actually the optimal matching of cost value. Therefore, by designing the matching degree according to the actual matching requirements, the final maximum matching can satisfy the matching requirements to the greatest extent. For example, in one implementation, the matching degree includes, but is not limited to, a first matching degree determined based on the overlapping area of ​​the detection box information in the first detection result of the traffic light corresponding to the left vertex and the contour box information of the traffic light corresponding to the right vertex in the high-precision map; a second matching degree determined based on the consistency of the traffic light type in the first detection result of the traffic light corresponding to the left vertex and the type information of the traffic light corresponding to the right vertex in the high-precision map; and a third matching degree determined based on the three-dimensional Euclidean distance defined by the three-dimensional predicted coordinates of the traffic light corresponding to the left vertex and the center point coordinates of the traffic light corresponding to the right vertex in the high-precision map. In this case, the first detection result also includes detection box information and traffic light type. In operation S22, the construction of the edge of the bipartite graph based on the matching degree of the traffic light at the left vertex and the traffic light at the right vertex can be specifically implemented by traversing the traffic lights at the left vertex and the traffic lights at the right vertex. When the matching degree of the traffic lights at the left vertex and the traffic lights at the right vertex meets the preset conditions, an edge is constructed between the corresponding traffic lights at the left vertex and the traffic lights at the right vertex. The preset conditions for the matching degree can include: the first matching degree is satisfied that the overlapping area of ​​the detection box information in the first detection result of the traffic light corresponding to the left vertex and the contour box information of the traffic light corresponding to the right vertex in the high-precision map is greater than a preset threshold; the second matching degree is satisfied that the traffic light type in the first detection result of the traffic light corresponding to the left vertex is the same as the type information of the traffic light corresponding to the right vertex in the high-precision map; and the third matching degree is satisfied that the three-dimensional Euclidean distance defined by the three-dimensional predicted coordinates of the traffic light corresponding to the left vertex and the center point coordinates of the traffic light corresponding to the right vertex in the high-precision map is less than a preset threshold.

[0045] In some other possible implementations, the first detection result may also include the orientation of the traffic light, and the matching degree may also include a fourth matching degree determined based on the consistency between the orientation of the traffic light in the first detection result of the traffic light corresponding to the left vertex and the orientation of the traffic light corresponding to the right vertex in the high-precision map. The preset conditions for the aforementioned matching degree may also include: the first matching degree satisfies that the overlapping area of ​​the detection box information in the first detection result of the traffic light corresponding to the left vertex and the outline box information of the traffic light corresponding to the right vertex in the high-precision map is greater than a preset threshold, and the second matching degree satisfies that the type of the traffic light in the first detection result of the traffic light corresponding to the left vertex is the same as the type information of the traffic light corresponding to the right vertex in the high-precision map, and the fourth matching degree satisfies that the orientation of the traffic light in the first detection result of the traffic light corresponding to the left vertex is the same as the orientation of the traffic light corresponding to the right vertex in the high-precision map.

[0046] In other possible implementations, the first detection result may also include the length, width, and height of the traffic light, and the matching degree may also include a fifth matching degree determined based on the consistency between the length, width, and height of the traffic light in the first detection result corresponding to the traffic light at the left vertex and the length, width, and height of the traffic light at the right vertex in the high-precision map. The preset conditions for the aforementioned matching degree may also include: the first matching degree satisfies that the overlapping area between the detection box information in the first detection result corresponding to the traffic light at the left vertex and the outline box information in the high-precision map corresponding to the traffic light at the right vertex is greater than a preset threshold; the second matching degree satisfies that the type of the traffic light in the first detection result corresponding to the traffic light at the left vertex is the same as the type information of the traffic light at the right vertex in the high-precision map; the fourth matching degree satisfies that the orientation of the traffic light in the first detection result corresponding to the traffic light at the left vertex is the same as the orientation of the traffic light at the right vertex in the high-precision map; and the fifth matching degree satisfies that the error between the length, width, and height of the traffic light in the first detection result corresponding to the traffic light at the left vertex and the length, width, and height of the traffic light at the right vertex in the high-precision map is less than a preset threshold. Of course, in other possible implementations, more diverse matching degrees can be designed according to actual needs, and preset conditions for satisfying the matching degree can be set according to the needs. This embodiment of the invention does not limit this. It should also be noted that the matching degree mentioned in this embodiment of the invention can be freely combined according to the preset conditions for satisfying the matching degree. The foregoing examples are only some examples and are not considered as specific restrictions on the preset conditions.

[0047] In operation S4, an existing algorithm for calculating the maximum matching of bipartite graphs, such as the spatial Hungarian algorithm, can be selected. The constructed bipartite graph is used as input, and the best matching of the left and right vertices in the bipartite graph is used as output. This achieves the calculation of the best matching of the constructed bipartite graph. The left and right vertices in the output best matching can be considered as the same traffic light. This achieves the best matching association between the traffic lights in the first detection result and the traffic lights in the high-precision map, using the traffic light data information in the high-precision map to verify the validity of the traffic light detection results. Since the left and right vertices used in the construction of the bipartite graph in this embodiment of the invention include three-dimensional spatial information data, the constructed bipartite graph can represent three-dimensional spatial information. Based on this, performing maximum matching on the bipartite graph considers not only two-dimensional information but also three-dimensional spatial information, thus effectively associating the traffic lights detected in the first detection result with the actual traffic lights in the high-precision map. This helps to determine the precise location of the traffic lights and improves the accuracy of the traffic light detection algorithm.

[0048] Figure 14 The diagram schematically illustrates a traffic light detection device 500 according to one embodiment of the present invention. This device can be applied to mobile vehicles such as autonomous / semi-autonomous vehicles, drones, and robots to detect traffic lights in the surrounding environment of the mobile vehicle. Figure 14 As shown, the device includes: The image detection result acquisition module 501 is used to acquire a first detection result of the traffic lights in the current frame image, wherein the first detection result includes the predicted coordinates of the traffic lights in the current frame image; The 3D conversion module 502 is used to project the predicted coordinates of traffic lights onto a high-precision map coordinate system to generate the corresponding 3D predicted coordinates of traffic lights. The bipartite graph construction module 503 is used to construct a bipartite graph containing three-dimensional spatial information based on the first detection result, the three-dimensional predicted coordinates and the high-precision map. The matching module 504 is used to perform maximum matching calculation on the bipartite graph containing three-dimensional spatial information, and determine the second detection result of the traffic light in the current frame image based on the maximum matching calculation result.

[0049] It should be noted that the specific implementation methods of the image detection result acquisition module, the 3D transformation module, the bipartite graph construction module, and the matching module, such as the method of generating the first detection result, the method of generating the 3D predicted coordinates, and the method of constructing the bipartite graph, can all be referred to the previous description and will not be repeated here.

[0050] Figure 15 A computing device according to an embodiment of the present invention is schematically shown in the figure, which is implemented as including: Memory 1 is used to store executable instructions; and The processor 2 is communicatively connected to the memory 1 and is used to execute executable instructions stored in the memory, which, when executed by the processor, implement the steps of the traffic light detection method described in any of the foregoing embodiments.

[0051] In practice, for example, the aforementioned traffic light detection device and computing device can be applied to autonomous vehicles, drones, robots, and other autonomous or semi-autonomous driving equipment to detect traffic lights in the driving area where these devices are located. This enables these semi-autonomous or unmanned driving tools to perform downstream processing such as vehicle posture optimization, path planning, and driving strategy formulation based on the actual traffic light conditions, thereby improving the safety, reliability, and comfort of the autonomous driving system.

[0052] Figure 16 An autonomous vehicle according to one embodiment of the present invention is illustrated schematically, such as... Figure 16 As shown, the autonomous vehicle includes a computing device 70, enabling it to detect and determine traffic lights in its surrounding environment using the functions provided by the computing device, and then perform subsequent processing such as path planning based on the determined traffic lights. The computing device can be... Figure 15 The computing device or other similar electronic device shown.

[0053] Optionally, in practical applications, autonomous vehicles may also include a perception and recognition module and other planning and control modules, such as a path planning controller, a low-level controller, etc. The functions of the computing device 70 may also be implemented in the perception and recognition module or the planner, etc., and the embodiments of the present invention do not limit this.

[0054] In some embodiments, the present invention provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by an electronic device (including but not limited to a computer, server, or network device, etc.) to perform the traffic light detection method of any of the above embodiments of the present invention.

[0055] In some embodiments, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the traffic light detection method of any of the above embodiments.

[0056] In some embodiments, the present invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the traffic light detection method of any of the above embodiments.

[0057] In some embodiments, the present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the traffic light detection method of any of the above embodiments.

[0058] Figure 17 This is a schematic diagram of the hardware structure of a traffic light detection device according to another embodiment of the present invention. The traffic light detection device can be implemented using the structure shown in this figure, as follows: Figure 17 As shown, the traffic light detection device includes: One or more processors 610 and memory 620, Figure 17 Take the 610 processor as an example.

[0059] The traffic light detection device may also include an input device 630 and an output device 640.

[0060] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 17 Taking the example of a connection between China and Israel via a bus.

[0061] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the traffic light detection method in this embodiment of the invention. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the traffic light detection method of the above-described method embodiment.

[0062] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the traffic light detection method, etc. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0063] Input device 630 can receive input digital or character information and generate signals related to user settings and function control of the image processing device. Output device 640 may include a display device such as a display screen.

[0064] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they execute the traffic light detection method in any of the above method embodiments.

[0065] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0066] The electronic devices of this invention exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0067] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0068] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0069] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0070] (5) Other electronic devices with data interaction functions.

[0071] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic light detection method, characterized in that, include: Obtain a first detection result for traffic lights in the current frame image, wherein the first detection result includes the predicted coordinates of traffic lights in the current frame image; The three-dimensional predicted coordinates of the traffic lights in the high-precision map coordinate system are determined based on the predicted coordinates of the traffic lights. Based on the first detection result, the three-dimensional predicted coordinates, and the high-precision map, a bipartite graph containing three-dimensional spatial information is constructed. The maximum matching calculation is performed on the bipartite graph containing three-dimensional spatial information, and the second detection result of the traffic light in the current frame image is determined based on the maximum matching calculation result.

2. The method according to claim 1, characterized in that, The construction of a bipartite map containing three-dimensional spatial information based on the first detection result, the three-dimensional predicted coordinates, and the high-precision map includes: The traffic lights in the current frame image determined based on the first detection result are taken as the left vertex of the bipartite graph, and the traffic lights in the target projection area in the high-precision map are taken as the right vertex of the bipartite graph. The matching degree between the traffic lights at the left and right vertices is determined based on the first detection result and three-dimensional predicted coordinates of the traffic light corresponding to the left vertex, as well as the information data of the traffic light corresponding to the right vertex in the high-precision map. Construct the edges of the bipartite graph based on the matching degree between the traffic lights at the left and right vertices.

3. The method according to claim 2, characterized in that, The first detection result also includes detection box information and traffic light type, and the matching degree includes a first matching degree, a second matching degree, and a third matching degree, wherein, The first matching degree is determined based on the overlap area between the detection box information in the first detection result of the traffic light corresponding to the left vertex and the outline box information of the traffic light corresponding to the right vertex in the high-precision map. The second matching degree is determined based on the consistency between the traffic light type in the first detection result of the traffic light corresponding to the left vertex and the type information of the traffic light corresponding to the right vertex in the high-precision map. The third matching degree is determined based on the three-dimensional Euclidean distance defined by the three-dimensional predicted coordinates of the traffic light corresponding to the left vertex and the center point coordinates of the traffic light corresponding to the right vertex in the high-precision map.

4. The method according to claim 1, characterized in that, The process of determining the three-dimensional predicted coordinates of the traffic light in the high-precision map coordinate system based on the predicted coordinates of the traffic light includes: Assign a hypothetical depth to the traffic light, and form the first hypothetical coordinates of the traffic light in the image coordinate system of the current frame image based on the assigned hypothetical depth and the predicted coordinates of the traffic light; The first hypothetical coordinates of the traffic light are projected onto the high-precision map coordinate system to form the second hypothetical coordinates of the traffic light in the high-precision map coordinate system; Obtain the camera coordinates of the camera center point in the high-precision map coordinate system, and determine the actual depth corresponding to the traffic light based on the camera coordinates, the second hypothetical coordinates of the traffic light, and the point cloud data in the high-precision map. The three-dimensional predicted coordinates of the traffic light are determined based on the actual depth of the traffic light and the second assumed coordinates.

5. The method according to claim 1, characterized in that, The first detection result is generated in the following way: The target region of interest in the current frame image is determined in real time based on the current frame image and the high-precision map. The first target image is extracted from the current frame image based on the region of interest; The extracted first target image is preprocessed to generate the second target image; The second target image is input into a pre-trained target detection algorithm, and the output result of the target detection algorithm is obtained as the first detection result. Alternatively, the coordinate data in the output result is projected onto the image coordinate system of the current frame image, and the first detection result is formed based on the projection result.

6. The method according to claim 5, characterized in that, The step of determining the region of interest in the current frame image in real time based on the current frame image and the high-precision map includes: Determine the target projection area on the high-precision map based on the current driving area; Obtain the high-precision map coordinates of traffic lights within the target projection area of ​​a high-precision map; Project the high-precision map coordinates of the traffic lights within the target projection area onto the image coordinate system of the current frame image to obtain the image coordinates corresponding to the high-precision map coordinates. The target region of interest in the current frame image is determined based on the image coordinates corresponding to the traffic lights within the target projection area.

7. The method according to claim 6, characterized in that, Determining the region of interest in the current frame image based on the image coordinates corresponding to the traffic lights within the target projection area includes: The target region of interest in the current frame image is determined by the image coordinates of the traffic lights within the target projection area and the coordinate region where the current frame image is located. Alternatively, the target region of interest in the current frame image can be determined based on the image coordinates of the traffic lights within the target projection area, the raster coordinates of the current frame image, and the raster coordinates of the previous frame image.

8. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

9. An autonomous vehicle, characterized in that, Including the electronic device according to claim 8.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

Images