Three-dimensional reconstruction method, device and equipment of traffic sign position and storage medium

Abstract

The embodiment of the present specification discloses a method and device for three-dimensional reconstruction of traffic sign position, equipment and storage medium. First, the position frame of the target traffic sign is determined in the continuous N frames of traffic images, and the first number of pixels between the position frames in the adjacent two frames of traffic images is determined. Since the smaller the first number is, the greater the possibility that the two position frames corresponding to the first number correspond to the same target traffic object is, therefore, based on the first number, the target associated frame combination is determined in the position frame in the N frames of traffic images. Further, based on the position information of the target position frame in the target associated frame combination and the shooting pose information corresponding to the target traffic image to which the target position frame belongs, three-dimensional reconstruction is performed to obtain the three-dimensional position information of the target traffic sign. Through the shooting pose information, the three-dimensional position of the target traffic sign is accurately determined.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, apparatus, device, and storage medium for three-dimensional reconstruction of traffic sign locations. Background Technology

[0002] In the field of intelligent transportation, traffic signs play a crucial role in improving intersection efficiency and safety, as well as in the application of intelligent transportation systems. However, urban road construction and rerouting often lead to discrepancies between the location data of traffic signs on maps and reality. Therefore, a binocular vision-based method can be used to reconstruct the 3D positions of traffic signs to determine their accurate locations.

[0003] In related technologies, binocular vision-based methods require multiple cameras to reconstruct the positions of traffic signs within a shared field of view, and the relative positions of these cameras need to be calibrated beforehand. However, the accuracy of traffic sign positions determined by binocular vision-based 3D reconstruction needs improvement. Summary of the Invention

[0004] The embodiments described in this specification aim to at least partially solve one of the technical problems in the related art. To this end, the embodiments of this specification propose a method, apparatus, computer equipment, and storage medium for three-dimensional reconstruction of traffic sign locations.

[0005] This specification provides a method for three-dimensional reconstruction of traffic sign locations, the method comprising:

[0006] Determine the position bounding box of the target traffic sign in N consecutive traffic images; wherein the position bounding box in two adjacent traffic images is separated by a first number of pixels; N is a positive integer greater than or equal to 2;

[0007] Based on the first quantity, a target associated border combination is determined from the location borders in the N frames of traffic images; wherein, the target associated border combination includes several target location borders corresponding to the same target traffic sign;

[0008] Based on the position information of the target location border and the shooting pose information of the target traffic image to which the target location border belongs, a three-dimensional reconstruction is performed to obtain the three-dimensional position information of the target traffic sign; wherein, the shooting pose information is the pose information of the traffic object at the time of shooting the target traffic image.

[0009] This specification provides a three-dimensional reconstruction device for the location of traffic signs, the device comprising;

[0010] The position bounding box determination module is used to determine the position bounding box of the target traffic sign in N consecutive frames of traffic images; wherein the position bounding boxes in two adjacent frames of traffic images are separated by a first number of pixels; N is a positive integer greater than or equal to 2;

[0011] The associated border combination determination module is used to determine a target associated border combination among the position borders in the N frames of traffic images based on the first quantity; wherein, the target associated border combination includes a plurality of target position borders corresponding to the same target traffic sign;

[0012] The three-dimensional position information determination module is used to perform three-dimensional reconstruction based on the position information of the target position border and the shooting pose information corresponding to the target traffic image to which the target position border belongs, to obtain the three-dimensional position information of the target traffic sign; wherein, the shooting pose information is the pose information of the traffic object at the shooting time of the target traffic image.

[0013] This specification provides a computer device comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, the instructions being executed by the one or more processors to cause the one or more processors to perform the steps of the method described in any of the above embodiments.

[0014] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0015] This specification provides a computer program product that includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0016] In the above-described embodiment, firstly, the position bounding box of the target traffic sign is determined in N consecutive traffic images, and a first number of pixels between the position bounding boxes in two adjacent traffic images is determined. Since a smaller first number increases the likelihood that the two position bounding boxes corresponding to the first number correspond to the same target traffic object, a target associated bounding box combination is determined from the position bounding boxes in the N traffic images based on the first number. Further, based on the position information of the target position bounding box in the target associated bounding box combination and the shooting pose information corresponding to the target traffic image to which the target position bounding box belongs, three-dimensional reconstruction is performed to obtain the three-dimensional position information of the target traffic sign. By performing three-dimensional reconstruction on multiple two-dimensional positions of the target traffic sign using shooting pose information, the three-dimensional position of the target traffic sign can be accurately determined, improving the accuracy of the three-dimensional reconstruction result of the traffic sign position. Attached Figure Description

[0017] Figure 1a This is a schematic diagram illustrating an application scenario of the three-dimensional reconstruction method for traffic sign locations as described in this specification.

[0018] Figure 1b This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0019] Figure 1c This is a schematic diagram showing the location border of the target traffic sign in the embodiments of this specification;

[0020] Figure 2a This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0021] Figure 2b This is a schematic diagram of the initial associated border combination determined in the embodiments of this specification;

[0022] Figure 3 This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0023] Figure 4 This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0024] Figure 5 This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0025] Figure 6a This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0026] Figure 6b This is a schematic diagram of location clustering in the embodiments of this specification;

[0027] Figure 7 This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0028] Figure 8 This is a flowchart illustrating the three-dimensional reconstruction method for the location of traffic signs in the embodiments of this specification.

[0029] Figure 9 This is a schematic diagram of a three-dimensional reconstruction device for the location of traffic signs provided in the embodiments of this specification. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0031] In related technologies, urban road construction and rerouting often lead to discrepancies between the location data of traffic signs on maps and reality. Therefore, it is necessary to update the location data of traffic signs on maps. In related technologies, 3D reconstruction algorithms for traffic signs typically use single-vehicle driving data, and the reconstruction accuracy of traffic signs is related to the vehicle's driving trajectory. Generally, the closer the traffic sign is to the driving trajectory, the more accurate the reconstruction result, but there is a certain degree of error in the depth direction.

[0032] As mentioned in the background section, some related technologies utilize binocular vision algorithms for 3D reconstruction of traffic signs. This requires vehicles to be pre-installed with multiple cameras, all participating in perception, and the relative positions of these cameras need to be pre-calibrated. Furthermore, the shared field of view of multiple cameras is small, non-shared areas cannot be reconstructed, and feature points are sensitive to noise and lighting conditions. Other related technologies based on LiDAR suffer from poor accuracy in heavy rain, dense fog, or other adverse weather conditions, and require costly data collection vehicles for reconstruction, making large-scale crowdsourced reconstruction infeasible.

[0033] Based on this, the embodiments of this specification provide a three-dimensional reconstruction method for the location of a traffic sign. First, the position bounding box of the target traffic sign is determined in N consecutive frames of traffic images. Second, based on the number of pixels between the position bounding boxes in two adjacent frames of traffic images, a target associated bounding box combination is determined from the position bounding boxes in the N frames of traffic images. Finally, three-dimensional reconstruction is performed based on the position information of the target position bounding box corresponding to the same target traffic sign in the target associated bounding box combination, and the shooting pose information of the target traffic image to which the target position bounding box belongs, to obtain the three-dimensional location information of the target traffic sign. Here, N is a positive integer greater than or equal to 2; the shooting pose information is the pose information of the traffic object at the time the target traffic image was captured.

[0034] Please see Figure 1a Traffic signs can be Figure 1a Traffic lights in China can be used to transport traffic. Figure 1a Vehicles in the vehicle. The three-dimensional reconstruction method for traffic sign locations provided in the embodiments of this specification can be applied to... Figure 1a In vehicle 110, or applied to other devices that have the function of controlling the vehicle (e.g. Figure 1a The cloud server 120, mobile terminal 130, etc. are included. Additionally, traffic signs can be ground-based traffic signs such as electronic eyes, speed limit signs, warning signs, and road name signs, or ground-based lane lines and directional indicators. The vehicle can be an autonomous vehicle, which can be a vehicle with partial or full autonomous driving capabilities. That is, the level of autonomous driving of the vehicle can be classified according to the classification standards of the Society of Automotive Engineers (SAE), namely, no automation (L0), driver assistance (L1), partial automation (L2), conditional automation (L3), high automation (L4), or full automation (L5). The vehicle or other equipment can implement this lane centerline detection method through its components (including hardware and software). It is understood that the vehicle can be any of the following: sedan, truck, motorcycle, bus, recreational vehicle, amusement park vehicle, construction equipment (such as engineering vehicles), tram, golf cart, train, and handcart; this specification does not specifically limit the implementation method in this way.

[0035] This specification provides an example scenario of a 3D reconstruction method for traffic sign locations to exemplify how the embodiments of this specification perform 3D reconstruction of target traffic sign locations. The target traffic sign can be a traffic light at intersection C (the number of traffic lights at intersection C is greater than or equal to 1), and the traffic object can be a vehicle equipped with a fisheye camera. During a certain period, several vehicles pass through intersection C. The fisheye camera can capture images of several traffic lights at intersection C (e.g., denoted as traffic light R, traffic light G, and traffic light Y), obtaining N consecutive frames of traffic images. A target detection model can be deployed on the vehicle to detect the traffic lights in the N frames of traffic images and output the bounding boxes of the traffic lights' positions in each frame. The vehicle can upload the detected bounding boxes, the vehicle's shooting pose information, the intrinsic parameters of the fisheye camera, and the extrinsic parameters of the fisheye camera to the cloud connected to the vehicle's communication network.

[0036] In this scenario example, the cloud can analyze the received location bounding boxes to determine the target associated bounding box combination G1 corresponding to traffic light R, the target associated bounding box combination G2 corresponding to traffic light G, and the target associated bounding box combination G3 corresponding to traffic light Y. Then, the cloud uses the 3D reconstruction method for traffic sign locations mentioned in any of the above embodiments to determine the 3D location information P1 of traffic light R based on the location information of the target location bounding box in target associated bounding box combination G1, the vehicle's shooting pose information, the intrinsic parameters of the fisheye camera, and the extrinsic parameters of the fisheye camera; it determines the 3D location information P2 of traffic light G based on the location information of the target location bounding box in target associated bounding box combination G2, the vehicle's shooting pose information, the intrinsic parameters of the fisheye camera, and the extrinsic parameters of the fisheye camera; and it determines the 3D location information P3 of traffic light Y based on the location information of the target location bounding box in target associated bounding box combination G3, the vehicle's shooting pose information, the intrinsic parameters of the fisheye camera, and the extrinsic parameters of the fisheye camera.

[0037] Thus, for vehicles passing through intersection C, some vehicles can reconstruct the 3D positions of traffic lights R, G, and Y; some can reconstruct the 3D positions of traffic lights R and G; some can reconstruct the 3D positions of traffic lights G and Y; some can reconstruct the 3D positions of traffic lights R and Y; some can reconstruct the 3D positions of traffic lights R; some can reconstruct the 3D positions of traffic lights G; and some can reconstruct the 3D positions of traffic lights Y. Collecting the reconstructed 3D position information of traffic lights from vehicles passing through intersection C yields a set M of 3D position information for the traffic lights.

[0038] In this scenario example, the set M of 3D location information for traffic lights is clustered to obtain location cluster C1 corresponding to traffic light R, location cluster C2 corresponding to traffic light G, and location cluster C3 corresponding to traffic light Y. It can be understood that the 3D location information in location cluster C1 corresponds to traffic light R, the 3D location information in location cluster C2 corresponds to traffic light G, and the 3D location information in location cluster C3 corresponds to traffic light Y.

[0039] In this scenario example, for location cluster C1, the 3D location information within cluster C1 is averaged to obtain the average 3D location of traffic light R. Using this average 3D location as the initial location, bundle adjustment is used to optimize the average 3D location of traffic light R, yielding the 3D location information of traffic light R corresponding to location cluster C1. Furthermore, when reconstructing the location of traffic light R using multiple vehicles, the extrinsic parameters of the fisheye camera change. Therefore, while optimizing the average 3D location of traffic light R using bundle adjustment, the extrinsic parameters of the fisheye camera can also be optimized simultaneously to obtain the optimized extrinsic parameters.

[0040] In this scenario example, for location cluster C2, the 3D location information within C2 is averaged to obtain the average 3D location of traffic light G. Using this average 3D location as the initial location, bundle adjustment is used to optimize the average 3D location of traffic light G, yielding the 3D location information of traffic light G corresponding to location cluster C2. Furthermore, when reconstructing the location of traffic light G using multiple vehicles, the extrinsic parameters of the fisheye camera change. Therefore, while optimizing the average 3D location of traffic light G using bundle adjustment, the extrinsic parameters of the fisheye camera can also be optimized simultaneously to obtain the optimized extrinsic parameters.

[0041] In this scenario example, for location cluster C3, the 3D location information within location cluster C3 is averaged to obtain the average 3D location of traffic light Y. Using the average 3D location of traffic light Y as the initial location, bundle adjustment is used to optimize the average 3D location of traffic light Y, obtaining the 3D location information of traffic light Y corresponding to location cluster C3. Furthermore, when reconstructing the location of traffic light Y using multiple vehicles, the extrinsic parameters of the fisheye camera change. Therefore, during the optimization of the average 3D location of traffic light Y using bundle adjustment, the extrinsic parameters of the fisheye camera can also be optimized simultaneously to obtain optimized extrinsic parameters.

[0042] In the above scenario example, firstly, for a single vehicle, the target associated bounding box combination corresponding to the traffic light is determined in N consecutive traffic images. Based on the positional information of the target bounding box and the shooting pose information of the target traffic image to which the target bounding box belongs, 3D reconstruction is performed to obtain the 3D position information of the traffic light. Secondly, clustering processing is performed using the 3D position information of traffic lights reconstructed from multiple vehicles to group the 3D position information corresponding to the same traffic light together. This allows for optimization of the traffic light position using the reconstruction results from multiple vehicles, and also optimizes the extrinsic parameters of the fisheye camera.

[0043] This specification provides a method for three-dimensional reconstruction of traffic sign locations. Please refer to [link / reference]. Figure 1b The three-dimensional reconstruction method for the location of the traffic sign may include the following steps:

[0044] S110. Determine the position border of the target traffic sign in N consecutive frames of traffic images.

[0045] In this traffic image, the positional borders of two adjacent frames have a first number of pixels; N is a positive integer greater than or equal to 2. When a vehicle is traveling on a road, it can capture images of surrounding traffic signs using an image acquisition device on the vehicle, obtaining traffic images. The traffic signs in the traffic images include at least one type. Due to urban road construction and rerouting, the positions of at least some traffic signs may change. Therefore, after urban road rerouting, it is necessary to identify the target traffic sign among the traffic signs and reconstruct its new position. For example, since traffic light data is crucial for vehicle driving, the target traffic sign can be a traffic light. For example, electronic eyes are also important for vehicle driving, so the target traffic sign can be an electronic eye. It should be noted that the image acquisition device can be a fisheye camera.

[0046] In some implementations, an object detection model can be deployed on the vehicle. N consecutive frames of traffic images are input into the model, which then detects traffic signs within the images to obtain the location information of those signs. This location information can be represented as a bounding box containing a certain number of pixels. (See also...) Figure 1c The location border can be a frame that outlines the target traffic sign in the traffic image. Figure 1c The bounding box (of a traffic light) in the image. For example, the object detection model can output the location bounding box of the target traffic sign. The location bounding box can be used to mark the area where the target traffic sign is located in the traffic image, such as using a rectangle to outline the target traffic sign in the traffic image. The location bounding box can be the bounding box determined by the object detection model when it detects the target traffic sign. The object detection model can use a deep learning network. The object detection model can also use the SSD (Single Shot MultiBox Detector) model.

[0047] In some implementations, the object detection model can be deployed on a cloud server connected to the vehicle communication network. The vehicle uploads the acquired traffic images to the cloud server, which then determines N consecutive traffic images from the received images. These N consecutive traffic images are then output to the object detection model to obtain the bounding box of the target traffic sign within the N consecutive traffic images. Deploying the object detection model in the cloud reduces the consumption of vehicle-side resources.

[0048] In this embodiment, during the movement of a traffic object, images of surrounding target traffic signs can be acquired to obtain a series of true traffic images. Since the time interval between two adjacent traffic image frames is very short (e.g., 0.1s or 0.2s), this embodiment directly compares the positional bounding boxes in adjacent traffic object frames to determine the pixel distance between the positional bounding boxes in two adjacent traffic image frames, i.e., the first number of pixels. Therefore, the first number can be used to represent the pixel distance between the positional bounding boxes in two adjacent traffic image frames.

[0049] Specifically, a convolutional network model is trained based on a deep neural network. Traffic images captured by the image acquisition module are input into the trained convolutional network model. One forward pass is required to detect the bounding box of the target traffic sign within the traffic image and its coordinates. Furthermore, each frame of the traffic image contains several pixels, and the coordinates of the detected bounding box can be considered as the pixel coordinates of a vertex of the bounding box. For example, P and Q represent the pixels corresponding to a vertex of the bounding box in two adjacent frames. (x p ,y P (x) represents the coordinates of pixel point P. q ,y q Let Q be the coordinates of the pixel. Further, according to the Euclidean distance... Calculate the pixel distance between two adjacent frames and denote it as the first quantity.

[0050] S120. Based on the first quantity, determine the target associated bounding box combination in the position bounding box in the N frames of traffic images.

[0051] The target-related bounding box combination includes several target position bounding boxes corresponding to the same target traffic sign. Specifically, in N frames of traffic images, for any frame of traffic image P, which may contain M target traffic signs, each frame of traffic image may have M (greater than or equal to 1) position bounding boxes. However, for any target traffic sign T among the M target traffic signs, due to objective factors during the shooting process (occlusion caused by other traffic objects, shooting angle of traffic objects, etc.), this target traffic sign T will not appear in the N frames of images. For example, this target traffic sign T may appear in the first frame of traffic image and the third frame of traffic image. That is, when shooting the second frame of traffic image, this target traffic sign T is occluded, and when shooting the second frame of traffic image, this target traffic sign T is exposed again. For example, any target traffic sign T appears in the fourth and fifth frames of the traffic image. That is, when the first, second, and third frames of the traffic image were captured, the target traffic sign T was obscured until the fourth frame was captured, at which point the target traffic sign T was exposed again. This target traffic sign T is a newly seen target traffic sign. It can be seen that some traffic images in the N frames correspond to the same target traffic sign. Therefore, it is necessary to find the target position borders that may correspond to the same target traffic sign in the position borders of the N frames of the traffic image. The target position borders corresponding to the same target traffic sign can form a target association border combination.

[0052] In some cases, since the closer the pixel distance between location bounding boxes, the greater the probability that they correspond to the same target traffic sign, and since the first quantity is used to represent the pixel distance between location bounding boxes in two adjacent traffic image frames, target associated bounding box combinations are determined based on the first quantity of pixels between location bounding boxes in N frames of traffic images. Specifically, in some implementations, the size of the first quantity of pixels between location bounding boxes is compared, and it is determined that the two location bounding boxes corresponding to the smaller first quantity can form a target associated bounding box combination. For example, the first quantities are sorted from largest to smallest, and the last few first quantities are determined. The two location bounding boxes corresponding to the last few first quantities are then used to determine several target associated bounding box combinations.

[0053] S130. Based on the position information of the target location border and the shooting pose information of the target traffic image to which the target location border belongs, perform three-dimensional reconstruction to obtain the three-dimensional position information of the target traffic sign.

[0054] The shooting pose information refers to the pose of the traffic object at the moment the target traffic image is captured. Pose information can include the traffic object's position and orientation angle at the time the image is captured. 3D reconstruction can recover the 3D position of a traffic sign from several 2D images of the traffic sign. Furthermore, 3D reconstruction can also render the traffic sign, ultimately representing it in a virtual reality on a computer device.

[0055] Specifically, as mentioned above, a target-associated bounding box combination has been determined from the location bounding boxes in the N frames of traffic images based on a first quantity. The target-associated bounding box combination includes at least two target location bounding boxes. The positions of the target location bounding boxes in the target traffic image are known, and the shooting pose information when the target traffic image was captured is also known. Therefore, 3D reconstruction can be performed based on the multi-frame triangulation principle to obtain the 3D location information of the target traffic sign.

[0056] Understandably, multi-frame triangulation involves taking multiple frames of 2D image observations [u,v], the camera intrinsic parameter matrix K, and the camera pose [R] for each observed image frame. ciw ,t ciw The process of determining the position p of a 3D point in the world coordinate system (from the world coordinate system w to the camera coordinate system ci, where the camera coordinate system is at the lower right front, i.e., the line connecting the optical center and the image center is the Z-axis, the width direction of the image is the X-axis, and the height direction of the image is the Y-axis) is used, where n ≥ 2 frames. Specifically, in this embodiment, based on the first quantity, at least two target position bounding boxes corresponding to the same target traffic sign can be determined in the adjacent N frames of traffic images. The position information of the i-th target position bounding box is denoted as [ui,vi]; the shooting pose information corresponding to the target traffic image to which the i-th target position bounding box belongs is denoted as P; then, based on the position information [ui,vi] of the i-th target position bounding box and the corresponding shooting pose information P, combined with the multi-frame triangulation principle, the three-dimensional position information of the target traffic sign is obtained.

[0057] It should be noted that the above-mentioned 3D reconstruction method for traffic sign locations can run on the traffic object itself or on a cloud server connected to the traffic object. Running the method on a cloud server can reduce the consumption of vehicle-side resources.

[0058] In the aforementioned 3D reconstruction method for traffic sign locations, firstly, the bounding box of the target traffic sign is determined in N consecutive traffic images, along with a first number of pixels between the bounding boxes in two adjacent traffic images. Since a smaller first number increases the likelihood that the two bounding boxes corresponding to the first number represent the same target traffic object, a target associated bounding box combination is determined from the bounding boxes in the N traffic images based on the first number. Further, 3D reconstruction is performed based on the position information of the target bounding box in the target associated bounding box combination and the shooting pose information of the target traffic image to which the target bounding box belongs, to obtain the 3D position information of the target traffic sign. By performing 3D reconstruction on multiple 2D positions of the target traffic sign using shooting pose information, the 3D position of the target traffic sign can be accurately determined, improving the accuracy of the 3D reconstruction results for the traffic sign location.

[0059] In some implementations, please refer to Figure 2a Determining a target-associated bounding box combination within the location bounding boxes of N frames of traffic images, based on a first quantity, may include the following steps:

[0060] S210. Determine the two positional borders corresponding to the number of targets less than the pixel number threshold in the first quantity as the initial associated border combination.

[0061] The initial associated border combination has the confidence that the two location borders included in the initial associated border combination correspond to the same target traffic sign.

[0062] The pixel count threshold is used to determine the initial associated bounding box combination. The pixel count threshold can be set based on the distance traveled by the traffic object in adjacent frames. It can also be set based on the area of ​​the location bounding boxes in adjacent traffic images. Furthermore, it can be set based on the number of pixels within the location bounding boxes in the traffic image. The confidence level represents the probability that the location bounding boxes in two adjacent traffic images represent the same target traffic sign. The confidence level can be set based on the pixel distance between the location bounding boxes in two adjacent traffic images. Specifically, there is a first number of pixels between the location bounding boxes in adjacent frames, and the pixel count threshold is set based on the actual situation. The first number is compared with the pixel count threshold. If the first number is less than the preset threshold, it indicates that the location bounding boxes in the adjacent traffic images are likely to be the location bounding boxes corresponding to the same traffic sign in reality, and can form an initial associated bounding box combination with the location bounding boxes corresponding to the same traffic sign.

[0063] In practice, two adjacent traffic image frames are denoted as the previous frame and the current frame, respectively. A positional border in the previous frame corresponds to at most one positional border in the current frame. Similarly, a positional border in the current frame corresponds to at most one positional border in the previous frame. However, the initial associated border combination contains positional borders in the current frame that correspond to more than two positional borders in the previous frame, and the determined initial associated border combination also contains positional borders in the previous frame that correspond to multiple positional borders in the current frame. This clearly does not match reality. Therefore, a confidence level is set for the initial associated border combination, representing the probability that two positional borders included in the initial associated border combination correspond to the same target traffic sign. Furthermore, the confidence level of the initial associated border combination can be used to filter the initial associated border combinations to determine the target associated border combination.

[0064] S220. In the initial associated border combination, the target associated border combination is determined based on the confidence level of the initial associated border combination.

[0065] For example, please refer to Figure 2b , Figure 2b The solid and dashed lines represent the first number of pixels between the two positions at the ends of the line that is less than the pixel number threshold. Figure 2b In this diagram, A, B, C, and D represent the bounding boxes of the four target traffic signs detected in the previous frame of the traffic image, while E, F, G, H, and I represent the bounding boxes of the five traffic signs detected in the current frame. Bounding boxes connected by lines between two frames can be considered as initial associated bounding box combinations. For example, bounding boxes A and E form an initial associated bounding box combination; bounding boxes A and F form an initial associated bounding box combination; bounding boxes B and F form an initial associated bounding box combination; bounding boxes B and H form an initial associated bounding box combination; bounding boxes C and E form an initial associated bounding box combination; bounding boxes D and G form an initial associated bounding box combination; and bounding boxes D and I form an initial associated bounding box combination. The confidence score is used to determine the target associated bounding box combination from the initial associated bounding box combinations. Please continue reading... Figure 2b The two borders connected by the solid lines in the figure can form a target-related border combination.

[0066] Specifically, the initial associated border combinations may exhibit one-to-many or many-to-one relationships, but in reality, they are typically one-to-one, one-to-zero, or zero-to-one. Therefore, based on the confidence level of the initial associated border combinations, these combinations are filtered to obtain the target associated border combinations. For example, please refer to [further details omitted]. Figure 2bIn the previous frame of the traffic image, position bounding box A corresponds to position bounding boxes E and F in the current frame of the traffic image. The confidence scores between position bounding boxes A and E are compared with those between position bounding boxes A and F. If the confidence score between position bounding boxes A and E is greater than that between position bounding boxes A and F, the initial associated bounding box combination formed by position bounding boxes A and F is deleted, while the initial associated bounding box combination formed by position bounding boxes A and E is retained. Further, the confidence scores between position bounding boxes A and E are compared with those between position bounding boxes E and C. If so, the initial associated bounding box combination formed by position bounding boxes E and C is deleted, while the initial associated bounding box combination formed by position bounding boxes A and E is retained. This process is repeated for each other: the initial associated bounding box combination formed by position bounding boxes B and H is deleted; the initial associated bounding box combination formed by position bounding boxes D and I is deleted; the initial associated bounding box combination formed by position bounding boxes B and F is retained; and the initial associated bounding box combination formed by position bounding boxes D and G is retained. It can be understood that the target traffic sign in position bounding box C is occluded in the current frame, while the target traffic sign in position bounding boxes H and I is a new target traffic sign captured in the current frame.

[0067] In the above-mentioned 3D reconstruction method for traffic sign locations, an initial associated bounding box combination is first constructed in the bounding boxes of two consecutive traffic images. Then, the initial associated bounding box combination is filtered based on the confidence level of the initial associated bounding box combination to obtain the target associated bounding box combination. The method aims to determine the target location bounding box corresponding to the same target traffic sign in the detected bounding boxes as much as possible, thereby accurately reconstructing the 3D location of the target traffic sign and accurately displaying the location of the target traffic sign on the map.

[0068] In some implementations, the confidence level is determined by the number of pixels between the two location borders included in the initial associated border combination.

[0069] Specifically, in the initial associated bounding box combination, each bounding box has a corresponding pixel coordinate, based on the Euclidean distance. The pixel distance between the two positional bounding boxes is calculated, and the reciprocal of this pixel distance can be used as the confidence level of the initial associated bounding box combination. Furthermore, in some implementations, an adjustment coefficient can be set, and the confidence level of the initial associated bounding box combination can be determined based on the adjustment coefficient and the reciprocal of the pixel distance.

[0070] In the above-mentioned three-dimensional reconstruction method for traffic sign locations, the confidence level of the initial associated border combination is set by the number of pixels between the two position borders included in the initial associated border combination. This can quantitatively represent the probability that the two position borders in the initial associated border combination correspond to the same target traffic sign, providing a data basis for accurately reconstructing the three-dimensional location of the target traffic sign.

[0071] In some implementations, the initial associated border combination includes two position borders, each containing a second number of pixels and a third number of pixels; the pixel count threshold is determined by the number of pixels within a specified position border; wherein, the specified position border is designated among the two position borders included in the initial associated border combination based on a comparison of the second and third counts.

[0072] Specifically, each location bounding box contains a number of pixels. The two location bounding boxes in the initial associated bounding box combination each have a corresponding number of pixels. The number of pixels contained in the location bounding box of the previous frame traffic image is denoted as the second number, and the number of pixels contained in the location bounding box of the current frame traffic image is denoted as the third number. The second number and the third number are compared, and a pixel number threshold is determined based on the smaller value. For example, if the second number is less than the third number, the location bounding box of the previous frame traffic image is determined as the designated location bounding box. If the third number is less than the second number, the location bounding box of the current frame traffic image is determined as the designated location bounding box.

[0073] In the above-mentioned three-dimensional reconstruction method for traffic sign locations, a pixel count threshold is set by the number of pixels within the two location bounding boxes in two adjacent traffic image frames. Furthermore, the pixel count threshold can be used to determine whether the two location bounding boxes may correspond to the same target traffic location, so as to accurately find the location bounding boxes that may correspond to the same target traffic sign in the location bounding boxes of two adjacent traffic image frames.

[0074] In some implementations, please refer to Figure 3 The two positional bounding boxes included in the initial associated bounding box combination are denoted as the first positional bounding box and the second positional bounding box with an initial connection relationship, respectively. The first positional bounding box belongs to the first traffic image, and the second positional bounding box belongs to the second traffic image. Determining the target associated bounding box combination based on the confidence level of the initial associated bounding box combination can include the following steps:

[0075] S310. If any location border in the first traffic image has an initial connection relationship with multiple location borders in the second traffic image, among the multiple location borders in the second traffic image, determine the location border with the target connection relationship with any location border based on the confidence level of the initial associated border combination.

[0076] S320. Determine any position border and any position border that has a target connection relationship with any position border as the target associated border combination.

[0077] The first traffic image can be the preceding frame in a two-frame traffic image sequence, and the second traffic image is the current frame in a two-frame traffic image sequence. The positional borders in the first and second traffic images can have one-to-one, one-to-many, or one-to-zero relationships. Please continue reading. Figure 2b Position borders A, B, C, and D can be considered to belong to the first traffic image, while position borders E, F, G, H, and I can be considered to belong to the second traffic image. Position borders A, B, and D are one-to-many cases, while position border C is one-to-one cases.

[0078] Specifically, in two adjacent traffic image frames, each frame has n (n ≥ 1) location bounding boxes. A location bounding box from the first traffic image is taken and denoted as location bounding box X. The pixel distance between this location bounding box and any location bounding box in the second traffic image (denoted as location bounding box Y) is determined. If the number of pixels corresponding to the pixel distance between location bounding box X and location bounding box Y is less than a pixel number threshold, it can be determined that location bounding box X and location bounding box Y in the second traffic image have an initial connection relationship. This process is repeated until all location bounding boxes in the first traffic image have been visited. An initial associated bounding box combination is constructed based on the two location bounding boxes with an initial connection relationship. Each initial associated bounding box combination has a confidence level, and any location bounding box in the first traffic image corresponds to at most one location bounding box in the second traffic image. Therefore, when any location bounding box in the first traffic image has an initial connection relationship with multiple location bounding boxes in the second traffic image, based on the confidence level of the initial associated bounding box combination, a target connection relationship is determined from the multiple initial connection relationships corresponding to any location bounding box in the first traffic image, and a location bounding box with a target connection relationship with any location bounding box is obtained. Furthermore, any position border and any position border that has a target connection relationship with any position border are defined as the target associated border combination.

[0079] For example, please continue reading Figure 2b There is an initial connection between position bounding box A in the previous frame of the traffic image and position bounding box E in the current frame of the traffic image. The confidence scores between position bounding boxes A and E are compared with those between position bounding boxes A and F. If the confidence score between position bounding boxes A and E is greater than that between position bounding boxes A and F, then a target connection relationship exists between position bounding boxes A and E, and position bounding boxes A and E are identified as the target position bounding boxes in the target associated bounding box combination.

[0080] In the above-mentioned 3D reconstruction method for traffic sign locations, an initial connection relationship is first constructed in the location bounding boxes of two consecutive traffic images. Then, the initial connection relationship is filtered based on the confidence level corresponding to the initial connection relationship to obtain the target connection relationship. The target location bounding box corresponding to the same target traffic sign is determined as much as possible in the detected location bounding boxes, so that the 3D location of the target traffic sign can be accurately reconstructed and the location of the target traffic sign can be accurately displayed on the map.

[0081] In some implementations, among multiple location bounding boxes in the second traffic image, determining the location bounding box with a target connection relationship to any location bounding box based on the confidence level of the initial associated bounding box combination includes: taking the location bounding box in the second traffic image corresponding to the maximum confidence level as the location bounding box with a target connection relationship to any location bounding box.

[0082] The maximum confidence score is used to determine the confidence scores of the two location bounding boxes with the highest correlation between the first and second traffic images. Specifically, any location bounding box in the first traffic image has an initial connection relationship with multiple location bounding boxes in the second traffic image; that is, any location bounding box in the first traffic image and multiple location bounding boxes in the second traffic image form multiple initial associated bounding box combinations. Each initial associated bounding box combination has a confidence score, and the confidence scores of each initial associated bounding box combination are compared to determine the highest confidence score. The highest confidence score corresponds to two location bounding boxes, and the location bounding box in the second traffic image corresponding to the highest confidence score can be identified as a location bounding box with a target connection relationship with any location bounding box in the first traffic image. Further, any location bounding box and the location bounding box with a target connection relationship with any location bounding box are determined as a target associated bounding box combination.

[0083] For example, position border B1 in the first traffic image has an initial connection relationship with position borders B2, B3, and B4 in the second traffic image. The confidence levels between position borders B1 and B2, B1 and B3, and B1 and B4 are compared. If the confidence level between position borders B1 and B4 is greater than the confidence levels between position borders B1 and B2 and B1 and B3, then it is determined that a target connection relationship exists between position borders B1 and B4, and position borders B1 and B4 are identified as target position borders in the target associated border combination.

[0084] In the above-mentioned three-dimensional reconstruction method for traffic sign locations, if any location bounding box in the first traffic image has an initial connection relationship with multiple location bounding boxes in the second traffic image, the target connection relationship is determined among multiple initial connection relationships by using the maximum confidence, so as to determine the location bounding box corresponding to the same target traffic sign as accurately as possible, which is beneficial for reconstructing the three-dimensional location of the target traffic sign.

[0085] In some implementations, please refer to Figure 4 The target traffic image is obtained by capturing images of the target traffic sign using an image acquisition device installed on the traffic object. Based on the position information of the target location border and the shooting pose information of the target traffic image to which the target location border belongs, three-dimensional reconstruction is performed to obtain the three-dimensional position information of the target traffic sign. This may include the following steps:

[0086] S410. Based on the position information of the target location bounding box and the shooting pose information, triangulation calculation is performed to obtain the estimated three-dimensional position of the target traffic sign.

[0087] In this process, triangulation can be performed when an image acquisition device (such as a camera) installed on a traffic object is observed. Based on the shooting pose and observation vector, an observation "ray" originating from the center of the camera can be obtained in 3D space. Multiple shooting pose observations will generate multiple observation rays. Ideally, these observation rays intersect at a point in space, and this intersection point is the estimated 3D position of the target traffic sign.

[0088] Specifically, the captured traffic images are processed by a convolutional network model for object detection to obtain the two-dimensional position information of the bounding boxes. i v i ], based on the detected position bounding box [u i v i [Time of capture pose information P] i The matrix D can be constructed as follows, and the solution DY=0 can be calculated by using the SVD algorithm. This value is the estimated three-dimensional position of the target traffic sign, where Y is the homogeneous coordinate of the estimated three-dimensional position of the target traffic sign.

[0089]

[0090] S420. Based on the criterion of minimizing reprojection error, the estimated three-dimensional position is optimized according to the intrinsic and extrinsic parameters of the image acquisition device, the estimated three-dimensional position, and the shooting pose information to obtain the three-dimensional position information of the target traffic sign.

[0091] The image acquisition device can be a camera. The intrinsic parameters are the focal length and the position of the camera's center point in the image, while the extrinsic parameters are the camera's position and orientation in the traffic object coordinate system. For example, if the traffic object is a vehicle, then the extrinsic parameters are the camera's position and orientation in the vehicle coordinate system.

[0092] In some cases, due to factors such as viewing angle, perceptual bias, and image noise, the initial position of the traffic light obtained through triangulation is inaccurate. Therefore, the bundle adjustment method is used to optimize the estimated 3D position of the target traffic sign, and this estimated 3D position is used as the initial position. Specifically, a secondary projection is performed based on the estimated 3D position of the target traffic sign obtained through triangulation and the shooting pose information to obtain the projected position of the estimated 3D position, which can be denoted as: K(Ex*P) -1 *p), where p represents the estimated three-dimensional position of the target traffic sign; P represents the shooting pose information; K represents the camera intrinsic parameters; and Ex represents the camera extrinsic parameters.

[0093] Due to factors such as viewing angle, perceptual bias, and image noise, there is an error between the projected position and the two-dimensional position information of the bounding box detected in the image captured by the pose information P. A cost function is constructed based on the projected position of the secondary projection and the two-dimensional position information of the bounding box:

[0094]

[0095] Minimizing this cost function makes the detected position bounding boxes in the secondary projection and captured images almost coincide, making the estimated 3D position of the target traffic sign more consistent with the actual location of the target traffic sign, thus improving the accuracy of the 3D position reconstruction of the target traffic sign.

[0096] The three-dimensional position optimization formula is:

[0097]

[0098] Where K represents the camera intrinsic parameter, Ex represents the camera extrinsic parameter, n represents the location bounding box detected in n traffic images of the target traffic sign, Ui represents the position of the i-th location bounding box, p on the left side of the equation represents the optimized three-dimensional position coordinates of the target traffic sign, p on the right side of the equation represents the estimated three-dimensional position of the target traffic sign obtained by triangulation calculation, and P represents the shooting pose information.

[0099] Next, the Levenberg-Marquardt algorithm is used to find the 3D position coordinates p that minimize the reprojection error, which are the optimized 3D position coordinates of the target traffic sign. The Levenberg-Marquardt algorithm is an optimization algorithm, an iterative algorithm that can be used to solve least squares problems.

[0100] In the above-mentioned three-dimensional reconstruction method for traffic sign locations, firstly, the estimated three-dimensional coordinates of the target traffic sign are obtained through the triangulation principle. Then, considering that the estimated three-dimensional coordinates of the target traffic sign may have certain errors due to factors such as viewing angle, perception bias, and image noise, the estimated three-dimensional coordinates are optimized by reprojection error, thereby improving the accuracy of the three-dimensional reconstruction results of the target traffic sign location.

[0101] In some implementations, please refer to Figure 5 The three-dimensional reconstruction method for traffic sign locations may include the following steps:

[0102] S510. Based on the three-dimensional location information of the target traffic signs obtained from the three-dimensional reconstruction of several traffic objects, clustering is performed to obtain several location clusters.

[0103] S520. Determine the three-dimensional location information of the target traffic sign corresponding to any location cluster based on the three-dimensional location information in any location cluster.

[0104] In this context, the 3D location information within a location cluster corresponds to the same target traffic sign. A location cluster is a set of 3D location information for target traffic signs that meet the clustering algorithm criteria.

[0105] In some cases, several traffic objects may exist around a target traffic sign, and each traffic object can capture an image of the target traffic sign. For each traffic object, the 3D reconstruction method for traffic sign location mentioned in any of the above embodiments can be used to determine the 3D position information of a single target traffic sign. Therefore, the 3D position information of the target traffic sign obtained from the 3D reconstruction of several traffic objects constitutes a 3D position information set. The 3D position information set includes the 3D position information of several target traffic signs. Furthermore, this 3D position information set can be clustered to group the 3D position information corresponding to the same target traffic sign together, and the 3D position information determined for a single traffic object can be optimized using the 3D position information corresponding to the same target traffic sign.

[0106] In some implementations, since the 3D position information of the target traffic sign obtained from the 3D reconstruction of a single traffic object is known, and each traffic object can determine the 3D position information of at least one target traffic sign, each traffic object can upload the 3D position information of at least one target traffic sign. To identify which 3D position information corresponds to the same target traffic sign, clustering processing can be performed based on the 3D position information of the target traffic signs obtained from the 3D reconstruction of several traffic objects to obtain several position clusters. The 3D position information in each position cluster corresponds to the same target traffic sign. Furthermore, for any position cluster, the position of the target traffic sign corresponding to that position cluster can be optimized using the 3D position information in that position cluster to obtain the 3D position information of the target traffic sign corresponding to that position cluster.

[0107] In the above-mentioned three-dimensional reconstruction method for traffic sign locations, the three-dimensional location information of the target traffic sign obtained by three-dimensional reconstruction of several traffic objects is clustered to obtain location clusters. The three-dimensional location information in the location clusters can be used to further optimize the location of the target traffic sign, thereby improving the accuracy of the three-dimensional reconstruction results of the target sign location.

[0108] In other implementations, since the 3D position information of the target traffic sign obtained from the 3D reconstruction of a single traffic object is known, each traffic object can upload its 3D position information for the same target traffic sign to the cloud. The cloud receives multiple 3D position information for the same target traffic sign, and can then optimize the position of the same target traffic sign based on these multiple 3D position information. For example, the average 3D position of the same target traffic sign can be calculated by averaging the multiple 3D position information. Further, bundle adjustment is used to optimize the average 3D position based on the intrinsic and extrinsic parameters of the image acquisition device, the average 3D position, and the shooting pose information, resulting in the optimized 3D position information of the same target traffic sign.

[0109] In some implementations, please refer to Figure 6a Based on the 3D location information of target traffic signs obtained from the 3D reconstruction of several traffic objects, clustering processing is performed to obtain several location clusters, including the following steps:

[0110] S610. Determine any one of the three-dimensional position information of the target traffic sign obtained from the three-dimensional reconstruction of several traffic objects as a data object point.

[0111] S620. Using data object points as the core points, perform clustering processing based on the preset neighborhood radius and the preset number threshold of target traffic signs within the preset neighborhood radius to obtain several location clusters.

[0112] In this system, a data object point can be any three-dimensional location information of a traffic sign. A core point is a point whose number of sample points within a preset neighborhood radius is greater than or equal to a preset threshold. The preset threshold can be the number of at least three-dimensional locations of traffic signs contained within the preset neighborhood radius. A location cluster is a set of data object points whose core point density is reachable. For example, the DBSCAN clustering algorithm can be used for clustering. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based spatial clustering algorithm. The DBSCAN algorithm has two important parameters: Eps and MmPtS. Eps is the neighborhood radius when defining the density, and MmPtS is the threshold when defining the core point.

[0113] Specifically, from the 3D reconstruction results of several traffic objects, the 3D location information of a target traffic sign is arbitrarily selected as a data object point. If there are more than or equal to a preset threshold number of data object points within a preset neighborhood radius, the selected data object point is designated as a core point. All data object points reachable from the core point are then identified, forming a cluster. If the selected data object point is an edge point, another data object point is selected (where edge points are data object points whose number of 3D locations of traffic signs within a preset neighborhood radius is less than a preset threshold). This process is repeated until all points are processed, resulting in several location clusters. Please refer to [link to relevant documentation]. Figure 6b , Figure 6b The result of the clustering process is 6 location clusters. Figure 6b The 3D location information in each location cluster displayed corresponds to the same target traffic sign. For example, a traffic sign can be a traffic light, with a preset neighborhood radius of 1.5 meters and a preset threshold of 20 target traffic signs within that radius. The reconstruction results of any traffic light are selected as the core point. If there are at least 20 single-vehicle traffic lights within a 1.5-meter radius of that traffic light, then all these reconstructed traffic lights can be considered to be the same traffic light in the real world. The process is repeated recursively, examining the neighborhood of each adjacent reconstructed traffic light. Then, other unincluded single-vehicle reconstruction results are arbitrarily selected, and the above process is repeated until all single-vehicle reconstruction results have been visited.

[0114] In the above-mentioned three-dimensional reconstruction method for traffic sign locations, by selecting core points and combining preset neighborhood radii and preset quantity thresholds, the three-dimensional location information of the target traffic sign obtained from the three-dimensional reconstruction of several traffic objects is clustered to obtain location clusters, which conform to the distribution characteristics of traffic objects and have high accuracy of clustering results, thereby improving the accuracy of the three-dimensional reconstruction results of the target sign location.

[0115] In some implementations, please refer to Figure 7 Determining the three-dimensional location information of a target traffic sign corresponding to any location cluster based on the three-dimensional location information in any location cluster may include the following steps:

[0116] S710. Average the three-dimensional position information in any location cluster to obtain the average three-dimensional position of the target traffic sign.

[0117] S720. Using the average three-dimensional position of the target traffic sign as the initial position, the average three-dimensional position is optimized using the bundle adjustment method to obtain the three-dimensional position information of the target traffic sign corresponding to any position cluster.

[0118] Bundle adjustment combines several images of the same scene taken from different perspectives. Using the projections of all points in the images as a standard, it extracts the 3D point coordinates, relative motion parameters, and camera optical parameters that describe the scene structure. Bundle adjustment can be used to optimize the 3D structure and viewpoint parameters (i.e., camera position, orientation, inherent calibration, and radial distortion) to obtain the optimal 3D position and extrinsic parameters of the image acquisition device. By providing initial estimates, bundle adjustment refines the 3D position information of the target traffic sign and the extrinsic parameters of the image acquisition device by minimizing the projection error between observed and predicted image points.

[0119] Specifically, in order to reduce the reconstruction error caused by different movement trajectories of traffic objects, after obtaining the reconstruction results of the same target traffic sign in the real world through clustering, the three-dimensional position information in any location cluster is averaged to obtain the average three-dimensional position of the target traffic sign. The average three-dimensional position of the target traffic sign is used as the initial position, and the bundle adjustment method is used to optimize the initial position to obtain the final three-dimensional position information.

[0120] In this embodiment, the average 3D position can be optimized based on the intrinsic and extrinsic parameters of the image acquisition device, the average 3D position, and the shooting pose information, with the minimum reprojection error as the criterion. This yields the 3D position information of the target traffic sign corresponding to any position cluster. Specifically, the Levenberg-Marquardt algorithm can also be used to optimize the average 3D position. As mentioned earlier, the obtained average 3D position and shooting pose are subjected to a secondary projection. There is an error between this projection and the position bounding box. Therefore, a cost function is constructed based on the information of the position bounding box detected in the image captured by the secondary projection and the shooting pose. Minimizing this cost function makes the position bounding box detected in the secondary projection and the captured image almost coincide, making the 3D position information more consistent with the actual location of the traffic sign. This will not be elaborated further here.

[0121] In the above-mentioned three-dimensional reconstruction method for traffic sign positions, the average three-dimensional position of the traffic sign is calculated and optimized using the average three-dimensional position of the bundle adjustment method. This method can optimize the three-dimensional reconstruction results of the target traffic sign based on multiple traffic objects, thereby reducing reconstruction errors caused by the movement trajectories of traffic objects.

[0122] In some implementations, the extrinsic parameters of the image acquisition device are optimized using the bundle adjustment method to obtain optimized extrinsic parameters.

[0123] The image acquisition device can be a device installed on a traffic object to acquire images of traffic signs; for example, the image acquisition device could be a fisheye camera. The extrinsic parameters of the image acquisition device can be its position and orientation in the vehicle coordinate system. These values ​​are usually considered fixed, but due to the aging of the traffic object or loosening of the image acquisition device due to sudden braking, its position and orientation in the traffic object's coordinate system may change. Therefore, assuming that the extrinsic parameter values ​​remain constant is inaccurate, and the extrinsic parameters of the image acquisition device need to be optimized. Specifically, the bundle adjustment method is used to optimize the extrinsic parameters of the image acquisition device to obtain optimized extrinsic parameters. As mentioned earlier, the Levenberg-Marquardt algorithm can be used to optimize the extrinsic parameters.

[0124] In the above-mentioned three-dimensional reconstruction method for traffic sign locations, considering the impact of changes in the extrinsic parameters of the image acquisition device on the reconstruction results, the extrinsic parameters of the image acquisition device are also optimized during the optimization process of the target traffic sign location using the bundle adjustment method. This can reduce the influence of external forces on the extrinsic parameters and further improve the accuracy of the reconstructed location of the target traffic sign.

[0125] This specification also provides a method for three-dimensional reconstruction of traffic sign locations. Please refer to [link / reference]. Figure 8 The three-dimensional reconstruction method for the location of the traffic sign may include the following steps:

[0126] S802. Determine the position border of the target traffic sign in N consecutive frames of traffic images.

[0127] In this context, the positional bounding boxes in two adjacent traffic image frames have a first number of pixels between them; N is a positive integer greater than or equal to 2.

[0128] S804. Determine the two positional bounding boxes corresponding to the number of targets less than the pixel number threshold in the first quantity as the initial associated bounding box combination.

[0129] The initial associated bounding box combination comprises two positional bounding boxes, each with a second number of pixels and a third number of pixels, respectively. The pixel count threshold is determined by the number of pixels within a specified positional bounding box. The specified positional bounding box is determined from the two positional bounding boxes included in the initial associated bounding box combination based on a comparison of the second and third counts.

[0130] S806. In the initial associated border combination, the target associated border combination is determined based on the confidence level of the initial associated border combination.

[0131] The target associated bounding box combination includes several target location bounding boxes corresponding to the same target traffic sign; the confidence level is determined by the number of pixels between the two location bounding boxes included in the initial associated bounding box combination. The two location bounding boxes included in the initial associated bounding box combination are denoted as the first location bounding box and the second location bounding box with an initial connection relationship, respectively. The first location bounding box belongs to the first traffic image, and the second location bounding box belongs to the second traffic image.

[0132] Specifically, if any location border in the first traffic image has an initial connection relationship with multiple location borders in the second traffic image, among the multiple location borders in the second traffic image, the location border in the second traffic image corresponding to the maximum confidence level is taken as the location border with a target connection relationship with any location border; any location border and the location border with a target connection relationship with any location border are determined as the target associated border combination.

[0133] S808. Based on the position information of the target location border and the shooting pose information of the target traffic image to which the target location border belongs, perform three-dimensional reconstruction to obtain the three-dimensional position information of the target traffic sign.

[0134] The shooting pose information refers to the pose of the traffic object at the moment the target traffic image is captured. The target traffic image is obtained by capturing images of the target traffic sign using an image acquisition device installed on the traffic object.

[0135] Specifically, triangulation calculations are performed based on the location information of the target location bounding box and the shooting pose information to obtain the estimated three-dimensional position of the target traffic sign. Using the minimum reprojection error as the criterion, the estimated three-dimensional position is optimized based on the intrinsic and extrinsic parameters of the image acquisition device, the estimated three-dimensional position, and the shooting pose information to obtain the three-dimensional position information of the target traffic sign.

[0136] S810. Determine any one of the three-dimensional position information of the target traffic sign obtained from the three-dimensional reconstruction of several traffic objects as a data object point.

[0137] S812. Using the data object point as the core point, perform clustering processing based on the preset neighborhood radius and the preset number threshold of target traffic signs within the preset neighborhood radius to obtain several location clusters.

[0138] In this cluster, the three-dimensional location information in the location cluster corresponds to the same target traffic sign;

[0139] S814. Average the three-dimensional position information in any location cluster to obtain the average three-dimensional position of the target traffic sign.

[0140] S816. Using the average three-dimensional position of the target traffic sign as the initial position, the average three-dimensional position is optimized using the bundle adjustment method to obtain the three-dimensional position information of the target traffic sign corresponding to any position cluster.

[0141] S818. The external parameters of the image acquisition device are optimized using the bundle adjustment method to obtain the optimized external parameters.

[0142] This specification provides a three-dimensional reconstruction device 900 for the location of traffic signs. Please refer to [link / reference]. Figure 9 The three-dimensional reconstruction device 900 for traffic sign locations includes: a location border determination module 910, an associated border combination determination module 920, and a three-dimensional location information determination module 930.

[0143] The position bounding box determination module 910 is used to determine the position bounding box of the target traffic sign in N consecutive frames of traffic images; wherein the position bounding box in two adjacent frames of traffic images is separated by a first number of pixels; N is a positive integer greater than or equal to 2;

[0144] The associated border combination determination module 920 is used to determine a target associated border combination in the position borders of the N frames of traffic images based on the first quantity; wherein the target associated border combination includes a plurality of target position borders corresponding to the same target traffic sign.

[0145] The three-dimensional position information determination module 930 is used to perform three-dimensional reconstruction based on the position information of the target position border and the shooting pose information corresponding to the target traffic image to which the target position border belongs, to obtain the three-dimensional position information of the target traffic sign; wherein, the shooting pose information is the pose information of the traffic object at the shooting time of the target traffic image.

[0146] This specification provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0147] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0148] One embodiment of this specification provides a computer program product including instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0149] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

Claims

1. A method for three-dimensional reconstruction of the location of a traffic sign, characterized in that, The method includes: Determine the position bounding box of the target traffic sign in N consecutive traffic images; wherein the position bounding box in two adjacent traffic images is separated by a first number of pixels; N is a positive integer greater than or equal to 2; Based on the first quantity, a target associated border combination is determined from the location borders in the N frames of traffic images; wherein, the target associated border combination includes several target location borders corresponding to the same target traffic sign; Based on the position information of the target location border and the shooting pose information of the target traffic image to which the target location border belongs, a three-dimensional reconstruction is performed to obtain the three-dimensional position information of the target traffic sign; wherein, the shooting pose information is the pose information of the traffic object at the time of shooting the target traffic image; The step of determining the target associated bounding box combination in the location bounding boxes of the N frames of traffic images based on the first quantity includes: The two location bounding boxes corresponding to the number of targets less than the pixel number threshold in the first quantity are determined as the initial associated bounding box combination; wherein, the initial associated bounding box combination has a confidence level that the two location bounding boxes included in the initial associated bounding box combination correspond to the same target traffic sign; In the initial associated border combination, the target associated border combination is determined based on the confidence level of the initial associated border combination.

2. The method according to claim 1, characterized in that, The confidence level is determined by the number of pixels between the two positional borders included in the initial associated border combination.

3. The method according to claim 1, characterized in that, The initial associated border combination includes two position borders, each containing a second number of pixels and a third number of pixels; the pixel count threshold is determined by the number of pixels within a specified position border; wherein, the specified position border is designated among the two position borders included in the initial associated border combination based on a comparison between the second number and the third number.

4. The method according to claim 1, characterized in that, The two positional borders included in the initial associated border combination are respectively denoted as the first positional border and the second positional border with an initial connection relationship. The first positional border belongs to the first traffic image, and the second positional border belongs to the second traffic image. In the initial associated border combination, determining the target associated border combination based on the confidence level of the initial associated border combination includes: If any location border in the first traffic image has an initial connection relationship with multiple location borders in the second traffic image, then among the multiple location borders in the second traffic image, the location border with the target connection relationship with any location border is determined based on the confidence level of the initial associated border combination. The border at any given location, and the border at any given location that has a target connection relationship with the border at that given location, are determined as the target associated border combination.

5. The method according to claim 4, characterized in that, Determining the location border with a target connection relationship to any one of the multiple location borders in the second traffic image based on the confidence level of the initial associated border combination includes: The location bounding box in the second traffic image corresponding to the maximum confidence level is taken as the location bounding box that has a target connection relationship with any of the location bounding boxes.

6. The method according to claim 1, characterized in that, The target traffic image is obtained by capturing the target traffic sign using an image acquisition device installed on the traffic object; the step of performing three-dimensional reconstruction based on the position information of the target location border and the shooting pose information corresponding to the target traffic image to which the target location border belongs, to obtain the three-dimensional position information of the target traffic sign, includes: Based on the position information of the target location bounding box and the shooting pose information, triangulation calculation is performed to obtain the estimated three-dimensional position of the target traffic sign; Using the minimum reprojection error as the criterion, the estimated three-dimensional position is optimized based on the intrinsic and extrinsic parameters of the image acquisition device, the estimated three-dimensional position, and the shooting pose information to obtain the three-dimensional position information of the target traffic sign.

7. The method according to claim 1, characterized in that, The method further includes: Clustering processing is performed on the three-dimensional location information of the target traffic sign obtained from the three-dimensional reconstruction of several traffic objects to obtain several location clusters; wherein the three-dimensional location information in the location clusters corresponds to the same target traffic sign; The three-dimensional location information of the target traffic sign corresponding to any location cluster is determined based on the three-dimensional location information in any location cluster.

8. The method according to claim 7, characterized in that, The clustering process, based on the 3D location information of the target traffic signs obtained from the 3D reconstruction of several traffic objects, yields several location clusters, including: From the three-dimensional position information of the target traffic sign obtained by three-dimensional reconstruction of several traffic objects, any one of the three-dimensional position information is determined as a data object point; Using the data object points as the core points, clustering is performed based on a preset neighborhood radius and a preset threshold for the number of target traffic signs within the preset neighborhood radius to obtain the several location clusters.

9. The method according to claim 7, characterized in that, The step of determining the three-dimensional location information of the target traffic sign corresponding to any location cluster based on the three-dimensional location information in any location cluster includes: The average three-dimensional position of the target traffic sign is obtained by averaging the three-dimensional position information in any of the location clusters. Using the average three-dimensional position of the target traffic sign as the initial position, the average three-dimensional position is optimized using the bundle adjustment method to obtain the three-dimensional position information of the target traffic sign corresponding to any position cluster.

10. The method according to claim 9, characterized in that, The method further includes: The extrinsic parameters of the image acquisition device were optimized using the bundle adjustment method to obtain the optimized extrinsic parameters.

11. A three-dimensional reconstruction device for the location of traffic signs, characterized in that, The device includes: The position bounding box determination module is used to determine the position bounding box of the target traffic sign in N consecutive frames of traffic images; wherein the position bounding boxes in two adjacent frames of traffic images are separated by a first number of pixels; N is a positive integer greater than or equal to 2; The associated border combination determination module is used to determine a target associated border combination among the position borders in the N frames of traffic images based on the first quantity; wherein, the target associated border combination includes a plurality of target position borders corresponding to the same target traffic sign; The three-dimensional position information determination module is used to perform three-dimensional reconstruction based on the position information of the target position border and the shooting pose information corresponding to the target traffic image to which the target position border belongs, to obtain the three-dimensional position information of the target traffic sign; wherein, the shooting pose information is the pose information of the traffic object at the time of shooting the target traffic image; The step of determining the target associated bounding box combination in the location bounding boxes of the N frames of traffic images based on the first quantity includes: The two location bounding boxes corresponding to the number of targets less than the pixel number threshold in the first quantity are determined as the initial associated bounding box combination; wherein, the initial associated bounding box combination has a confidence level that the two location bounding boxes included in the initial associated bounding box combination correspond to the same target traffic sign; In the initial associated border combination, the target associated border combination is determined based on the confidence level of the initial associated border combination.

12. A three-dimensional reconstruction device for the location of traffic signs, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images