Signboard positioning method, electronic device and storage medium

Abstract

The present application discloses a signboard positioning method, electronic device and storage medium, and relates to the field of computer vision technology, wherein the method includes: based on a binocular camera and radar set on traffic equipment, synchronously collecting a binocular view and an initial point cloud containing a target signboard with the same field of view; fusing the point cloud obtained after stereo matching of the binocular view with the initial point cloud, and determining the relative position between the target signboard and the traffic equipment based on the fusion result; determining the spatial position of the target signboard based on the relative position and the spatial position of the traffic equipment at the time of synchronous collection. The method and device provided by the present application improve the positioning accuracy of the target signboard and improve the positioning stability of the target signboard.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a signboard positioning method, an electronic device and a storage medium. Background Art

[0002] Traffic signs are an important road infrastructure, used to control, warn and guide traffic safety. They play an important role in modern road traffic management. Since traffic signs are placed outdoors in natural scenes, they are often affected by the natural environment, such as sun scorching, rain, temperature changes, vegetation shielding, etc. They are also easily affected by human factors such as deliberate damage, random replacement, and vehicle collisions, causing damage, loss, addition, displacement, and layout changes. Therefore, it is particularly important to inspect traffic signs regularly.

[0003] How to accurately determine the position of traffic signs on the road has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The present application provides a signboard positioning method, an electronic device and a storage medium, which are used to solve the technical problem of how to accurately determine the position of a traffic sign on a road.

[0005] The present application provides a signboard positioning method, comprising:

[0006] Based on the binocular camera and radar installed on the traffic equipment, the binocular view and initial point cloud containing the target sign are synchronously collected with the same field of view;

[0007] Fusing the point cloud obtained by stereo matching the binocular view with the initial point cloud, and determining the relative position between the target sign and the traffic equipment based on the fusion result;

[0008] Based on the relative position and the spatial position of the traffic equipment at the time of synchronous collection, the spatial position of the target sign is determined.

[0009] In some embodiments, the step of fusing the point cloud obtained by stereo matching the binocular view with the initial point cloud, and determining the relative position between the target sign and the traffic equipment based on the fusion result, includes:

[0010] Determine a position prediction frame of the target sign in the binocular view;

[0011] Segmenting the initial point cloud based on the position prediction frame to obtain a first point cloud of the target sign;

[0012] Cropping the binocular view based on the position prediction frame, and performing stereo matching on the cropping result to obtain a second point cloud of the target sign;

[0013] Based on the fusion result of the first point cloud and the second point cloud, the relative position between the target sign and the traffic equipment is determined.

[0014] In some embodiments, determining a position prediction frame of the target sign in the binocular view includes:

[0015] Inputting the binocular view into a sign detection model to obtain a position prediction frame of the target sign in the binocular view output by the sign detection model;

[0016] Wherein, the landmark detection model is constructed based on the YOLO model;

[0017] The landmark detection model is further used to perform a pooling operation on the output feature map based on the regression branch in the Head module in the YOLO model to obtain a plurality of grid points; the pooling operation includes at least one of corner point pooling, middle point pooling and edge pooling;

[0018] Based on the positions of the multiple grid points in the binocular view, a position prediction box of the target sign is determined.

[0019] In some embodiments, the step of cropping the binocular view based on the position prediction frame and performing stereo matching on the cropping result to obtain a second point cloud of the target sign includes:

[0020] Adjusting the size of the position prediction frame, and cropping the binocular view based on the adjusted position prediction frame to obtain a visually adjusted image;

[0021] The visual adjustment image is input into the stereo matching model to obtain a second point cloud of the target sign output by the stereo matching model.

[0022] In some embodiments, the adjusting the size of the position prediction frame and cropping the binocular view based on the adjusted position prediction frame to obtain a visually adjusted image includes:

[0023] When the size of the position prediction frame is greater than or equal to the first size and smaller than the second size, the position prediction frame is expanded to the second size in the binocular view, and the binocular view is cropped based on the expanded position prediction frame to obtain the visual adjustment image;

[0024] When the size of the position prediction box is smaller than the first size, the position prediction box is expanded to the first size in the binocular view, the binocular view is cropped based on the expanded position prediction box, and the cropped result is upsampled based on the second size, and the upsampled result is used as the visual adjustment image.

[0025] In some embodiments, the stereo matching model is constructed based on the ACVNet model;

[0026] The cost space building block in the ACVNet model is also used to:

[0027] Determine an image point with the same name corresponding to any pixel point in the first view in the binocular view in the second view in the binocular view;

[0028] Determine a first disparity value of any pixel point in the binocular view;

[0029] Determine a second disparity value of the same-name image point in the binocular view;

[0030] Accordingly, the disparity prediction module in the ACVNet model is also used for:

[0031] When the absolute value of the difference between the first disparity value and the second disparity value is greater than or equal to a preset threshold, the any pixel point is determined as an occlusion point, and the disparity value of the any pixel point is determined based on the disparity values ​​of the adjacent pixels of the any pixel point.

[0032] In some embodiments, determining the relative position between the target sign and the traffic equipment based on the fusion result of the first point cloud and the second point cloud includes:

[0033] Registering the first point cloud and the second point cloud, and determining Gaussian distribution features of the registered first point cloud and Gaussian distribution features of the registered second point cloud;

[0034] Determining a fusion weight of the first point cloud and a fusion weight of the second point cloud based on a Gaussian distribution feature of the first point cloud and a Gaussian distribution feature of the second point cloud;

[0035] Based on the fusion weight of the first point cloud and the fusion weight of the second point cloud, the first point cloud and the second point cloud are fused, and the relative position between the target sign and the traffic equipment is determined based on the fusion result.

[0036] In some embodiments, before registering the first point cloud and the second point cloud, the method further includes:

[0037] In the case where it is determined that the first point cloud and the second point cloud cannot be time-aligned, determining a current first point cloud and a current second point cloud; the difference between the acquisition time of the current second point cloud and the current first point cloud is the smallest;

[0038] Determine the coordinate representation of the spatial point in the current first point cloud in the world coordinate system based on the coordinate representation of the spatial point in the current first point cloud in the radar coordinate system and the coordinate transformation relationship between the radar coordinate system where the current first point cloud is located and the world coordinate system;

[0039] Determine the coordinate representation of the spatial point in the current first point cloud in the camera coordinate system based on the coordinate representation of the spatial point in the current first point cloud in the world coordinate system and the coordinate transformation relationship between the camera coordinate system where the current second point cloud is located and the world coordinate system;

[0040] The current first point cloud and the current second point cloud are aligned based on the coordinate representation of the spatial points in the current first point cloud in the camera coordinate system and the coordinate representation of the spatial points in the current second point cloud in the camera coordinate system.

[0041] The present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the signboard positioning method when executing the program.

[0042] The present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program implements the signboard positioning method when executed by a processor.

[0043] The signboard positioning method, electronic device and storage medium provided by the present application are characterized in that a binocular camera and a radar are arranged on the traffic equipment, and a binocular view including a target signboard and an initial point cloud including the target signboard are synchronously collected with the same field of view; the point cloud obtained after stereo matching of the binocular view is fused with the initial point cloud to determine the relative position between the target signboard and the traffic equipment; the spatial position of the target signboard is further determined according to the position of the traffic equipment; since the binocular vision data collected by the camera and the point cloud data collected by the radar are fused, the point cloud obtained after the fusion has higher accuracy, thereby improving the positioning accuracy of the target signboard; in addition, different types of sensors are used to collect data on the target signboard, which can eliminate the influence of environmental factors such as light and improve the positioning stability of the target signboard. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is one of the flow charts of a signboard positioning method provided by an embodiment of the present application;

[0047] Figure 2 is a structural diagram of a landmark detection model provided by an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of grid point pooling provided by an embodiment of the present application;

[0049] Figure 4 is a distribution diagram of grid points provided by an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of left-right consistency detection provided by an embodiment of the present application;

[0051] Figure 6 is a structural schematic diagram of a stereo matching model provided by an embodiment of the present application;

[0052] Figure 7 is a structural diagram of a feature extraction module provided by an embodiment of the present application;

[0053] Figure 8 is a schematic diagram of spatial pyramid pooling provided by an embodiment of the present application;

[0054] Fig. 9 It is a schematic diagram of the structure of a cost space aggregation module provided by an embodiment of the present application;

[0055] Fig.10 It is a schematic diagram of data collection provided by an embodiment of the present application;

[0056] Fig.11 This is the second flow chart of a signboard positioning method provided by an embodiment of the present application;

[0057] Fig.12 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0059] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0060] The application scenario of the signboard positioning method provided in this application is to detect and locate the signboards on both sides of the road. The road here can include land transportation roads such as highways and railways, and can also include water transportation roads such as rivers, and can also be a track designed for racing or rowing. The signboard here can be a traffic signboard, or other landmarks such as an advertising signboard.

[0061] In the embodiments of the present application, the detection and positioning of traffic signs on both sides of a highway are taken as an example to illustrate the sign positioning method provided by the present application. The present application does not specifically limit the application scenario of the provided sign positioning method.

[0062] Figure 1 This is one of the flow charts of a signboard positioning method provided by an embodiment of the present application, such as Figure 1 As shown, the method includes step 110, step 120, and step 130.

[0063] Step 110: Based on the binocular camera and radar set on the traffic equipment, the binocular view and the initial point cloud containing the target sign are synchronously collected with the same field of view.

[0064] Specifically, the subject of executing the signboard positioning method provided in the embodiment of the present application is a signboard positioning device. The device can be embodied in the form of hardware, such as a computer set on the traffic equipment or a remotely set cloud server that communicates with the traffic equipment, etc.; it can also be embodied in the form of software, such as a software program that executes the signboard positioning method running on a computer or a cloud server.

[0065] Taking the detection and positioning of traffic signs on both sides of the road as an example, the means of transportation can be vehicles driving on the road or drones flying along the road. Binocular cameras and radars can be set on the traffic equipment. The binocular camera can take pictures of the road ahead and obtain binocular views, including left and right views, also known as binocular visual images. The radar can be a millimeter wave radar or a laser radar, which can collect point clouds of the road ahead.

[0066] A binocular camera and a millimeter-wave radar can be installed on a vehicle, and the field of view of the camera and the radar can be set so that the field of view of the two is consistent. When the vehicle is driving on the road, if there is a target sign on the road ahead, a data collection instruction can be sent to the binocular camera and the millimeter-wave radar synchronously, so that the binocular camera and the millimeter-wave radar can synchronously collect data on the road ahead, and obtain a binocular view containing the target sign and an initial point cloud containing the target sign.

[0067] Step 120: Fuse the point cloud obtained by stereo matching the binocular view with the initial point cloud, and determine the relative position between the target sign and the traffic equipment based on the fusion result.

[0068] Specifically, the binocular view is based on the parallax principle and uses imaging equipment to obtain two images of the target sign from different positions. Stereo matching of the binocular view can obtain a point cloud that records the three-dimensional geometric information of the target sign.

[0069] Since the same field of view angle is used for synchronous acquisition, the obtained binocular view and initial point cloud are both acquired from the front field of view of the traffic equipment. Therefore, the object information in the binocular view corresponds to the object information in the initial point cloud. Furthermore, the object information in the point cloud after stereo matching of the binocular view corresponds to the object information in the initial point cloud.

[0070] Therefore, the point cloud obtained after stereo matching of the binocular view can be fused with the initial point cloud to obtain a fusion result. According to the three-dimensional coordinates of each point in the fusion result, the shape and position of each object in the front field of view of the traffic equipment can be determined. The position information of the target sign can be obtained from these objects.

[0071] Since both the binocular camera and the millimeter-wave radar are installed on the traffic equipment, the camera coordinate system where the point cloud data collected by the binocular camera is located and the radar coordinate system where the point cloud data collected by the millimeter-wave radar is located are both based on the location of the traffic equipment as the origin of the coordinate system, so the location information here is the relative location information of the target marker relative to the traffic equipment. Based on this location information, the relative position between the target sign and the traffic equipment can be determined.

[0072] The initial point cloud is a sparse point cloud obtained by radar. Although the accuracy is low, the radar can operate all day and night, is not easily affected by environmental factors such as lighting, and has high stability. The point cloud obtained after stereo matching of the binocular view is a dense point cloud. Although the accuracy is high, the camera is easily affected by environmental factors such as lighting, and has low stability. The two types of point clouds are obtained by different sensors. The point cloud obtained after fusion has high accuracy and high stability.

[0073] Step 130: Determine the spatial position of the target sign based on the relative position and the spatial position of the traffic equipment at the time of synchronous collection.

[0074] Specifically, a positioning device, such as an inertial measurement unit (IMU) and a global positioning system (GPS), can be installed on the traffic equipment to obtain the spatial position of the traffic equipment at each moment during the driving process in real time.

[0075] The spatial position of the target sign can be determined based on the spatial position of the traffic equipment at the time of synchronous collection and the relative position between the target sign and the traffic equipment obtained after point cloud fusion. The synchronous collection time here is the time of synchronously collecting the binocular view and the initial point cloud.

[0076] The signboard positioning method provided in the embodiment of the present application is to set a binocular camera and a radar on the traffic equipment, and synchronously collect a binocular view containing a target signboard and an initial point cloud containing the target signboard with the same field of view; the point cloud obtained after stereo matching of the binocular view is fused with the initial point cloud to determine the relative position between the target signboard and the traffic equipment; further determine the spatial position of the target signboard according to the position of the traffic equipment; since the binocular vision data collected by the camera and the point cloud data collected by the radar are fused, the point cloud obtained after the fusion has higher accuracy, thereby improving the positioning accuracy of the target signboard; in addition, the use of different types of sensors to collect data on the target signboard can eliminate the influence of environmental factors such as light, thereby improving the positioning stability of the target signboard.

[0077] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0078] In some embodiments, step 120 includes:

[0079] Determine the position prediction frame of the target sign in the binocular view;

[0080] The initial point cloud is segmented based on the position prediction frame to obtain the first point cloud of the target sign;

[0081] The binocular view is cropped based on the position prediction frame, and the cropped result is stereo matched to obtain the second point cloud of the target sign;

[0082] Based on the fusion result of the first point cloud and the second point cloud, the relative position between the target sign and the traffic equipment is determined.

[0083] Specifically, the target detection model can be used to perform target detection on the binocular view to determine the position prediction box of the target sign in the binocular view. The target detection model may include a YOLO (You Only Look Once) model, an R-CNN (Region Convolutional Neural Networks) model, etc. The position prediction box is a frame that marks the position area of ​​the target object in the image, such as a rectangular detection box (bounding box).

[0084] Since the object information in the binocular view corresponds to the object information in the initial point cloud, the initial point cloud can be segmented according to the position prediction frame of the target sign in the binocular view, thereby obtaining the first point cloud of the target sign. The position prediction frame of the target sign in any view of the binocular view can be selected to segment the initial point cloud, for example, the left view can be selected.

[0085] At the same time, the binocular view is cropped according to the position prediction frame, and then the cropped result is stereo matched to obtain the second point cloud of the target sign. Alternatively, the stereo matching point cloud of the binocular view can be segmented according to the position prediction frame of the target sign in the binocular view to obtain the second point cloud of the target sign.

[0086] The first point cloud and the second point cloud both describe the three-dimensional geometric information of the same target sign. The data types are both point clouds. The radar coordinate system where the first point cloud is located and the camera coordinate system where the second point cloud is located are both spatial coordinate systems established with the traffic equipment as the origin. Therefore, the first point cloud and the second point cloud can be fused. The relative position between the target sign and the traffic equipment is determined by calculating the coordinates of each spatial point in the fused point cloud.

[0087] For example, the coordinate average of all spatial points is calculated based on the coordinates of each spatial point in the fused point cloud, the center of gravity of the point cloud is determined based on the average value of the coordinates, and the coordinates of the center of gravity are determined as the position of the target sign. Since the origin of the coordinate system is the traffic equipment, the position is a relative position, which is used to describe the positional relationship of the target sign relative to the traffic equipment.

[0088] The signboard positioning method provided in the embodiment of the present application performs target detection on the binocular view, determines the position prediction frame of the target signboard, and then segments the initial point cloud according to the position prediction frame to obtain the first point cloud, and obtains the second point cloud according to the position prediction frame and the binocular view. Since the image target detection method is adopted, the point cloud data can be efficiently segmented and processed, the amount of calculation of the point cloud data is reduced, the influence of irrelevant point cloud data is eliminated, the positioning accuracy of the target signboard is improved, and the efficiency of signboard positioning is improved.

[0089] In some embodiments, determining a position prediction frame of a target sign in a binocular view includes:

[0090] Input the binocular view into the sign detection model to obtain the position prediction frame of the target sign in the binocular view output by the sign detection model;

[0091] Among them, the landmark detection model is built based on the YOLO model;

[0092] The landmark detection model is used to perform a pooling operation on the output feature map based on the regression branch in the Head module of the YOLO model to obtain multiple grid points; the pooling operation includes corner pooling, middle point pooling, and edge pooling;

[0093] Based on the positions of multiple grid points in the binocular view, the position prediction box of the target sign is determined.

[0094] Specifically, a sign detection model can be constructed based on the YOLO model, and the binocular view can be input into the sign detection model to obtain a position prediction box of the target sign in the binocular view.

[0095] Taking the left view as an example, the left view is input into the sign detection model, which extracts features from the left view and identifies the position prediction box of the target sign in the left view based on the extracted features.

[0096] The landmark detection model in the embodiment of the present application is based on the YOLO model and has been improved. In the related art, the YOLO model generally includes a backbone network module, a feature enhancement module and a result prediction module. Figure 2 is a schematic diagram of the structure of a landmark detection model provided by an embodiment of the present application. Figure 2As shown in the figure, compared with the YOLO model in the related art, the improvements of the landmark detection model are as follows:

[0097] (1) The backbone network module 210 (Backbone) adopts the CSPDarknet model, and uses the residual network (Residual Net), CSPnet network structure, Focus network structure and SPP network structure.

[0098] Specifically, the backbone network module consists of an input layer (Inputs), a Focus network layer, a Conv2D_BN_SiLU convolution layer, and four residual network modules (ResBlock_body) connected in sequence. Each residual network module includes a Conv2D_BN_SiLU convolution layer and a CSPLayer layer. The last residual network module also includes an SPPBottleneck module. The SPPBottleneck module is a spatial pyramid pooling module that includes at least two Conv2D_BN_SiLU convolution layers.

[0099] (2) The feature enhancement module 220 uses the FPN network (Feature Pyramid Networks).

[0100] Specifically, the module includes an upsampling branch and a downsampling branch. The upsampling branch includes the first convolutional layer (Conv2D), the first upsampling layer (UpSampling2D), the first fusion layer (Concat+CSPLayer), the second convolutional layer (Conv2D), the second upsampling layer (UpSampling2D) and the second fusion layer (Concat+CSPLayer) connected in sequence; the downsampling branch includes the first downsampling layer (DownSampling), the third fusion layer (Concat+CSPLayer), the second downsampling layer (DownSampling) and the fourth fusion layer (Concat+CSPLayer) connected in sequence.

[0101] (3) The result prediction module 230 uses the YOLO Head detection head module.

[0102] Specifically, the module includes a Conv2D_BN_SiLU convolution layer, and a classification branch (Cls) and a regression branch (Reg+Obj) connected to the Conv2D_BN_SiLU convolution layer. Each branch is composed of sequentially connected Conv2D_BN_SiLU convolution layers.

[0103] Among them, the sign detection model in the embodiment of the present application is used to perform a pooling operation on the output feature map based on the regression branch in the Head module in the YOLO model to obtain multiple grid points. According to the positions of the multiple grid points in the binocular view, the position prediction frame of the target sign is determined. The pooling operation includes corner pooling, middle point pooling and edge pooling.

[0104] For the signs in the target sign, for example, the upper left corner of the traffic sign is not on the sign, and this point and its adjacent points and surrounding pixels share similar local features, so it is very difficult to accurately predict the corner points. To this end, the embodiment of the present application introduces a multi-grid point supervision mechanism, and 3×3 grid points (3 rows and 3 columns) can be used to optimize the detection position prediction box. These grid points supervise the position of the position prediction box to facilitate the quick and accurate positioning of the position prediction box.

[0105] The corner point in the upper left corner can be corrected using the ordinate information of the center point of the upper boundary of the traffic sign and the corner point in the upper right corner. It is worth noting that the information of the center position of each side only serves as an auxiliary correction for the corner point and the center point, and does not directly participate in the calculation of the loss function.

[0106] For the 9 grid points, the output feature map obtained by the feature enhancement module is visualized and converted into a heat map, and corner pooling, edge pooling, and center point pooling are performed respectively.

[0107] Figure 3 is a schematic diagram of grid point pooling provided by an embodiment of the present application, such as Figure 3 As shown in the figure, after obtaining the heat map of the grid points by grouping, the position corresponding to the point with the largest heat value in each category is taken as the reference point, and the position of the grid point in the original image is obtained by translation mapping back to the original image through the relationship between pixels. In order to simplify the description of the detection box, the upper left corner point, lower right corner point and center point are used to replace the 9 grid points for output. The circles in the figure represent grid points.

[0108] Figure 4 is a distribution diagram of grid points provided by an embodiment of the present application, such as Figure 4 As shown in Figure 1, the four edges of the position prediction box are E1, E2, E3 and E4. x and E y The center line where the center point is located.

[0109] The coordinates of the top left and bottom right corners of the position prediction box (x l ,y u ), (x r ,y b ), the coordinates of the center point are (x c ,y c ). The position description information of the position prediction box becomes (x l ,yu ,x r ,y b ,x c ,y c ).

[0110] These coordinates are calculated as:

[0111]

[0112]

[0113]

[0114] Where N = 3, p j is the probability value in the heat map, x j is the horizontal coordinate of the jth point, y j is the ordinate of the jth point, and j is the serial number of the point on each edge.

[0115] The signboard positioning method provided in the embodiment of the present application adds grid point supervision to the regression branch in the Head module in the YOLO network, thereby improving the detection accuracy of the sign in the target signboard, improving the detection accuracy of the position prediction box, reducing the amount of calculation of the point cloud data, eliminating the influence of irrelevant point cloud data, improving the positioning accuracy of the target signboard, and improving the efficiency of signboard positioning.

[0116] In addition, it is also helpful to improve the accuracy of optimal distance screening in binocular stereo matching. After obtaining the depth map of the mark, it is necessary to count the distribution of distance values ​​inside the position prediction frame, which includes some background pixels. The position prediction frame obtained by using 9 grid points can focus more on the statistics of the effective area, which is helpful to improve the prediction accuracy of the distance value.

[0117] In some embodiments, the binocular view is cropped based on the position prediction frame, and stereo matching is performed on the cropped result to obtain a second point cloud of the target sign, including:

[0118] The size of the position prediction frame is adjusted, and the binocular view is cropped based on the adjusted position prediction frame to obtain a visually adjusted image;

[0119] The visually adjusted image is input into the stereo matching model to obtain a second point cloud of the target sign output by the stereo matching model.

[0120] Specifically, when the traffic equipment is far away from the target, the stereo matching effect is not ideal. The reason is that the image size of the distant target is small, its feature points are relatively sparse, and the stereo matching effect is poor.

[0121] Therefore, the size of the position prediction box can be adjusted. For example, when the size of the position prediction box is small, it can be adjusted in the binocular view. The adjustment operation may include cropping and scaling, etc., to adaptively adjust the size of the position prediction box to crop the image content containing the target sign, thereby removing the interference of irrelevant content in the binocular view, reducing the amount of point cloud data finally generated, improving the positioning accuracy of the target sign, and improving the efficiency of sign positioning.

[0122] The stereo matching model is used to perform stereo matching on the visual adjustment image to obtain the second point cloud of the target sign. The stereo matching model can be constructed using a pyramid network, a twin network, a generative adversarial network, or an ACVNet model.

[0123] In some embodiments, adjusting the size of the position prediction frame, and cropping the binocular view based on the adjusted position prediction frame to obtain a visually adjusted image includes:

[0124] When the size of the position prediction frame is greater than or equal to the first size and smaller than the second size, the position prediction frame is expanded to the second size in the binocular view, and the binocular view is cropped based on the expanded position prediction frame to obtain a visually adjusted image;

[0125] When the size of the position prediction box is smaller than the first size, the position prediction box is expanded to the first size in the binocular view, the binocular view is cropped based on the expanded position prediction box, and the cropped result is upsampled based on the second size, and the upsampled result is used as a visual adjustment image.

[0126] Specifically, the first size and the second size can be set as needed. The first size is smaller than the second size. For example, the first size can be set to The second size may be set to (w0, h0), where w0 is a preset width and h0 is a preset height.

[0127] The size of the position prediction box can be expressed as (w i ,h i ). Among them, w i is the width of the position prediction box, h i is the height of the position prediction box.

[0128] If the size of the position prediction box (w i ,h i ) is greater than or equal to the second size, it can be considered that the image content determined by the position prediction box can meet the needs of stereo matching and no adjustment is required;

[0129] If the size of the position prediction box (w i ,h i) is smaller than the second size, it can be considered that the image content determined by the position prediction box does not meet the requirements of stereo matching and needs to be adjusted, and its size should be adjusted to at least the second size.

[0130] It can be expressed as:

[0131]

[0132] Among them, w is the width of the adjusted position prediction box, and h is the height of the adjusted position prediction box.

[0133] For position prediction boxes smaller than the second size:

[0134] If the size of the position prediction frame is greater than or equal to the first size and smaller than the second size, the position prediction frame is expanded to the second size in the binocular view, and the binocular view is cropped according to the expanded position prediction frame, and a portion containing the target mark frame is used as a visual adjustment image;

[0135] If the size of the position prediction frame is smaller than the first size, the position prediction frame is expanded to the first size in the binocular view, and the binocular view is cropped based on the expanded position prediction frame. The cropped result (the portion including the target mark frame) is then upsampled so that the upsampled image size reaches the second size, and the upsampled image (upsampled result) is used as the visual adjustment image.

[0136] The signboard positioning method provided in the embodiment of the present application adjusts the size of the position prediction box, changes the image information input into the stereo matching model, solves the problem of inability to accurately obtain point clouds for distant signboards, improves the accuracy of the stereo matching point cloud, and improves the positioning accuracy of the target signboard.

[0137] In some embodiments, the stereo matching model is built based on the ACVNet model;

[0138] The cost space building blocks in the ACVNet model are also used to:

[0139] Determine an image point with the same name corresponding to any pixel point in the first view in the binocular view in the second view in the binocular view;

[0140] Determine a first disparity value of any pixel point in the binocular view;

[0141] Determine a second disparity value of the same-name image point in the binocular view;

[0142] Accordingly, the disparity prediction module in the ACVNet model is also used to:

[0143] When the absolute value of the difference between the first disparity value and the second disparity value is greater than a preset threshold, any pixel point is determined as an occlusion point, and the disparity value of any pixel point is determined based on the disparity values ​​of its adjacent pixels.

[0144] Specifically, Figure 5 is a schematic diagram of the structure of a stereo matching model provided by an embodiment of the present application, such as Figure 5 As shown, the stereo matching model in the embodiment of the present application is constructed based on the ACVNet model, including a feature extraction module 510 , a cost space construction module 520 , a cost space aggregation module 530 and a disparity prediction module 540 .

[0145] Among them, the feature extraction module is used to downsample the input image through convolution and residual operations (for example, downsample to one-fourth the size of the original image), increase the number of channels (for example, increase to 320 channels), and thus obtain feature data of the entire image.

[0146] The cost space construction module is used to introduce the attention mechanism, which divides the network into two parts. One part uses group-wise correlation to group and construct a multi-dimensional cost space (for example, 4 dimensions), and the other part introduces multi-level adaptive patches to obtain attention weights. The weights are added to the cost space to obtain the cost space body of the attention mechanism.

[0147] The cost space aggregation module enriches the information between cost space levels through certain convolution operations, while reducing the disparity by the same size, improving the speed and increasing the receptive field.

[0148] The disparity prediction module uses the soft argmin method to solve the problem that general disparity prediction methods cannot be back-propagated and find the optimal disparity value between the left and right views.

[0149] The improvement of the stereo matching model in the embodiment of the present application is to add left-right consistency detection in the cost space construction module and to add left-right consistency judgment in the disparity prediction module to eliminate the influence of disparity errors caused by left and right occlusions.

[0150] Figure 6 is a schematic diagram of left-right consistency detection provided by an embodiment of the present application, such as Figure 6 As shown, the left view in the binocular view First view, right view For the second view, determine any pixel point in the left view Among them, l represents the left view, t0 represents the image acquisition time, is the horizontal coordinate of the pixel in the left view, is the vertical coordinate of the pixel in the left view.

[0151] The same-name image points are obtained by taking the same object point by two cameras in the binocular camera, and are reflected in the pixels in the binocular view. The same-name image points are also called corresponding image points, which refer to the corresponding pixels of any target point in the photos of different views. According to the shape characteristics of the target marker, the corresponding relationship between each pixel in the left and right views can be determined, and the same-name image points of each pixel can also be determined.

[0152] Pixels The corresponding image points with the same name in the second view are Among them, r represents the right view, is the horizontal coordinate of the pixel in the right view, is the vertical coordinate of the pixel in the right view.

[0153] Pixels can be used The horizontal and vertical coordinates express The horizontal and vertical coordinates are in, Pixel The first disparity value in the binocular view.

[0154] Similarly, pixel points can be used The horizontal and vertical coordinates express The horizontal and vertical coordinates are in, Pixel The second disparity value in the binocular view.

[0155] The disparity prediction module in the ACVNet model in the embodiment of the present application is also used to perform left-right consistency judgment. The specific method is:

[0156] The absolute value of the difference between the first disparity value and the second disparity value If the pixel is greater than or equal to the preset threshold Δ, the pixel Determined as an occluded point. At this time, the point needs to be eliminated, and the disparity values ​​of the surrounding adjacent pixels are used for interpolation to obtain the disparity value of the pixel. Wherein, Δ is a preset threshold value, which can be set as needed. The embodiment of the present application does not specifically limit the interpolation algorithm.

[0157] The absolute value of the difference between the first disparity value and the second disparity value If the pixel is smaller than the preset threshold Δ, the pixel If a pixel is determined to be a normal point, the disparity value of the pixel may be determined according to a disparity calculation method in the related art.

[0158] The signboard positioning method provided in the embodiment of the present application performs left-right consistency detection in the cost space construction module in the ACVNet model, and determines the left-right consistency in the disparity prediction module, so as to achieve accurate determination of occlusion points, thereby eliminating the erroneous effects caused by left-right occlusions, thereby improving the stereo matching accuracy of the binocular view, improving the accuracy of the stereo matching point cloud, and improving the positioning accuracy of the target signboard.

[0159] In some embodiments, the feature extraction module includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a spatial pyramid pooling layer, and a feature fusion layer. The outputs of the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, and the spatial pyramid pooling layer are all used as inputs to the feature fusion layer.

[0160] Specifically, Figure 7 is a schematic diagram of the structure of a feature extraction module provided by an embodiment of the present application, such as Figure 7 As shown, the feature extraction module 510 includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a spatial pyramid pooling layer and a feature fusion layer.

[0161] Among them, the first feature extraction layer, the second feature extraction layer and the third feature extraction layer can all be implemented using convolutional layers.

[0162] Figure 8 is a schematic diagram of spatial pyramid pooling provided by an embodiment of the present application, such as Figure 8 As shown in the figure, the spatial pyramid pooling layer is used to perform pooling, convolution and upsampling operations on feature maps from different scales, and finally fuse feature maps of different scales. The blocks in the figure represent feature maps.

[0163] The feature fusion layer can be implemented using a multi-level adaptive patch matching layer (MAPM), which can be seamlessly connected to various stereo matching networks, making the network more lightweight and accurate.

[0164] The signboard positioning method provided in the embodiment of the present application adopts spatial pyramid pooling in the feature extraction module, which can improve the matching accuracy of the stereo matching model for small target feature points, improve the accuracy of the stereo matching point cloud, and improve the positioning accuracy of the target signboard.

[0165] In some embodiments, the cost space aggregation module includes a first three-dimensional convolutional layer, a second three-dimensional convolutional layer, a first three-dimensional hourglass network layer, and a second three-dimensional hourglass network layer connected in sequence.

[0166] Fig. 9 is a schematic diagram of the structure of a cost space aggregation module provided by an embodiment of the present application, such as Fig. 9 As shown, the cost space aggregation module 530 includes a first three-dimensional convolution layer 531 , a second three-dimensional convolution layer 532 , a first three-dimensional hourglass network layer 533 , and a second three-dimensional hourglass network layer 534 .

[0167] The input of the second 3D convolutional layer is convolved with the 1×1×1 (width×height×channel) convolutional network layer, and then fused with the output of the second 3D convolutional layer and input into the first 3D hourglass network layer, which serves as the first output of the cost space aggregation module.

[0168] The input of the first 3D hourglass network layer is convolved with the 1×1×1 (width×height×channel) convolutional network layer, and then fused with the output of the first 3D convolutional layer and input into the second 3D hourglass network layer, which serves as the second output of the cost space aggregation module.

[0169] The input of the second 3D hourglass network layer is convolved with the 1×1×1 (width×height×channel) convolutional network layer and then fused with the output of the second 3D convolutional layer as the third output of the cost space aggregation module.

[0170] Among them, the three-dimensional convolution layer can be constructed using a batch normalization layer and a ReLU activation function layer. The three-dimensional hourglass network layer can be constructed using an encoder-decoder structure, which is achieved by stacking four three-dimensional convolution layers and two three-dimensional deconvolution layers.

[0171] The signboard positioning method provided in the embodiment of the present application associates the outputs of each network layer in the cost space aggregation module, realizes residual connection, can prevent model overfitting, improves the accuracy of stereo matching point cloud, and improves the positioning accuracy of the target signboard.

[0172] In some embodiments, determining the relative position between the target sign and the traffic equipment based on the fusion result of the first point cloud and the second point cloud includes:

[0173] Registering the first point cloud and the second point cloud, and determining Gaussian distribution features of the registered first point cloud and Gaussian distribution features of the registered second point cloud;

[0174] Determining a fusion weight of the first point cloud and a fusion weight of the second point cloud based on a Gaussian distribution feature of the first point cloud and a Gaussian distribution feature of the second point cloud;

[0175] The first point cloud and the second point cloud are fused based on the fusion weight of the first point cloud and the fusion weight of the second point cloud, and the relative position between the target sign and the traffic equipment is determined based on the fusion result.

[0176] Specifically, before fusing the first point cloud and the second point cloud, the first point cloud and the second point cloud need to be registered to avoid a large deviation in the ranging result when the binocular stereo matching fails.

[0177] The first point cloud and the second point cloud may be registered by combining a PCA (Principal Component Analysis) algorithm and an ICP (Iterative Closest Point) algorithm.

[0178] The steps of using PCA algorithm are:

[0179] For the first point cloud, its dimension is n, denoted as point set P. Calculate the average value And the covariance matrix cov(P). Perform eigenvector decomposition of the Jacobian matrix on the covariance matrix cov(P), select the main feature direction, and keep as many points as possible in the "discrete" state. Select the average value As the origin of the coordinate system, the main feature direction is the coordinate axis (X axis, Y axis and Z axis), and a new coordinate system of the point set P is constructed. The coordinate system of the second point cloud Q is constructed in the same way, and then the two coordinate systems are adjusted to be consistent, so as to achieve the purpose of rough registration.

[0180] The steps of using the ICP algorithm are:

[0181] The first point cloud P can be used as the target point cloud and the second point cloud Q can be used as the origin point cloud. i and q i is the point with the smallest Euclidean distance between point clouds P and Q (i is the point number), calculated by p i and q i The translation matrix T and rotation matrix R of the corresponding point set are formed, and the obtained rotation matrix and translation matrix are applied to the point cloud P. The Euclidean distance between the new point cloud P′ and Q is calculated by continuous iteration to calculate the loss function f(R,T). The calculation formula is:

[0182]

[0183] Where k represents the number of points that match the two point cloud sets. When the loss function f is less than or equal to the given threshold, the iteration stops, and the transformation matrix (including the translation matrix and the rotation matrix) obtained at this time is the optimal matrix. Through the transformation matrix, the first point cloud and the second point cloud can be accurately registered.

[0184] After registering the first point cloud and the second point cloud, the two groups of point clouds can be subjected to spatial Gaussian distribution statistics to obtain their Gaussian distribution features. Gaussian distribution features include mean and covariance, and the fusion weights of the two groups of point clouds are obtained using Gaussian distribution features.

[0185] For example, taking the covariance calculation fusion weight as an example, the covariance of the first point cloud is The covariance of the second point cloud is The fusion weight r1 of the first point cloud and the fusion weight r2 of the second point cloud can be calculated by the following formula:

[0186]

[0187] According to the fusion weights corresponding to each point cloud, the fused point cloud can be obtained after weighted summation of each point cloud.

[0188] The fusion weight can also be calculated by the mean. For example, the weight is determined based on the average value of the millimeter-wave radar and the binocular camera. If the true value is 10, the millimeter-wave radar average value is 9, and the binocular average value is 12. By comparing the range, it can be determined that the weight of the millimeter-wave radar is 2 / 3 and the binocular is 1 / 3.

[0189] After the first point cloud and the second point cloud are fused, the relative position between the target sign and the traffic equipment can be calculated based on the coordinates of each point in the fused point cloud.

[0190] The signboard positioning method provided in the embodiment of the present application improves the accuracy of the stereo matching point cloud and the positioning accuracy of the target signboard by aligning the first point cloud and the second point cloud and determining the fusion weight of each point cloud according to the Gaussian distribution characteristics of the aligned point clouds.

[0191] In some embodiments, before registering the first point cloud and the second point cloud, the method further includes:

[0192] When it is determined that the first point cloud and the second point cloud cannot be time-aligned, determining the current first point cloud and the current second point cloud; the difference between the acquisition time of the current second point cloud and the current first point cloud is the smallest;

[0193] Determine the coordinate representation of the spatial point in the current first point cloud in the world coordinate system based on the coordinate representation of the spatial point in the current first point cloud in the radar coordinate system and the coordinate conversion relationship between the radar coordinate system where the current first point cloud is located and the world coordinate system;

[0194] Determine the coordinate representation of the spatial point in the current first point cloud in the camera coordinate system based on the coordinate representation of the spatial point in the current first point cloud in the world coordinate system and the coordinate transformation relationship between the camera coordinate system where the current second point cloud is located and the world coordinate system;

[0195] Based on the coordinate representation of the spatial points in the current first point cloud in the camera coordinate system and the coordinate representation of the spatial points in the current second point cloud in the camera coordinate system, the current first point cloud and the current second point cloud are aligned.

[0196] Specifically, during the data collection process, it is necessary to synchronize the collection time of the millimeter wave radar and the binocular camera. If the collection time of the two is the same, the first point cloud and the second point cloud can be aligned in time.

[0197] In actual data collection, in order to obtain as many initial point clouds and binocular views as possible and achieve the best detection effect, traffic equipment will continuously collect initial point clouds and binocular views at multiple collection times. Due to the time difference between different collection devices, the multiple first point clouds and multiple second point clouds obtained may not be aligned in time, thus affecting the point cloud registration effect and sign detection effect.

[0198] For point clouds that cannot be aligned in time, we can rely on the fact that the position of the signboard in space remains unchanged to perform spatial transformation and alignment on the first point cloud and the second point cloud with the smallest collection time interval.

[0199] Fig.10 is a schematic diagram of data collection provided by an embodiment of the present application, such as Fig.10 As shown, the traffic equipment is traveling on the road, and multiple first point clouds (obtained based on the initial point clouds collected by the millimeter-wave radar) and multiple second point clouds (obtained after stereo matching of the binocular views collected by the binocular camera) are continuously collected.

[0200] The points in the current first point cloud can be expressed in the radar coordinate system as Among them, t1 is the acquisition time of the millimeter wave radar, and the position of the traffic equipment at this time is represented by a dotted line. The current point in the second point cloud can be expressed in the camera coordinate system as Among them, t2 is the acquisition time of the binocular camera, and the position of the traffic equipment at this time is represented by a solid line.

[0201] Among them, the time interval between t1 and t2 is the smallest.

[0202] According to the coordinate representation of the spatial point in the current first point cloud in the radar coordinate system and the coordinate conversion relationship between the radar coordinate system where the current first point cloud is located and the world coordinate system, the coordinate representation of the spatial point in the current first point cloud in the world coordinate system is determined, which is expressed by the formula:

[0203]

[0204] in, is the coordinate representation of the spatial point in the current first point cloud in the radar coordinate system, is the coordinate representation of the spatial point in the current first point cloud in the world coordinate system, is the translation matrix between the radar coordinate system and the world coordinate system at time t1, R t1is the rotation matrix between the radar coordinate system and the world coordinate system at time t1, for The inverse matrix of . and Together they represent the coordinate transformation relationship between the radar coordinate system and the world coordinate system at time t1.

[0205] According to the coordinate representation of the spatial point in the current first point cloud in the world coordinate system and the coordinate transformation relationship between the camera coordinate system where the current second point cloud is located and the world coordinate system, the coordinate representation of the spatial point in the current first point cloud in the camera coordinate system is determined, which is expressed by the formula:

[0206]

[0207] in, is the coordinate representation of the spatial point in the current first point cloud in the camera coordinate system, is the translation matrix between the camera coordinate system and the world coordinate system at time t2, is the rotation matrix between the camera coordinate system and the world coordinate system at time t2, and Together they represent the coordinate transformation relationship between the camera coordinate system and the world coordinate system at time t2.

[0208] Through the above transformation, the spatial point in the radar coordinate system at time t1 is transformed into the corresponding spatial point in the camera coordinate system at time t2. The first point cloud at time t2 obtained by the transformation can be further fused with the second point cloud at the same time, thereby aligning the two point clouds in time and space.

[0209] The signboard positioning method provided in the embodiment of the present application adopts a spatial coordinate conversion method to achieve spatial conversion alignment of the first point cloud and the second point cloud, so that the first point cloud and the second point cloud can be registered and fused, thereby improving the accuracy of the stereo matching point cloud and improving the positioning accuracy of the target signboard.

[0210] Fig.11 This is a second flow chart of a signboard positioning method provided by an embodiment of the present application, such as Fig.11 As shown, the method includes:

[0211] Step 1110: The binocular view (including the left view and the right view) with a size of 1920x1080 collected by the binocular camera (including the left camera and the right camera) is adjusted to an image with a size of 960x540 and input into the improved YOLOx network for detection to obtain the area of ​​the traffic sign detection frame.

[0212] Step 1120: Input the binocular view and traffic sign detection frame of size 960x540 into the improved ACVNet network to obtain the point cloud data after binocular stereo matching.

[0213] Step 1130: PCA coarse registration is performed on the millimeter wave radar point cloud data and the binocular stereo matching point cloud data in the traffic sign detection frame, and then ICP fine registration is performed, wherein the millimeter wave radar point cloud is used as the target point cloud and the binocular stereo matching point cloud is used as the source point cloud.

[0214] Step 1140: Perform Gaussian distribution statistics on the two groups of registered point clouds to obtain their respective means and covariance matrices.

[0215] Step 1150: Calculate the weights of the millimeter-wave radar and binocular stereo matching point clouds using the covariance matrix, bring the weights into the calculation of the fused point cloud, and determine the relative coordinates of the traffic sign based on the mean of the fused point cloud.

[0216] Step 1160: Rotate and translate the obtained relative coordinates of the traffic sign and the global coordinate position of the vehicle at this moment to obtain the global coordinate position of the traffic sign, thereby completing the positioning task.

[0217] The signboard positioning method provided in this application has the following beneficial effects:

[0218] (1) The traffic sign detection algorithm uses an improved YOLOX algorithm, which improves the overlap between the traffic sign detection frame and the real target and improves the detection accuracy;

[0219] (2) The traffic sign positioning method uses the method of integrating millimeter-wave radar and binocular vision to improve positioning accuracy and stability;

[0220] (3) The binocular vision positioning algorithm uses the improved ACVNet algorithm to improve the accuracy of binocular stereo matching ranging and the speed of binocular positioning.

[0221] Fig.12 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application, such as Fig.12 As shown, the electronic device may include: a processor (Processor) 1210, a communication interface (Communications Interface) 1220, a memory (Memory) 1230 and a communication bus (Communications Bus) 1240, wherein the processor 1210, the communication interface 1220 and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call the logic command in the memory 1230 to execute the following method:

[0222] Based on the binocular camera and radar installed on the traffic equipment, the binocular view and initial point cloud containing the target sign are synchronously collected with the same field of view; the point cloud obtained after stereo matching of the binocular view is fused with the initial point cloud, and the relative position between the target sign and the traffic equipment is determined based on the fusion result; based on the relative position and the spatial position of the traffic equipment at the time of synchronous collection, the spatial position of the target sign is determined.

[0223] In addition, the logic commands in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0224] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0225] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.

[0226] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0227] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0228] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0229] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for positioning a sign, characterized in that: include: Based on the binocular camera and radar installed on the traffic equipment, the binocular view and initial point cloud containing the target sign are synchronously collected with the same field of view; Fusing the point cloud obtained by stereo matching the binocular view with the initial point cloud, and determining the relative position between the target sign and the traffic equipment based on the fusion result; Determine the spatial position of the target sign based on the relative position and the spatial position of the traffic equipment at the time of synchronous collection; The step of fusing the point cloud obtained by stereo matching the binocular view with the initial point cloud, and determining the relative position between the target sign and the traffic equipment based on the fusion result, includes: Determine a position prediction frame of the target sign in the binocular view; Segmenting the initial point cloud based on the position prediction frame to obtain a first point cloud of the target sign; Cropping the binocular view based on the position prediction frame, and performing stereo matching on the cropping result to obtain a second point cloud of the target sign; Based on the fusion result of the first point cloud and the second point cloud, the relative position between the target sign and the traffic equipment is determined.

2. The signboard positioning method according to claim 1, characterized in that: The step of determining a position prediction frame of the target sign in the binocular view includes: Inputting the binocular view into a sign detection model to obtain a position prediction frame of the target sign in the binocular view output by the sign detection model; Wherein, the landmark detection model is constructed based on the YOLO model; The landmark detection model is further used to perform a pooling operation on the output feature map based on the regression branch in the Head module in the YOLO model to obtain a plurality of grid points; the pooling operation includes at least one of corner point pooling, middle point pooling and edge pooling; Based on the positions of the multiple grid points in the binocular view, a position prediction box of the target sign is determined.

3. The signboard positioning method according to claim 1, characterized in that: The step of cropping the binocular view based on the position prediction frame and performing stereo matching on the cropping result to obtain a second point cloud of the target sign includes: Adjusting the size of the position prediction frame, and cropping the binocular view based on the adjusted position prediction frame to obtain a visually adjusted image; The visually adjusted image is input into a stereo matching model to obtain a second point cloud of the target sign output by the stereo matching model.

4. The signboard positioning method according to claim 3, characterized in that: The adjusting the size of the position prediction frame and cropping the binocular view based on the adjusted position prediction frame to obtain a visually adjusted image includes: When the size of the position prediction frame is greater than or equal to the first size and smaller than the second size, the position prediction frame is expanded to the second size in the binocular view, and the binocular view is cropped based on the expanded position prediction frame to obtain the visual adjustment image; When the size of the position prediction box is smaller than the first size, the position prediction box is expanded to the first size in the binocular view, the binocular view is cropped based on the expanded position prediction box, and the cropped result is upsampled based on the second size, and the upsampled result is used as the visual adjustment image.

5. The signboard positioning method according to claim 3, characterized in that: The stereo matching model is built based on the ACVNet model; The cost space building block in the ACVNet model is also used to: Determine an image point with the same name corresponding to any pixel point in the first view in the binocular view in the second view in the binocular view; Determine a first disparity value of any pixel point in the binocular view; Determine a second disparity value of the same-name image point in the binocular view; Accordingly, the disparity prediction module in the ACVNet model is also used for: When the absolute value of the difference between the first disparity value and the second disparity value is greater than or equal to a preset threshold, the any pixel point is determined as an occlusion point, and the disparity value of the any pixel point is determined based on the disparity values ​​of the adjacent pixels of the any pixel point.

6. The signboard positioning method according to claim 1, characterized in that: The determining the relative position between the target sign and the traffic equipment based on the fusion result of the first point cloud and the second point cloud includes: Registering the first point cloud and the second point cloud, and determining Gaussian distribution features of the registered first point cloud and Gaussian distribution features of the registered second point cloud; Determining a fusion weight of the first point cloud and a fusion weight of the second point cloud based on a Gaussian distribution feature of the first point cloud and a Gaussian distribution feature of the second point cloud; Based on the fusion weight of the first point cloud and the fusion weight of the second point cloud, the first point cloud and the second point cloud are fused, and the relative position between the target sign and the traffic equipment is determined based on the fusion result.

7. The signboard positioning method according to claim 6, characterized in that: Before registering the first point cloud and the second point cloud, the method further includes: In the case where it is determined that the first point cloud and the second point cloud cannot be time-aligned, determining a current first point cloud and a current second point cloud; the difference between the acquisition time of the current second point cloud and the current first point cloud is the smallest; Determine the coordinate representation of the spatial point in the current first point cloud in the world coordinate system based on the coordinate representation of the spatial point in the current first point cloud in the radar coordinate system and the coordinate transformation relationship between the radar coordinate system where the current first point cloud is located and the world coordinate system; Determine the coordinate representation of the spatial point in the current first point cloud in the camera coordinate system based on the coordinate representation of the spatial point in the current first point cloud in the world coordinate system and the coordinate transformation relationship between the camera coordinate system where the current second point cloud is located and the world coordinate system; Aligning the current first point cloud and the current second point cloud based on the coordinate representation of the spatial points in the current first point cloud in the camera coordinate system and the coordinate representation of the spatial points in the current second point cloud in the camera coordinate system; The determining, based on the coordinate representation of the spatial point in the current first point cloud in the world coordinate system and the coordinate transformation relationship between the camera coordinate system where the current second point cloud is located and the world coordinate system, the coordinate representation of the spatial point in the current first point cloud in the camera coordinate system includes: ； in, is the coordinate representation of the spatial point in the current first point cloud in the camera coordinate system; for The rotation matrix between the camera coordinate system and the world coordinate system at the moment; for The translation matrix between the camera coordinate system and the world coordinate system at each moment; is the coordinate representation of the spatial point in the current first point cloud in the world coordinate system; for The rotation matrix between the radar coordinate system and the world coordinate system at this moment, for The inverse matrix of is the coordinate representation of the spatial point in the current first point cloud in the radar coordinate system; for The translation matrix between the radar coordinate system and the world coordinate system at the moment; Time is the collection time of the millimeter wave radar; The time is the acquisition time of the binocular camera.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the signboard positioning method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the signboard positioning method according to any one of claims 1 to 7 is implemented.

Images