Road side visual positioning method and device for traffic participants, equipment and medium

By determining the three-dimensional coordinates of traffic participants in the virtual world coordinate system in the two-dimensional image collected by the roadside camera, the problem of high sensor maintenance costs is solved, high-precision traffic participant positioning is achieved, reducing hardware costs and improving Positioning efficiency.

CN118941626APending Publication Date: 2024-11-12CHINA INTELLIGENT & CONNECTED VEHICLES (BEIJING) RES INST CO LTD

Patent Information

Application Number
CN202410755162.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, when positioning traffic participants at the roadside perspective, the sensor maintenance cost is too high and it is difficult to deploy on a large scale.

Method used

By acquiring the two-dimensional images collected by the roadside camera, the three-dimensional coordinates of the target traffic participant in the virtual world coordinate system are determined, and combined with the estimated height and similar triangle relationships, the traffic participant positioning is achieved, reducing hardware costs and improving positioning accuracy .

Benefits of technology

The cost of positioning traffic participants through roadside equipment is greatly reduced, and high-precision positioning results can be obtained using lower computing power, taking into account the operating costs and actual positioning performance of roadside infrastructure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941626A_ABST
    Figure CN118941626A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of road side visual positioning, and discloses a road side visual positioning method, device and equipment for traffic participants and a medium, and the method comprises the steps: obtaining two-dimensional coordinates of pixel points corresponding to the traffic participants in a two-dimensional image collected by a road side camera in an image coordinate system; obtaining three-dimensional coordinates of different reference points in a camera coordinate system according to the coordinates of the reference points corresponding to the pixel points and a camera internal reference matrix, and converting the three-dimensional coordinates into coordinates in a virtual world coordinate system; and determining the three-dimensional coordinate of each pixel point in the roadside image corresponding to the reference point in the virtual world coordinate system according to the proportional relation between the traffic participant and the roadside camera height, thereby obtaining the positioning result of the target traffic participant. According to the method, the two-dimensional image is acquired based on the roadside camera, the positioning results of different traffic participants in the virtual world coordinate system are obtained through derivable mathematical calculation, and the cost required for positioning the traffic participants under the roadside view angle is greatly reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of roadside visual positioning, and in particular to a roadside visual positioning method, device, equipment and medium for traffic participants. Background Art

[0002] The vehicle-road-cloud integrated system, also known as the vehicle-road-cloud integrated fusion control system / intelligent connected vehicle cloud control system, is a complex large system composed of vehicles and other traffic participants, roadside infrastructure, cloud control platform, related support platform, communication network and other parts. The vehicle-road-cloud integrated system can give full play to the advantages of road system, cloud and network, carry out the coordinated cooperation between vehicles, vehicles and roads, vehicles and clouds, and vehicles and people, and finally realize fully autonomous driving. Under complex road scenes and traffic conditions, the roadside sensing equipment used for real-time identification of moving targets on the road can maintain a high accuracy and recognition speed, that is, it can identify moving vehicles, pedestrians, non-motor vehicles on the road, and targets obscured from the vehicle's perspective. It is a prerequisite for the vehicle-road-cloud integrated system to assist intelligent connected vehicles in reasonably executing downstream tasks such as path planning and decision control.

[0003] In the vehicle-road-cloud integrated system, the roadside perception system is an important part of the roadside infrastructure and cloud control basic platform. The roadside perception system mainly includes roadside perception equipment (cameras, millimeter-wave radars, laser radars, etc.), roadside units, positioning equipment and roadside computing units, etc., which can provide information on moving targets on the road that are difficult to observe from different perspectives, and make up for the shortcomings of limited vehicle-side perception perspective, insufficient sight distance, and susceptibility to bad weather. Compared with vehicle-mounted sensors, roadside perception equipment has more options and is easy to achieve multi-directional and long-distance continuous deployment. It can transform "unknown" scenes into "known" scenes, thereby reducing the safety risks of single-vehicle intelligent driving, and further improving the intelligent perception capabilities of single vehicles through vehicle-road-cloud integrated collaboration. The roadside perception system can form a continuous, high-precision, blind-spot-free perception area for traffic participants, traffic events and traffic flow status within its coverage area, realize holographic perception of intersections and sections, local auxiliary positioning and other functions, and provide low-latency, high-granularity, and high-precision perception data empowerment services for users of connected vehicles, users of traffic management functional departments and other users in the industrial chain.

[0004] Among the existing technical solutions for perceiving traffic participants from the roadside perspective, a multi-sensor data fusion perception method is often used. This method requires high sensor installation and maintenance costs and is difficult to deploy on a large scale. Summary of the invention

[0005] In view of this, the present invention provides a roadside visual positioning method, device, equipment and medium for traffic participants to solve the problem of high sensor maintenance cost when positioning traffic participants from a roadside perspective.

[0006] In a first aspect, the present invention provides a roadside visual positioning method for traffic participants, the method comprising:

[0007] Acquire a two-dimensional image captured by a roadside camera, and determine the road type where the target traffic participant is located in the two-dimensional image and the two-dimensional coordinates of the pixel point corresponding to the target traffic participant;

[0008] Determine the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and the preset depth, and convert the three-dimensional image coordinates of the reference point into three-dimensional coordinates in the camera coordinate system according to the intrinsic parameter matrix of the roadside camera;

[0009] Determine the three-dimensional coordinates of the reference point in the virtual world coordinate system based on a first conversion matrix between the preset virtual world coordinate system and the camera coordinate system and in combination with the three-dimensional coordinates of the reference point in the camera coordinate system;

[0010] Determine an estimated height of the target traffic participant according to the road type, and determine the three-dimensional coordinates of a pixel point corresponding to the reference point in the virtual world coordinate system based on a similar triangle relationship between the estimated height and the actual height of the roadside camera and in combination with the three-dimensional coordinates of the reference point in the virtual world coordinate system;

[0011] According to the three-dimensional coordinates of each pixel in the pixel point set corresponding to the target traffic participant in the virtual world coordinate system, a first positioning result of the target traffic participant in the virtual world coordinate system is determined.

[0012] By identifying the traffic participants in the two-dimensional images collected by the roadside camera equipment, and determining the corresponding reference points based on their pixel points, and determining the three-dimensional coordinates of the reference points in the virtual world coordinate system, combined with the similar triangle relationship between the estimated height of the traffic participant in the virtual world coordinate system and the actual height of the roadside camera, the three-dimensional coordinates of the pixel points in the virtual world coordinate system are determined, thereby obtaining the positioning results of the traffic participants in the virtual world coordinate system. This solution only needs to collect the two-dimensional image through the roadside camera and deduce the mathematical relationship to obtain its positioning results in the virtual world coordinate system, which greatly reduces the cost required to locate the traffic participants through the roadside equipment. At the same time, a high-precision traffic participant positioning result can be obtained using lower computing power, taking into account the operating cost of the roadside infrastructure and the actual positioning performance.

[0013] In an optional implementation, determining the three-dimensional image coordinates of a reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and a preset depth includes:

[0014] The two-dimensional coordinates of the pixel point are determined as the x-axis coordinates and y-axis coordinates of the corresponding reference point in the image coordinate system, and the preset depth value is determined as the z-axis coordinate of the corresponding reference point in the image coordinate system.

[0015] By adding a preset depth to the two-dimensional coordinates of the pixel point as the value of the third dimension, the three-dimensional image coordinates of the reference point in the image coordinate system are formed to ensure the subsequent derivation of the three-dimensional coordinates in the virtual world coordinate system.

[0016] In an optional embodiment, the method further comprises:

[0017] According to the conversion matrix between the virtual world coordinate system and the vehicle-side coordinate system, the first positioning result of the target traffic participant in the virtual world coordinate system is converted into positioning information in the vehicle-side coordinate system, and the positioning information is sent to the vehicle-side.

[0018] Through the conversion matrix between the virtual world coordinate system and the vehicle-side coordinate system, the positioning results of traffic participants in the current scene can be converted into positioning in the vehicle-side coordinate system, thereby ensuring the vehicle-side's perception of traffic participants in the current scene and assisting the vehicle-side in making corresponding decisions.

[0019] In a second aspect, the present invention provides another roadside visual positioning method for traffic participants, the method comprising:

[0020] A traffic participant positioning model is constructed by using the roadside visual positioning method for traffic participants as described in the first aspect and its corresponding implementation manner;

[0021] The two-dimensional image to be detected collected by the roadside camera is input into the traffic participant positioning model to obtain the target positioning result.

[0022] By constructing a traffic participant positioning model, the two-dimensional images captured by the roadside camera can be recognized more quickly to obtain the corresponding positioning results.

[0023] In an optional embodiment, the method further includes:

[0024] Using laser radar to obtain a three-dimensional point cloud of traffic participants in the scene corresponding to the two-dimensional image to be detected collected by the roadside camera;

[0025] A second positioning result of the traffic participant is determined based on the three-dimensional point cloud.

[0026] The second positioning result of the traffic participants in the scene corresponding to the two-dimensional image is obtained by lidar, which can be compared with the target positioning result to evaluate the accuracy of the target positioning result.

[0027] In an optional embodiment, the method further includes:

[0028] Based on the second positioning result and the target positioning result, calculating the values ​​of various evaluation indicators of the traffic participant positioning model, the evaluation indicators including: multi-category average detection accuracy, average ground center similarity, average orientation angle similarity, average area similarity and average endpoint distance similarity;

[0029] The evaluation index is calculated according to a preset algorithm to obtain a positioning score of the traffic participant positioning model.

[0030] By calculating the information corresponding to the target positioning result and the second positioning result, evaluation indicators in multiple dimensions are obtained, and these evaluation indicators are calculated to obtain the positioning score, so that the positioning effect of the traffic participant positioning model can be evaluated more comprehensively.

[0031] In an optional embodiment, the method further includes:

[0032] According to the positioning score and the values ​​of the various evaluation indicators, the model parameters of the traffic participant positioning model are adjusted.

[0033] According to the positioning score and the specific values ​​of each evaluation index, the parameters of the positioning model can be adjusted to further improve the positioning accuracy of the traffic participant positioning model.

[0034] In a third aspect, the present invention provides a roadside visual positioning device for traffic participants, the device comprising:

[0035] A data acquisition module is used to acquire a two-dimensional image captured by a roadside camera, and determine the road type where the target traffic participant is located in the two-dimensional image and the two-dimensional coordinates of the pixel point corresponding to the target traffic participant;

[0036] A coordinate conversion module, for determining the three-dimensional image coordinates of a reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and a preset depth, and converting the three-dimensional image coordinates of the reference point into three-dimensional coordinates in a camera coordinate system according to an intrinsic parameter matrix of a roadside camera; determining the three-dimensional coordinates of the reference point in a virtual world coordinate system based on a first conversion matrix between a preset virtual world coordinate system and the camera coordinate system, in combination with the three-dimensional coordinates of the reference point in the camera coordinate system; determining an estimated height of a target traffic participant according to the road type, and determining the three-dimensional coordinates of a pixel point corresponding to the reference point in the virtual world coordinate system based on a similar triangle relationship between the estimated height and the actual height of the roadside camera, in combination with the three-dimensional coordinates of the reference point in the virtual world coordinate system;

[0037] The perception and positioning module is used to determine the first positioning result of the target traffic participant in the virtual world coordinate system according to the three-dimensional coordinates of each pixel in the pixel point set corresponding to the target traffic participant in the virtual world coordinate system.

[0038] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the roadside visual positioning method for traffic participants of any of the above aspects or any of its corresponding embodiments by executing the computer instructions.

[0039] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the roadside visual positioning method for traffic participants of any of the above aspects or any of its corresponding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a flow chart of a roadside visual positioning method for traffic participants according to an embodiment of the present invention;

[0042] Figure 2 This is a diagram showing an example of a scenario according to an embodiment of the present invention;

[0043] Figure 3 is a flow chart of another roadside visual positioning method for traffic participants according to an embodiment of the present invention;

[0044] Figure 4 is an example diagram of a roadside camera positioning process according to an embodiment of the present invention;

[0045] Figure 5 is a flow chart of another roadside visual positioning method for traffic participants according to an embodiment of the present invention;

[0046] Figure 6 is a flow chart of another roadside visual positioning method for traffic participants according to an embodiment of the present invention;

[0047] Figure 7 is an example diagram of a traffic participant positioning process according to an embodiment of the present invention;

[0048] Figure 8 is a diagram showing an example of a training data file structure according to an embodiment of the present invention;

[0049] Fig. 9 is a structural block diagram of a roadside visual positioning device for traffic participants according to an embodiment of the present invention;

[0050] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] The vehicle-road-cloud integrated system, also known as the vehicle-road-cloud integrated fusion control system / intelligent connected vehicle cloud control system, is a complex large system composed of vehicles and other traffic participants, roadside infrastructure, cloud control platform, related support platform, communication network and other parts. The vehicle-road-cloud integrated system can give full play to the advantages of road system, cloud and network, carry out the coordinated cooperation between vehicles, vehicles and roads, vehicles and clouds, and vehicles and people, and finally realize fully autonomous driving. Under complex road scenes and traffic conditions, the roadside sensing equipment used for real-time identification of moving targets on the road can maintain a high accuracy and recognition speed, that is, it can identify moving vehicles, pedestrians, non-motor vehicles on the road, and targets obscured from the vehicle's perspective. It is a prerequisite for the vehicle-road-cloud integrated system to assist intelligent connected vehicles in reasonably executing downstream tasks such as path planning and decision control.

[0053] In the vehicle-road-cloud integrated system, the roadside perception system is an important part of the roadside infrastructure and cloud control basic platform. The roadside perception system mainly includes roadside perception equipment (cameras, millimeter-wave radars, lidars, etc.), roadside units (RSUs), positioning equipment (RTK, GPS, etc.) and roadside computing units, etc., which can provide information on moving targets on the road that are difficult to be observed by the vehicle's own sensors from different perspectives, making up for the shortcomings of limited vehicle-side perception perspectives, insufficient sight distance, and susceptibility to bad weather. Compared with vehicle-mounted sensors, roadside perception equipment has more options and is easy to achieve multi-directional and long-distance continuous deployment. It can transform "unknown" scenes into "known" scenes, thereby reducing the safety risks of single-vehicle intelligent driving and further improving the intelligent perception capabilities of single vehicles through vehicle-road-cloud integrated collaboration. The roadside perception system can form a continuous, high-precision, blind-spot-free perception area for traffic participants, traffic events and traffic flow status within its coverage area, realize holographic perception of intersections and sections, local auxiliary positioning and other functions, and provide low-latency, high-granularity, and high-precision perception data empowerment services for users of connected vehicles, users of traffic management functional departments, and other users in the industrial chain.

[0054] Compared with the roadside 3D target perception method based on multi-sensor data fusion, the roadside 3D target perception method based on monocular camera requires lower computing power and calibration cost, and can make more full use of the large number of image acquisition devices deployed in the roadside infrastructure. Unlike the traditional 3D target perception task of autonomous driving, the roadside monocular 3D target perception task needs to solve three difficulties: First, since each roadside camera has different configurations, such as camera internal parameters, pitch angle and installation height, this greatly increases the difficulty of the roadside monocular 3D target perception task; second, the roadside camera can be installed on roadside poles of different shapes and is no longer located directly above the roof of the car, so the assumption that the camera optical axis is parallel to the ground is no longer valid, and the monocular 3D target perception method with this prior cannot be directly applied; finally, since the target perception range under the roadside perspective is larger and more traffic participants can be observed, the detection density and recognition difficulty of the roadside perception system will be further increased compared with the vehicle-side perception system.

[0055] In the existing technical solutions for perceiving traffic participants through the roadside perspective, a multi-sensor data fusion perception method is generally adopted. The sensor installation and maintenance costs of this type of method are high, and it is difficult to deploy on a large scale.

[0056] To this end, the present invention provides an embodiment of a roadside visual positioning method for traffic participants, which identifies traffic participants in two-dimensional images collected by roadside camera equipment, determines corresponding reference points based on their pixel points, and determines the three-dimensional coordinates of the reference points in the virtual world coordinate system, and combines the similar triangle relationship between the estimated height of the traffic participant in the virtual world coordinate system and the actual height of the roadside camera to determine the three-dimensional coordinates of the pixel points in the virtual world coordinate system, thereby obtaining the positioning results of the traffic participants in the virtual world coordinate system. This greatly reduces the cost required to locate traffic participants through roadside equipment, and at the same time, high-precision traffic participant positioning results can be obtained using lower computing power, taking into account the operating costs of roadside infrastructure and actual positioning performance.

[0057] According to an embodiment of the present invention, an embodiment of a roadside visual positioning method for traffic participants is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0058] In this embodiment, a roadside visual positioning method for traffic participants is provided, which can be used to position traffic participants through roadside equipment as mentioned above. Figure 1 is a flow chart of a roadside visual positioning method for traffic participants according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0059] Step S101, obtaining a two-dimensional image captured by a roadside camera, and determining the road type where the target traffic participant is located and the two-dimensional coordinates of the pixel points corresponding to the target traffic participant in the two-dimensional image.

[0060] The two-dimensional image of the current road scene is collected by a roadside camera, which can be a camera located on the traffic road. After collecting the image of the current scene, the traffic participants on the current road are identified, and the road type where the traffic participants are located is determined, such as three types of roads: motor vehicle lanes, non-motor vehicle lanes and sidewalks. At the same time, the two-dimensional coordinates of the pixel points corresponding to the traffic participants in the current two-dimensional image are determined. In an example, the two-dimensional coordinates of all pixel points of the traffic participants in the current two-dimensional image can be determined, or only the two-dimensional coordinates of some characteristic pixel points of some traffic participants can be extracted for subsequent positioning. In an example, there may be multiple traffic participants in the two-dimensional image, and these different traffic participants need to be labeled to ensure that the behavior results corresponding to different traffic participants can be determined later.

[0061] Step S102, determining the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and the preset depth, and converting the three-dimensional image coordinates of the reference point into three-dimensional coordinates in the camera coordinate system according to the intrinsic parameter matrix of the roadside camera.

[0062] It can be understood that the two-dimensional coordinates corresponding to the pixel points of the traffic participants are assigned a preset depth to determine the three-dimensional image coordinates of the reference point corresponding to the pixel point. The three-dimensional image coordinates can be understood as adding a depth dimension on the basis of the two-dimensional coordinates of the image. The depth value is the preset depth. Usually, for the convenience of calculation, the preset depth is a fixed value of 1. Then, the coordinates of the reference point at this preset depth are converted through the intrinsic parameter matrix of the camera to obtain the three-dimensional coordinates of the reference point in the camera coordinate system. The intrinsic parameter matrix of the camera can be determined according to the specific situation of the camera, and the intrinsic parameter matrix of different camera devices will also be different.

[0063] In an example, the above steps can be understood as constructing a special reference plane parallel to the image plane and with a fixed distance of 1. For each point p on the two-dimensional image plane i =(u,v), first select the corresponding reference point p in the reference plane r , the depth of the reference point is fixed to 1, that is, d r =1. Then, through the camera's intrinsic parameter matrix K, p r Projecting the two-dimensional reference plane into the camera coordinate system, we can obtain the three-dimensional coordinates of the reference point in the camera coordinate system. The specific calculation process is as follows:

[0064] P r c =K -1 d r [u,v,1] T =K -1 [u,v,1] T

[0065] Among them, P r c is the three-dimensional coordinate of the reference point in the camera coordinate system, d r is the default depth, and its value is 1.

[0066] Step S103, based on a first conversion matrix between a preset virtual world coordinate system and a camera coordinate system, and in combination with the three-dimensional coordinates of the reference point in the camera coordinate system, determine the three-dimensional coordinates of the reference point in the virtual world coordinate system.

[0067] It can be understood that a virtual world coordinate system is pre-constructed. For example, to protect data privacy, a virtual world coordinate system is constructed with the origin coinciding with the origin of the camera coordinate system, the x-axis is parallel to the ground plane and points to the left of the camera, the y-axis is perpendicular to the ground and downward, and the z-axis is parallel to the ground plane and points to the front of the camera. There is a transformation matrix between the virtual world coordinate system and the camera coordinate system corresponding to the roadside camera. The three-dimensional coordinates of the reference point in the camera coordinate system can be converted to the three-dimensional coordinates in the virtual world coordinate system through the transformation matrix. In an example, the conversion method is as follows:

[0068]

[0069] in, is the transformation matrix from the camera 3D coordinate system to the virtual world coordinate system, It is the three-dimensional coordinate corresponding to the reference point in the virtual world coordinate system.

[0070] Step S104, according to the road type, determines the estimated height of the target traffic participant, based on the similar triangle relationship between the estimated height and the actual height of the roadside camera, combined with the three-dimensional coordinates of the reference point in the virtual world coordinate system, determines the three-dimensional coordinates of the pixel point corresponding to the reference point in the virtual world coordinate system.

[0071] In step S101, the road type of the traffic participant is determined, and the estimated height of the traffic participant can be determined according to the road type. It can be understood that for the roadside image acquisition device installed at the intersection, its field of view must include three types of roads: motor vehicle lanes, non-motor vehicle lanes and sidewalks, so as to further determine whether the traffic participant is a motor vehicle, non-motor vehicle or pedestrian, each corresponding to a different estimated height.

[0072] For example, for a traffic participant representing a pedestrian in a two-dimensional image captured by a roadside camera, the two-dimensional image is the roadside image. The roadside image contains pixel points corresponding to the target traffic participant, and the corresponding pixel point set p = {p i}, i = 0, 1, ..., n, assuming that the actual height estimate corresponding to the i-th pixel point contained in the target in the passable area is h i (i.e., the height of a point in the image relative to the ground in the virtual three-dimensional coordinate system), then h iIt conforms to the normal distribution with a mean of 1.7 and a variance of 0.01. That is, when traffic participants are on the sidewalk, they are uniformly regarded as pedestrians, and the actual estimated heights corresponding to different pixel points containing pedestrian category information in the roadside image in the virtual world coordinate system conform to the normal distribution with a mean of 1.7 and a variance of 0.01. Different traffic participants in the roadside camera correspond to different passable areas. Different estimated heights are preset according to the actual road type in the passable area. For example, the main road is usually a car traffic participant, and the auxiliary road is usually a bicycle, electric vehicle and other traffic participants. The corresponding estimated height is determined according to the average height of these different traffic participants. It should be noted that the mean and variance mentioned here are only used to assist in understanding the method proposed in the patent of this invention. The specific mean and variance need to be set according to the actual situation to ensure the accuracy of the roadside visual positioning results.

[0073] At the same time, the actual installation height of the roadside camera can be obtained. There is a visual similar triangle relationship between the actual installation height and the estimated height of different traffic participants. For details, please refer to Figure 2 , Figure 2 This is a scene example diagram according to an embodiment of the present invention, combined with Figure 2 The virtual three-dimensional coordinate system, the camera coordinate system, and the subsequent vehicle-side coordinate system can be understood as examples. Among them, the origin of the virtual world coordinate system is the same as the camera coordinate system. In this coordinate system, taking pedestrians as an example, the actual three-dimensional coordinates of the pixels of traffic participants in the two-dimensional image in the virtual world coordinate system and the three-dimensional coordinates of the reference point in the virtual world coordinate system can form a similar triangle, and the specific proportional relationship is as follows:

[0074]

[0075] Where H is the height of the roadside camera, h i is the estimated height of the traffic participant corresponding to the pixel point, P i virt is the three-dimensional coordinate of pixel i in the virtual world coordinate system, is the y coordinate value of the three-dimensional coordinates of the reference point in the virtual world coordinate system, is the three-dimensional coordinate of the reference point in the virtual world coordinate system.

[0076] Step S105 , determining a first positioning result of the target traffic participant in the virtual world coordinate system according to the three-dimensional coordinates of each pixel in the pixel point set corresponding to the target traffic participant in the virtual world coordinate system.

[0077] According to the above steps, the three-dimensional coordinates of all the pixels of the traffic participants in the two-dimensional image in the virtual world coordinate system can be obtained. By summarizing these three-dimensional coordinates, the positioning result of the traffic participant in the virtual world coordinate system can be obtained. In one example, based on the three-dimensional projection coordinates of different pixels in the virtual world coordinate system, combined with the three-dimensional target perception model, several three-dimensional rectangular boxes can be predicted and output, each of which represents the three-dimensional outline, orientation angle and actual position of a certain traffic participant on the road plane.

[0078] The roadside visual positioning method for traffic participants provided in this embodiment identifies traffic participants in two-dimensional images captured by roadside camera equipment, determines corresponding reference points based on their pixels, and determines the three-dimensional coordinates of the reference points in the virtual world coordinate system, and determines the three-dimensional coordinates of the pixels in the virtual world coordinate system based on the similar triangle relationship between the estimated height of the traffic participant in the virtual world coordinate system and the actual height of the roadside camera, thereby obtaining the positioning result of the traffic participant in the virtual world coordinate system. This solution only needs to capture a two-dimensional image through a roadside camera and derive mathematical relationships to obtain its positioning result in the virtual world coordinate system, which greatly reduces the hardware cost required to locate traffic participants through roadside equipment, and can obtain high-precision traffic participant positioning results using lower computing power, taking into account both the operating cost of the roadside infrastructure and the actual positioning performance of the roadside perception model.

[0079] According to an embodiment of the present invention, another embodiment of a roadside visual positioning method for traffic participants is provided, which can be used to position traffic participants through roadside equipment as described above. Figure 3 is a flow chart of another roadside visual positioning method for traffic participants according to an embodiment of the present invention, such as Figure 3 As shown, the process includes the following steps:

[0080] Step S201, obtain a two-dimensional image captured by a roadside camera, determine the road type where the target traffic participant is located in the two-dimensional image and the two-dimensional coordinates of the pixel points corresponding to the target traffic participant. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0081] Step S202 determines the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and the preset depth, and converts the three-dimensional image coordinates of the reference point into three-dimensional coordinates in the camera coordinate system according to the intrinsic parameter matrix of the roadside camera.

[0082] Specifically, determining the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and the preset depth includes:

[0083] The two-dimensional coordinates of the pixel point are determined as the x-axis coordinates and y-axis coordinates of the corresponding reference point in the image coordinate system, and the preset depth value is determined as the z-axis coordinate of the corresponding reference point in the image coordinate system.

[0084] It can be understood that when determining the three-dimensional image coordinates of the reference point corresponding to the pixel point, the values ​​corresponding to the x-axis and y-axis of the reference point are the same as the two-dimensional image coordinates of the pixel point, and the value corresponding to the z-axis is the preset depth corresponding value, which is usually set to 1. By adding a preset depth to the two-dimensional coordinates of the pixel point as the value in the third dimension, the three-dimensional image coordinates of the reference point in the image coordinate system are formed to ensure the derivation of the three-dimensional coordinates in the subsequent virtual world coordinate system.

[0085] Step S203: Based on the first conversion matrix between the preset virtual world coordinate system and the camera coordinate system, the three-dimensional coordinates of the reference point in the camera coordinate system are combined to determine the three-dimensional coordinates of the reference point in the virtual world coordinate system. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0086] Step S204: Determine the estimated height of the target traffic participant according to the road type, and determine the 3D coordinates of the pixel point corresponding to the reference point in the roadside image in the virtual world coordinate system based on the similar triangle relationship between the estimated height and the actual height of the roadside camera and the 3D coordinates of the reference point in the virtual world coordinate system. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0087] Step S205: Determine the first positioning result of the target traffic participant in the virtual world coordinate system according to the three-dimensional coordinates of each pixel in the pixel point set corresponding to the target traffic participant in the virtual world coordinate system. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0088] Step S206, according to the conversion matrix between the virtual world coordinate system and the vehicle-side coordinate system, convert the first positioning result of the target traffic participant in the virtual world coordinate system into positioning information in the vehicle-side coordinate system, and send the positioning information to the vehicle-side.

[0089] It can be understood that there is a corresponding conversion matrix between the constructed virtual world coordinate system and the vehicle-side coordinate system corresponding to different vehicles. The positioning result of the target traffic participant in the virtual world coordinate system can be converted into positioning information in the vehicle-side coordinate system, and this positioning information is sent to the target vehicle-side. After the vehicle-side obtains the positioning status of the traffic participants in the current scene, it can provide a basis for downstream tasks such as planning, decision-making, and control on the vehicle-side based on the received positioning results.

[0090] Among them, the method of obtaining the conversion matrix between the virtual world coordinate system and the vehicle-side coordinate system can be based on the conversion matrix between the camera coordinate system and the vehicle-side coordinate system Transformation matrix between the virtual world coordinate system and the camera coordinate system Calculated, because usually, the transformation matrix between the camera coordinate system and the vehicle coordinate system is known, and the transformation matrix between the virtual world coordinate system and the camera coordinate system is It is determined after constructing the virtual world coordinate system, so based on these two matrices we can get:

[0091]

[0092] in, It is the transformation matrix between the virtual world coordinate system and the vehicle-side coordinate system, thus combining the three-dimensional coordinates P of the pixel point in the virtual world coordinate system mentioned above i virt , we can get the pixel p i Coordinate P in the vehicle-side coordinate system i veh for:

[0093]

[0094] In summary, the calculation method of projecting a two-dimensional image to the three-dimensional coordinates in the vehicle-side coordinate system can be summarized as follows:

[0095]

[0096] With the projection coordinates of the roadside two-dimensional image features in the vehicle-side three-dimensional coordinate system, it can provide a basis for downstream tasks such as planning, decision-making and control of intelligent connected vehicles. Through the conversion matrix between the virtual world coordinate system and the vehicle-side coordinate system, the positioning results of traffic participants from the roadside perspective can be converted into the perception results in the vehicle-side coordinate system, thereby assisting the vehicle side to make corresponding decisions and ensuring the perception effect of intelligent connected vehicles on traffic participants in the current scene.

[0097] In order to assist in understanding the contents of the above-mentioned embodiments of the present invention, an exemplary embodiment is provided to assist in understanding. Figure 4This is an example diagram of a roadside camera positioning process according to an embodiment of the present invention. As shown in the figure, a virtual three-dimensional coordinate system is first constructed, and then the conversion matrix between the camera coordinate system and the virtual three-dimensional coordinate system is determined based on the conversion relationship between the coordinate systems. A reference plane parallel to the image collected by the roadside equipment and with a depth of 1 is set, and the points in the reference plane are projected to the camera coordinate system according to the intrinsic parameter matrix of the roadside camera, and then the reference points in the camera coordinate system are converted to the virtual three-dimensional coordinate system according to the conversion matrix. Determine the height estimate of the specific traffic participant corresponding to different areas within the roadside perception range, and combine the installation height of the roadside camera and the y coordinate of the reference point in the virtual three-dimensional coordinate system to determine the three-dimensional coordinates of the two-dimensional pixel points in the roadside image in the virtual world coordinate system according to the principle of similar triangles. Based on the above method, a three-dimensional target detection algorithm based on a roadside monocular camera is determined to perform subsequent perception result evaluation in combination with the information collected by the roadside equipment.

[0098] The projection method of two-dimensional pixel points to three-dimensional space proposed in the embodiment of the present invention is derived based on mathematical theory in all steps, and has reliability and retrospective properties, which can improve the accuracy of pure visual perception of the roadside while improving the interpretability of the roadside visual perception model. In addition, the sensors required for the method are inexpensive, and high-precision perception results of traffic participants can be obtained using relatively low computing power, taking into account both the operating cost of the roadside infrastructure and the actual perception performance.

[0099] The roadside visual positioning method for traffic participants of the present invention estimates the height estimation value within a specific range according to the passable areas of different traffic participants, which is in line with the actual operating conditions of most intersections. However, in order to detect traffic participants in intersections with irregular shapes (such as the presence of an elevated bridge above the intersection), it is still necessary to combine V2X communication, vehicle-road collaborative perception, visual pre-training large models, vehicle-road-cloud integrated systems and other technologies to further improve the accuracy and efficiency of depth estimation of traffic participants under pure visual conditions on the roadside.

[0100] According to an embodiment of the present invention, another embodiment of a roadside visual positioning method for traffic participants is provided, which can be used to position traffic participants through the roadside equipment mentioned above. Figure 5 FIG. 4 is a flow chart of another roadside visual positioning method for traffic participants according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0101] Step S301: construct a traffic participant positioning model using the roadside visual positioning method for traffic participants of any embodiment.

[0102] It can be understood that, based on the principles corresponding to the above method embodiments, a model for locating traffic participants is constructed, and the input image can be directly processed by this model to obtain the corresponding traffic participant positioning result.

[0103] Step S302: input the two-dimensional image to be detected collected by the roadside camera into the traffic participant positioning model to obtain the target positioning result.

[0104] After constructing the traffic participant positioning model, the two-dimensional image captured by the roadside camera can be directly input to obtain the positioning results of the traffic participants in the corresponding image.

[0105] In one example, the constructed traffic participant positioning model pre-processes the input two-dimensional image, identifies and annotates the traffic participants therein, extracts the characteristic pixel points of the traffic participants, and determines the positioning results of the traffic participants in the image according to the roadside visual positioning method for traffic participants described in the above method embodiment.

[0106] According to an embodiment of the present invention, another embodiment of a roadside visual positioning method for traffic participants is provided, which can be used to position traffic participants through the roadside equipment mentioned above. Figure 6 FIG. 1 is a flow chart of another roadside visual positioning method for traffic participants according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:

[0107] Step S401: Use the roadside visual positioning method for traffic participants in any embodiment to build a traffic participant positioning model. Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.

[0108] Step S402: Input the two-dimensional image to be detected collected by the roadside camera into the traffic participant positioning model to obtain the target positioning result. Figure 3 Step S302 of the illustrated embodiment will not be described in detail here.

[0109] Step S403: Use laser radar to obtain a three-dimensional point cloud of the traffic participant in the scene corresponding to the two-dimensional image to be detected captured by the roadside camera; and determine a second positioning result of the traffic participant based on the three-dimensional point cloud.

[0110] It can be understood that in addition to the roadside camera, a laser radar can also be arranged in the roadside acquisition equipment. The laser radar is used to obtain the three-dimensional point cloud of the traffic participants in the scene corresponding to the image collected by the roadside camera, and the second positioning result of the traffic participants in the scene is determined by the three-dimensional point cloud. The positioning result determined by the laser radar is more accurate. Since the parameters of the traffic participant positioning model based on the roadside image need to be adjusted, the positioning effect is not good. At this time, the positioning result of the traffic participant can be determined in combination with the positioning result of the laser radar, and the positioning effect of the traffic participant positioning model can be judged according to the corresponding positioning result of the laser radar in the future, and the model parameters can be adjusted to further improve its positioning effect.

[0111] Step S404, based on the second positioning result and the target positioning result, calculate the values ​​of various evaluation indicators of the traffic participant positioning model, the evaluation indicators include: multi-category average detection accuracy, average ground center similarity, average orientation angle similarity, average area similarity and average endpoint distance similarity; calculate the evaluation indicators according to a preset algorithm to obtain the positioning score of the traffic participant positioning model.

[0112] After obtaining the target positioning result according to the traffic participant positioning model and the second positioning result according to the lidar, various evaluation indicators of the traffic participant positioning model can be calculated according to the specific data in the two positioning results to characterize the positioning effect of the model.

[0113] Specifically, the evaluation indicators include: multi-category average detection accuracy, average ground center similarity, average orientation angle similarity, average area similarity and average endpoint distance similarity.

[0114] In one example, when evaluating the positioning effect of the traffic participant positioning model, the multi-category mean average detection accuracy (mAP) is used as the primary evaluation indicator of roadside visual perception accuracy. The specific calculation process is to first calculate the average detection accuracy (AP) of a single category, and the calculation formula is as follows:

[0115]

[0116] Where R is a preset value, for example, 40, which means that the PR curve (Presession-Recall) is divided into 40 equal parts along the horizontal axis; Represents the accuracy under a certain recall rate threshold, r∈{1 / 40,2 / 40,...,1}. After obtaining the average detection accuracy AP of a certain category, the AP of all detected categories is averaged to obtain the multi-category average detection accuracy mAP. The invention only considers the average detection accuracy of three types of traffic participants: motor vehicles, non-motor vehicles, and pedestrians.

[0117] Average Ground Center Similarity (ACS) represents the distance between different objects on the road plane and is used to assess the risk of collision between traffic participants. After projecting the predicted 3D bounding box center point onto the ground plane, the Euclidean distance is further calculated:

[0118]

[0119] Where P represents the set of all traffic participants with positive prediction values, Represents the 2-norm of the center point of the true 3D bounding box, Represents the Euclidean distance between the projection points of the predicted box center s and the true value box center c on the road plane.

[0120] Average Orientation Similarity (AOS) can measure the difference between the estimated orientation angle and the true value of each 3D bounding box. In order to simplify the evaluation process of orientation angle similarity, the front and rear of the vehicle are not distinguished, that is, the bounding box is regarded as a 3D box that does not distinguish the front and back relationship. The angle multiple formula of trigonometric function can be obtained:

[0121]

[0122] In the above formula represents the difference between the orientation angle of the predicted box s and the orientation angle of the corresponding true value box. 2 ) has a time complexity greater than the constant order O(1), and the upper bound of the trigonometric function cos(n) is 1, so the time complexity of O(cos(n)) can also be expressed as a constant order complexity O(1). Based on the above analysis, in order to further simplify the calculation process of AOS, the formula can be expressed as:

[0123]

[0124] Average Area Similarity (AAS) measures the difference between the area occupied by the predicted box and the corresponding true value on the ground plane. When the difference between the two is greater than the area of ​​the true value bounding box, the predicted bounding box will be considered a negative prediction value:

[0125]

[0126] In the formula Represents the absolute value of the difference between the area occupied by the prediction box s and the true value box on the ground plane, Represents the actual area occupied by the truth box corresponding to the predicted box s on the ground plane.

[0127] Average Ground Points Distance and Similarity (APDS) measures the average distance between the four endpoints in the true 3D bounding box and the predicted 3D bounding box:

[0128]

[0129] Where v i and They respectively represent the coordinates of the i-th predicted point and the true point of the 3D bounding box corresponding to the s-th traffic participant. Here, only the four endpoints located on the road plane are considered.

[0130] After obtaining the specific values ​​of the above evaluation indicators, these evaluation indicators are weighted averaged to obtain the final positioning score of the traffic participant positioning model, which is used to represent its positioning effect. The specific scoring calculation formula is as follows:

[0131]

[0132] CIPS is the positioning score. The values ​​of λ1 and λ2 can be adjusted appropriately according to different downstream tasks such as path planning, decision control, and traffic control. For example, λ1 can be set to 8.5 and λ2 can be set to 1.5 according to actual conditions.

[0133] By calculating the information corresponding to the target positioning result and the second positioning result, we can obtain evaluation indicators in multiple dimensions, and calculate these evaluation indicators to obtain the positioning score, so as to more comprehensively evaluate the positioning effect of the traffic participant positioning model. According to the measured area and target category from the roadside perspective, we further improve and expand the evaluation indicators of traffic participant perception results applicable to roadside image acquisition equipment from the dimensions of accuracy, orientation angle, occupied area, edge error, etc., so as to more comprehensively evaluate the roadside perception method based on pure vision.

[0134] Step S405, adjusting the model parameters of the traffic participant positioning model according to the positioning score and the values ​​of each evaluation index.

[0135] After obtaining the score of the traffic participant positioning model, when the score is too low, the internal parameters of the model can be adjusted accordingly according to the specific circumstances of each indicator to improve the positioning effect of the model, thereby ensuring the positioning accuracy of the traffic participant positioning model.

[0136] The embodiment of the present invention constructs a traffic participant positioning model, and combines the positioning results corresponding to the three-dimensional point cloud collected by the laser radar to judge the traffic participant positioning model, obtain evaluation indicators in multiple dimensions, and calculate these evaluation indicators to obtain a positioning score. The positioning effect of the traffic participant positioning model can be evaluated more comprehensively, and the parameters of the model can be adjusted according to the evaluation results, thereby further ensuring the positioning accuracy of the model.

[0137] In order to more comprehensively evaluate the impact of pure visual perception results of the roadside on the vehicle-road-cloud integrated system, based on the invention, combined with downstream tasks such as target tracking, high-precision mapping, occupancy prediction, and path planning, we can design and improve perception result evaluation indicators that are more in line with specific task scenarios. This evaluation system is not only suitable for pure visual perception algorithm evaluation, but can also be used to build a roadside truth value system based on multi-sensor fusion.

[0138] In order to facilitate understanding of the above embodiments, an exemplary embodiment is provided to assist understanding. Figure 7 This is an example diagram of a traffic participant positioning process according to an embodiment of the present invention. As shown in the figure, it is necessary to first deploy roadside perception sensors, that is, to place perception devices such as laser radar, camera, carrier phase difference (RTK) on fixed or mobile poles on the roadside, adjust the sensor position and angle, ensure that the perception angles of the roadside camera and the roadside laser radar overlap, and facilitate the acquisition of three-dimensional point clouds and two-dimensional images in the same scene.

[0139] Then, these sensors are spatially synchronized, and the cameras and lidars installed on the roadside poles, the vehicle-mounted lidars, and the RTKs from the two perspectives of the vehicle-side roadside are jointly calibrated to make the video frames and point clouds overlap in time and space in the geodetic coordinate system. Through the joint calibration and calibration of multiple sensors, the conversion relationship between the lidar, the virtual three-dimensional coordinate system, and the camera coordinate system is determined, and the ground plane equation and the camera internal parameters can also be obtained together. And the sensors are time synchronized, that is, all roadside sensors are connected to the same network time protocol server to complete the time synchronization. When conditions are lacking, GPS hardware timing or network cable direct connection can be used to calculate the real-time time difference. After starting all sensors, set the data acquisition parameters, such as save format, segment interval, storage location, etc., and the data acquisition process can be checked through visualization tools.

[0140] The above steps are the preliminary preparations for the positioning of the traffic participant positioning model. Then open the post-processing tool, load the processing parameters and the image truth value annotation algorithm, pre-process and automatically annotate the original data of the roadside image, and use the post-fusion algorithm based on the real positioning information of the vehicle-mounted RTK and vehicle-mounted laser radar to further refine the three-dimensional annotations of different traffic participants in the roadside image. It can be understood as establishing an association between the traffic participants in the image and the positioning results obtained by the laser radar, and performing true value annotation. Then the perception data collected in real time by the roadside camera is input into the roadside visual perception model, that is, the traffic participant positioning model mentioned above. The model begins to pre-process the original roadside image. The roadside visual perception model uses mainstream image feature extraction backbone networks such as convolutional neural networks and vision-transformer to downsample the original image and obtain the intermediate features of the roadside image. The purpose is to reduce the training time and computational cost of the roadside visual perception model.

[0141] According to the passable areas divided by the traffic management department on the road plane for traffic participants such as pedestrians, motor vehicles and non-motor vehicles, the passable areas in the roadside image are proportionally copied and obtained from a bird's-eye view, and the actual height averages of specific traffic participants are preset in different areas. Then, based on the similar triangle relationship between the actual height averages and the actual heights of the roadside cameras, the roadside visual positioning method for traffic participants mentioned in the above embodiment is performed to obtain the three-dimensional projection coordinates of each pixel point in the roadside image in the virtual three-dimensional coordinate system.

[0142] The projected three-dimensional pixel points are regarded as the pseudo point cloud of the LiDAR. According to the three-dimensional target perception model paradigm based on the LiDAR point cloud, the voxel pooling operation of the point cloud is performed on the projected pixel points. The pseudo point cloud after voxel pooling is input into the three-dimensional bounding box generation module, and the perception result of the roadside visual perception model is output. The result mainly includes the coordinate values ​​of each three-dimensional boundary prediction box in the virtual three-dimensional coordinate system, which is convenient for measuring and comparing different model evaluation indicators.

[0143] The perception results corresponding to each traffic participant include the predicted 3D bounding box length (height), width (width), height (length), orientation angle (i.e. the rotation angle of the moving object around the Z axis) and the predicted 3D coordinates (x, y, z) of the center point of the 3D bounding box. Based on these seven types of data, combined with the five sub-item evaluation indicators proposed in the patent of this invention, the final roadside visual perception scores of different roadside visual perception models can be obtained.

[0144] In an illustrative example, a complete roadside perception system, that is, a system that uses roadside cameras to perceive and locate traffic participants in the current scene, the construction of the system mainly includes three parts, namely algorithm design, model training and result evaluation. The above embodiment mainly mentions the design of the algorithm and the evaluation of the positioning results. An illustrative example is used to illustrate the training process of the system. First, the roadside perception data is preprocessed to facilitate the algorithm training of the model. In order to speed up the model training and convergence, the original data format in the roadside perception dataset should be preprocessed and converted into a format similar to the mainstream public dataset, so as to facilitate the evaluation and comparison of the performance gap between different roadside visual perception models.

[0145] For example, the file format can be json format file and pkl format, Figure 8 This is an example diagram of a training data file structure according to an embodiment of the present invention. Both data files in two formats contain the real annotation and index information required for training and evaluating the model. The difference is that the files in the coco.json format in the training data are mainly used to train the roadside visual perception model, while the files in the pkl format are used to train the model using the lidar point cloud data. The ImageSets folder in the training data file contains the txt format files required to divide the data set. Generally, the roadside perception data set is divided into a training set, a validation set, and a test set in a ratio of 5:2:3. The lable folder contains the real annotation information of each frame of the image and the lidar point cloud, all stored in the json file format. Among them, the real annotations on the image are generated in the camera coordinate system, and the real annotations in the lidar point cloud frame are converted to the virtual lidar coordinate system (similar to the vehicle-side lidar coordinate system) to be generated so as to be fused with the vehicle-side perception information.

[0146] The image folder and velodyne folder contain the original camera images and original lidar point cloud files collected by the sensor, respectively. During the data preprocessing process, in addition to integrating the information of the real annotation files, the original jpg format images will be converted to png format; similarly, the pcd format lidar point cloud files will also be converted to bin format. The converted raw data and its corresponding sensor calibration files will be divided into files according to the data set in the ImageSets folder and saved in the training set (training) and test set (testing) folders respectively.

[0147] Then set the main training parameters of the roadside visual perception model. The detection range of each frame image and lidar point cloud used in the experiment will be limited to a rectangular frame of fixed size. Specifically, the detection distance along the X-axis is [0,100m], that is, the detection range of targets within 100m in front of the roadside lidar installation location; the detection distance along the Y-axis is [-40m,40m], that is, the detection range in the direction perpendicular to the X-axis at the road intersection; the detection distance along the Z-axis is [-3m,1m], so that the points emitted by the lidar can successfully capture the reflection intensity information of vehicles and other traffic targets on the ground, that is, objects within three meters below the roadside lidar will be regarded as detection targets.

[0148] The step of "projecting 3D points as point clouds and performing voxel pooling operations on point clouds" mentioned in the above content requires setting the relevant parameters of the voxel grid. We preset the length, width and height of each voxel grid to [0.05m, 0.05m, 0.1m]. Correspondingly, the number of voxels along the X-axis, Y-axis and Z-axis in each frame of point cloud is [1600, 2000, 40] respectively. It is worth noting that due to the influence of algorithm accuracy, in the process of multiple downsampling, too small numbers will be processed as 0, which will seriously affect the training process of the model and even cause the program to enter the Not a Number (NaN) trap. Therefore, in the actual program implementation process, the dimension along the Z axis will be filled with one more dimension before downsampling, that is, the number of voxels along the Z axis is 41, not 40. Roadside visual perception models generally use sparse convolution to extract intermediate features of images, and the dimensions of sparse convolution are [41, 1600, 2000].

[0149] Data enhancement is performed on the projection points of the roadside image in three-dimensional space to improve the robustness of the model. In the perception task of vehicle-road cooperative autonomous driving, the labor cost of collecting and annotating point cloud data is relatively high. Existing data can be fully utilized to expand the diversity of training data. The online optimization strategy based on the truth library is to first generate the truth library offline based on the existing training data. The generated truth data files will be saved in the "kitti_gt_database" folder in the above file structure example diagram. During the training process, some true value targets are randomly sampled from the truth library. On the premise that there is no overlap between the sampled target and all the true value targets in the current frame, the point cloud corresponding to the sampled target is pasted into the current training frame to synthesize a new point cloud frame, thereby improving the generalization ability of the model. For each sampled target, the maximum number of point clouds that can be pasted in each selection process can be freely defined. In the roadside visual perception model, the maximum number of sampling points corresponding to each true value target of the Car category in the truth library is 12. Similarly, for the true value targets of the Pedestrian and Cyclist categories, the corresponding maximum number of sampling points is 10 to improve the robustness of the model for small target perception.

[0150] Global random flipping, horizontal random rotation, scene scaling, and position migration in three-dimensional space are also commonly used data enhancement methods for LiDAR point clouds. Random flipping and random rotation will only change the spatial position information of the point cloud, and will not change the shape of the point cloud. For example, after a car in a certain frame of the point cloud is flipped, its shape is still a car. In this embodiment, the probability of global flipping is uniformly set to 0.5, that is, during the model training and verification process, each frame of the point cloud has a 50% probability of being flipped in a direction parallel to the ground. Similarly, the probability of rotating the object and its corresponding true bounding box in the horizontal direction from a bird's-eye view is also set to 50%, and the angle of each rotation of the scene and the true value target is a random number that conforms to the uniform distribution in the interval [-π / 4,π / 4]. Scene scaling is to scale the point cloud within a limited range with the origin as the center, but the scaling ratio should be controlled within a smaller range, because the roadside perception model based on two-dimensional-three-dimensional visual projection is based on voxel grids for feature extraction, and a large scaling range may affect this process. Therefore, the ratio of random scaling of the scene is controlled between 0.95 and 1.05. For the position migration operation, the probability of the true value object migrating in the three directions of the X-axis, Y-axis and Z-axis is set to conform to the normal distribution with a mean of 0 and a variance of [0.2, 0.2, 0.2]. After a series of data enhancement operations, the robustness of the roadside visual perception model and its generalization ability in different scenarios can be effectively improved. Compared with the model without data enhancement, its perception accuracy will also be improved to a certain extent.

[0151] Reasonable model training strategy and model verification strategy are set to further improve the accuracy of roadside visual perception. The experimental results presented in this embodiment are based on the actual roadside visual perception data set and are run on a self-owned server, including the entire process of training and verification. In order to protect data privacy, the results presented in this embodiment are fine-tuned based on the actual results. Considering the maintenance cost and operating life of roadside perception facilities, it is not practical to deploy multiple computing units (such as GPUs) in a single roadside unit. Therefore, all experiments in this embodiment are run on a single GPU. In order to achieve the best perception performance in the shortest possible training time and with limited computing resources, under the maximum batch size (that is, the number of data samples captured in one training, the size affects the number of training models in each round, and the appropriate batch size can make the direction of gradient descent more accurate), in order to fairly compare the experimental results, the number of training rounds for all models is unified to 80 rounds. The evaluation process of all models (including the three-dimensional data enhancement method) is the same as their corresponding training process.

[0152] In the training process of all models, AdamW is used as the main optimizer to accelerate the convergence of the model, and its weight decay coefficient is set to 0.01. The initial learning rate used to train the model is set to 0.001 and eventually decays to 0.0000001. The change process of the learning rate follows a cyclic schedule to prevent the model from falling into a local optimal solution. All models do not use any post-processing operations, such as the common non-maximal suppression (NMS), so there is no need to set any parameters for post-processing operations.

[0153] After the roadside visual perception model is trained, all models, including the traffic participant positioning model constructed based on the traffic positioning method mentioned in the embodiment of the present invention and other existing roadside visual perception models, are fed with randomly selected roadside visual perception data to obtain the model accuracy and overall evaluation score. The experimental results of all models are shown in the following table

[0154]

[0155] All data in the table are in percentage, where IoU represents the accuracy under different intersection over union (IoU) thresholds. It is worth noting that model 4 uses the roadside visual projection method proposed in the patent of this invention, that is, the roadside visual positioning method for traffic participants in the above embodiment, and its perception accuracy and overall evaluation score are significantly higher than those of the other three models.

[0156] In this embodiment, a traffic participant positioning device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0157] This embodiment provides a traffic participant positioning device, such as Fig. 9 As shown, including:

[0158] The data acquisition module 501 is used to acquire a two-dimensional image collected by a roadside camera, and determine the road type where the target traffic participant is located in the two-dimensional image and the two-dimensional coordinates of the pixel point corresponding to the target traffic participant;

[0159] A coordinate conversion module 502 is used to determine the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and the preset depth, and convert the three-dimensional image coordinates of the reference point into three-dimensional coordinates in the camera coordinate system according to the intrinsic parameter matrix of the roadside camera; based on a first conversion matrix between a preset virtual world coordinate system and the camera coordinate system, in combination with the three-dimensional coordinates of the reference point in the camera coordinate system, determine the three-dimensional coordinates of the reference point in the virtual world coordinate system; according to the road type, determine the estimated height of the target traffic participant, and based on the similar triangle relationship between the estimated height and the actual height of the roadside camera, in combination with the three-dimensional coordinates of the reference point in the virtual world coordinate system, determine the three-dimensional coordinates of the pixel point corresponding to the reference point in the virtual world coordinate system;

[0160] The perception positioning module 503 is used to determine the first positioning result of the target traffic participant in the virtual world coordinate system according to the three-dimensional coordinates of each pixel in the pixel point set corresponding to the target traffic participant in the virtual world coordinate system.

[0161] In some optional implementations, determining the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and a preset depth includes:

[0162] The two-dimensional coordinates of the pixel point are determined as the x-axis coordinates and y-axis coordinates of the corresponding reference point in the image coordinate system, and the preset depth value is determined as the z-axis coordinate of the corresponding reference point in the image coordinate system.

[0163] In some optional embodiments, the device also includes a vehicle-side perception and positioning module, which is used to convert the first positioning result of the target traffic participant in the virtual world coordinate system into positioning information in the vehicle-side coordinate system according to the transformation matrix between the virtual world coordinate system and the vehicle-side coordinate system, and send the positioning information to the vehicle-side.

[0164] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0165] The traffic participant positioning device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0166] The embodiment of the present invention also provides a computer device having the above Fig. 9 A road user location device is shown.

[0167] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0168] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0169] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0170] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0172] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0173] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0174] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0175] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A roadside visual positioning method for traffic participants, It is characterized in that The method comprises: Acquire a two-dimensional image captured by a roadside camera, and determine the road type where the target traffic participant is located in the two-dimensional image and the two-dimensional coordinates of the pixel point corresponding to the target traffic participant; Determine the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and the preset depth, and convert the three-dimensional image coordinates of the reference point into three-dimensional coordinates in the camera coordinate system according to the intrinsic parameter matrix of the roadside camera; Determine the three-dimensional coordinates of the reference point in the virtual world coordinate system based on a first conversion matrix between the preset virtual world coordinate system and the camera coordinate system and in combination with the three-dimensional coordinates of the reference point in the camera coordinate system; Determine an estimated height of the target traffic participant according to the road type, and determine the three-dimensional coordinates of a pixel point corresponding to the reference point in the virtual world coordinate system based on a similar triangle relationship between the estimated height and the actual height of the roadside camera and in combination with the three-dimensional coordinates of the reference point in the virtual world coordinate system; According to the three-dimensional coordinates of each pixel in the pixel point set corresponding to the target traffic participant in the virtual world coordinate system, a first positioning result of the target traffic participant in the virtual world coordinate system is determined.

2. The method according to claim 1, It is characterized in that The step of determining the three-dimensional image coordinates of the reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and the preset depth includes: The two-dimensional coordinates of the pixel point are determined as the x-axis coordinates and y-axis coordinates of the corresponding reference point in the image coordinate system, and the preset depth value is determined as the z-axis coordinate of the corresponding reference point in the image coordinate system.

3. The method according to claim 1, It is characterized in that The method further comprises: According to the conversion matrix between the virtual world coordinate system and the vehicle-side coordinate system, the first positioning result of the target traffic participant in the virtual world coordinate system is converted into positioning information in the vehicle-side coordinate system, and the positioning information is sent to the vehicle-side.

4. A roadside visual positioning method for traffic participants, It is characterized in that The method comprises: A traffic participant positioning model is constructed by using the roadside visual positioning method for traffic participants as described in any one of claims 1 to 3; The two-dimensional image to be detected collected by the roadside camera is input into the traffic participant positioning model to obtain the target positioning result.

5. The method according to claim 4, It is characterized in that The method further comprises: Using laser radar to obtain a three-dimensional point cloud of traffic participants in the scene corresponding to the two-dimensional image to be detected collected by the roadside camera; A second positioning result of the traffic participant is determined based on the three-dimensional point cloud.

6. The method according to claim 5, It is characterized in that The method further comprises: Based on the second positioning result and the target positioning result, calculating the values ​​of various evaluation indicators of the traffic participant positioning model, the evaluation indicators including: multi-category average detection accuracy, average ground center similarity, average orientation angle similarity, average area similarity and average endpoint distance similarity; The evaluation index is calculated according to a preset algorithm to obtain a positioning score of the traffic participant positioning model.

7. The method according to claim 6, It is characterized in that The method further comprises: According to the positioning score and the values ​​of the various evaluation indicators, the model parameters of the traffic participant positioning model are adjusted.

8. A roadside visual positioning device for traffic participants, It is characterized in that The device comprises: A data acquisition module is used to acquire a two-dimensional image captured by a roadside camera, and determine the road type where the target traffic participant is located in the two-dimensional image and the two-dimensional coordinates of the pixel point corresponding to the target traffic participant; A coordinate conversion module, for determining the three-dimensional image coordinates of a reference point corresponding to the pixel point according to the two-dimensional coordinates of the pixel point and a preset depth, and converting the three-dimensional image coordinates of the reference point into three-dimensional coordinates in a camera coordinate system according to an intrinsic parameter matrix of a roadside camera; determining the three-dimensional coordinates of the reference point in a virtual world coordinate system based on a first conversion matrix between a preset virtual world coordinate system and the camera coordinate system, in combination with the three-dimensional coordinates of the reference point in the camera coordinate system; determining an estimated height of a target traffic participant according to the road type, and determining the three-dimensional coordinates of a pixel point corresponding to the reference point in the virtual world coordinate system based on a similar triangle relationship between the estimated height and the actual height of the roadside camera, in combination with the three-dimensional coordinates of the reference point in the virtual world coordinate system; The perception and positioning module is used to determine the first positioning result of the target traffic participant in the virtual world coordinate system according to the three-dimensional coordinates of each pixel in the pixel point set corresponding to the target traffic participant in the virtual world coordinate system.

9. A computer device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the roadside visual positioning method for traffic participants as described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the roadside visual positioning method for traffic participants according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Traffic flow element sensing and positioning method based on road-end monocular camera and application of traffic flow element sensing and positioning method

    CN114998849A

  • Vehicle positioning method, device and equipment and readable storage medium

    CN117152253A

Cited By

  • Vehicle-road cloud cooperative positioning method and device based on road live-action three-dimensional data

    CN119152477A

  • Vehicle-road-cloud collaborative positioning method and device based on road real-scene three-dimensional data

    CN119152477B

  • Cloud road collaborative learning method of traffic multi-mode large language model

    CN119692471A

  • Material distribution pattern detection method and device based on array type interdigital capacitors

    CN120427699A

  • Material distribution pattern detection method and device based on arrayed interdigital capacitor

    CN120427699B