A method for fusing and positioning multiple cameras at the field end

Abstract

The present invention discloses a method for fusing and positioning multiple cameras at the field end, including: obtaining images of the area to be positioned recognized by multiple cameras at different positions at the current moment; performing global nearest neighbor association on the observed objects in the multiple images of the area to be positioned to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras; determining the observed pose of the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras; using the observed pose of the observed object to perform Kalman filter update to obtain the estimated pose of the observed object; The embodiment of the present invention performs fusion positioning on the images of the area to be positioned in a certain actual scene captured by roadside cameras, and combines Kalman filter technology for positioning update, so as to achieve high-precision positioning of vehicles and pedestrians in the area to be positioned.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-camera fusion positioning, and in particular, to a field-side multi-camera fusion positioning method, device, and storage medium. Background Art

[0002] Currently, the commonly used vehicle positioning technologies are mainly divided into direct positioning and dead reckoning. Among them, direct positioning is mainly based on the spatial intersection measurement of GPS signals and the matching positioning of environmental features, and dead reckoning is the integration positioning based on information such as vehicle acceleration, angular velocity, and speed combined with an initial value. However, in actual scenarios, such as indoors, underground, and complex environments, due to weak GPS signals or complex environments, it is difficult to determine the accurate position of the vehicle in the above scenarios using traditional vehicle positioning technologies.

[0003] Therefore, how to perform high-precision positioning of vehicles in various actual scenarios has become an urgent problem to be solved in this field. Summary of the Invention

[0004] An embodiment of the present invention provides a field-side multi-camera fusion positioning method, which can achieve high-precision positioning of a vehicle.

[0005] An embodiment of the present invention provides a field-side multi-camera fusion positioning method, which includes:

[0006] Obtaining images of the area to be located recognized by multiple cameras at different positions at the current moment;

[0007] Performing global nearest neighbor association on the observed objects in the multiple images of the area to be located, and obtaining the grounding point positions and pixel coordinates of the same observed object under different cameras;

[0008] Determining the observed pose of the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras;

[0009] Performing Kalman filter update using the observed pose of the observed object to obtain the estimated pose of the observed object.

[0010] As an improvement of the above solution, the determining the observed pose of the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras includes:

[0011] Calculating the central pose of the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras;

[0012] Performing three-dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object.

[0013] As an improvement to the above solution, calculating the central pose of the observed object based on the grounding point positions and pixel coordinates of the observed object under different cameras includes:

[0014] Performing epipolar constraint on the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras to obtain the central pixel coordinates of the 2D bounding box of the observed object under different cameras;

[0015] Wherein, the intersection of the epipolar constraints is the center of the 2D bounding boxes of multiple images of the regions to be located where the observed object is located; the 2D bounding box is the circumscribed rectangle of the observed object in the corresponding image of the region to be located;

[0016] Solving for the central pose of the observed object by triangulation according to the central pixel coordinates of the 2D bounding boxes of the observed object under different cameras.

[0017] As an improvement to the above solution, when the observed object is a vehicle, the central pose includes the central coordinates of the vehicle and its heading angle;

[0018] Then, performing three-dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object includes:

[0019] Performing pose estimation through a preset cuboid projection model according to the central coordinates of the vehicle and its heading angle to obtain the observed pose of the vehicle.

[0020] As an improvement to the above solution, when the observed object is a pedestrian, the central pose includes the central coordinates of the pedestrian;

[0021] Then, performing three-dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object includes:

[0022] Performing position estimation through a preset cylinder projection model according to the central coordinates of the pedestrian to obtain the observed pose of the pedestrian.

[0023] As an improvement to the above solution, using the observed pose of the observed object for Kalman filter update to obtain the estimated pose of the observed object includes:

[0024] Obtaining the filtered poses of each observed object output by the Kalman filter at the current moment; wherein, the filtered pose of the observed object is estimated by the Kalman filter according to the filtered pose of the corresponding observed object at the previous moment and the corresponding motion model;

[0025] Perform global nearest neighbor association on the observed poses of each of the observed objects and the filtered poses of each of the observed objects to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter;

[0026] According to the first association result of each observed object and its observed pose, use the Kalman filter to update the pose of each observed object to obtain the estimated pose of the observed object.

[0027] As an improvement to the above solution, the performing global nearest neighbor association on the observed poses of each of the observed objects and the filtered poses of each of the observed objects to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter includes:

[0028] Calculate the first Euclidean distance between the observed pose of each observed object and the filtered pose of each observed object;

[0029] Construct a first cost matrix based on the calculated first Euclidean distance;

[0030] Perform global nearest neighbor association solution based on the first cost matrix to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter.

[0031] As an improvement to the above solution, the performing global nearest neighbor association solution based on the first cost matrix to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter includes:

[0032] Update the first Euclidean distance exceeding a preset threshold in the first cost matrix to a first fixed value;

[0033] Perform global nearest neighbor association solution based on the updated first cost matrix to obtain an initial first association result between the observed pose of each observed object and its filtered pose in the Kalman filter;

[0034] Perform filtering processing on the initial first association result between the observed pose of each observed object and its filtered pose in the Kalman filter to obtain a final first association result; wherein, the first Euclidean distance in the final first association result does not exceed the preset threshold.

[0035] As an improvement to the above solution, the using the Kalman filter to update the pose of each observed object according to the first association result of each observed object and its observed pose to obtain the estimated pose of the observed object includes:

[0036] When the correlation result of the observed object is that the Kalman filter has pose tracking of the observed object, according to the observed pose of the observed object, the Kalman filter is used to update the pose of the corresponding observed object to obtain the estimated pose of the observed object;

[0037] When the correlation result of the observed object is that the Kalman filter does not have pose tracking of the observed object, the observed object is added to the Kalman filter, and its filtering pose in the Kalman filter is initialized according to the observed pose of the observed object.

[0038] As an improvement of the above solution, the global nearest neighbor correlation of the observed objects in the multiple images of the area to be located is performed to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras, including:

[0039] According to the pixel coordinates of each observed object in the image of the area to be located and the external parameters of the corresponding camera, calculate the grounding point positions of each observed object in the image of the area to be located;

[0040] Calculate the second Euclidean distances between the pixel coordinates and between the grounding point positions of each observed object in any two images of the area to be located;

[0041] According to the calculated second Euclidean distances, construct a second cost matrix;

[0042] According to the second cost matrix, perform global nearest neighbor correlation solution to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras.

[0043] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: For a certain actual scenario, multiple cameras installed on the roadside are set to capture images of the area to be located in the same area to be located and identify the observed objects in the corresponding images of the area to be located; then, the global nearest neighbor correlation of the observed objects in the multiple images of the area to be located is performed to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras; according to the grounding point positions and pixel coordinates of the observed object under different cameras, determine the observed pose of the observed object; use the observed pose of the observed object for Kalman filter update to obtain the estimated pose of the observed object. The embodiments of the present invention perform fusion positioning based on the images of the area to be located captured by the roadside cameras and combine Kalman filter technology for positioning update, and can achieve high-precision positioning of each observed object (such as vehicles, pedestrians) in the area to be located. Description of the Drawings

[0044] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required for the implementation will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 is a flowchart of a field-end multi-camera fusion positioning method provided by an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the camera layout provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of solving the position by epipolar constraint provided by an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of the three-dimensional projection of a vehicle provided by an embodiment of the present invention;

[0049] Figure 5 is a schematic diagram of the three-dimensional projection of a pedestrian provided by an embodiment of the present invention.

[0050] Figure 6 is a schematic diagram of the positioning process provided by an embodiment of the present invention; Specific Embodiments

[0051] The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0052] Please refer to Figure 1 , which is a schematic diagram of the process of a field-end multi-camera fusion positioning method provided by an embodiment of the present invention. The field-end multi-camera fusion positioning method is executed by a cloud server, and the method specifically includes:

[0053] S11: Obtain the images of the area to be located recognized by multiple cameras at different positions at the current moment;

[0054] Among them, the image of the area to be located is marked with at least one observed object and carries the pixel coordinates of the observed object; or the image of the area to be located does not recognize the observed object.

[0055] S12: Perform global nearest neighbor association on the observed objects in the multiple images of the area to be located to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras;

[0056] Exemplarily, a plurality of the cameras are arranged at different positions on the roadside of the same area to be located, and are arranged at a set height from the ground. For example, a camera is installed at a height of 3 meters from the ground, and the external and internal parameters of each camera are pre-calibrated to capture the area to be located from different angles, so as to obtain images of the area to be located from different perspectives. As Figure 2 shown, it gives an example of arranging a camera at each of the four vertices of a rectangular area to be located. The cameras in the same area to be located trigger image recognition synchronously, and the observation object can be a vehicle and / or a pedestrian. For example, each of the cameras respectively performs recognition of the observation object on the images of the area to be located captured at the same moment, and marks the recognized observation object. For example, the camera can use a neural network to perform machine learning on the image of the area to be located captured by it, so as to mark the observation object and the observation object category in the corresponding image of the area to be located. The observation object category includes a vehicle category and a pedestrian category. Each of the cameras calculates the grounding point position of each of the observation objects under the corresponding camera with the center of the lower edge of the corresponding image of the area to be located as the grounding center point according to its own external parameters. Among them, calculating the position of the target object in the corresponding image according to the external parameters of the camera belongs to the prior art and will not be elaborated here. The camera sends the recognized image of the area to be located, its observation object, the observation object type, the pixel coordinates, and the grounding point position to the cloud server for subsequent optimized positioning.

[0057] The cloud server performs correlation matching on each observation object in the images of the area to be located captured by multiple cameras through global nearest neighbor association, and matches each observation object under different cameras one by one, so as to obtain the grounding point position and pixel coordinates of the same observation object in different camera images.

[0058] S13: Determine the observation pose of the observation object according to the grounding point position and pixel coordinates of the observation object under different cameras;

[0059] S14: Use the observation pose of the observation object to perform Kalman filter update to obtain the estimated pose of the observation object.

[0060] In the embodiment of the present invention, fusion positioning is performed based on the images of the area to be located captured by roadside cameras, and positioning update is combined with Kalman filter technology, so as to realize high-precision positioning of each observation object (such as a vehicle, a pedestrian) in the area to be located, and provide data support for the automatic driving of the vehicle. At the same time, low-cost cameras installed on the roadside are used, and there is no need to install in-vehicle cameras on the vehicle, which saves costs. In addition, images with a full view of the vehicle to be located can be obtained, further improving the positioning accuracy.

[0061] In an alternative embodiment, globally associating the observed objects in the images of multiple regions to be located to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras includes:

[0062] Calculate the grounding point positions of the observed objects in the images of the regions to be located according to the pixel coordinates of the observed objects in the images of the regions to be located and the external parameters of the corresponding cameras;

[0063] Calculate the second Euclidean distances between the pixel coordinates and between the grounding point positions of the observed objects in any two images of the regions to be located;

[0064] Construct a second cost matrix according to the calculated second Euclidean distances;

[0065] Perform global nearest neighbor association solution according to the second cost matrix to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras.

[0066] Exemplarily, the process of global nearest neighbor association solution of the same observed object under different cameras is as follows: The second Euclidean distance sqrt((x 3 -x 4 ) 2 +(y 3 -y 4 ) 2 ) can be calculated, where (x 3 , y 3 ) represents the observed pose of an observed object under a certain camera, and (x 4 , y 4) represents the observed pose of the observed object under another camera; taking the calculated second Euclidean distance as matrix elements, an m×m second cost matrix can be obtained; the element in the i-th row and j-th column of the second cost matrix represents the second Euclidean distance between the observed objects i and j of the camera. Among them, when it is necessary to associate the observed objects of the vehicle category, the second Euclidean distance related to the pedestrian category in the second cost matrix is set to the maximum value. Among them, the maximum value can be preset to 100. For example, the first Euclidean distance between a certain pedestrian and a certain vehicle, and between a certain pedestrian and a certain pedestrian is set to 100; similarly, when it is necessary to associate the observed objects of the pedestrian category, the second Euclidean distance related to the vehicle category in the second cost matrix is set to the maximum value. For example, the second Euclidean distance between a certain pedestrian and a certain vehicle, and between a certain vehicle and a certain vehicle is set to 100. The second cost matrix of the vehicle or pedestrian is obtained according to the above settings; then, a threshold judgment is performed on the second cost matrix, that is, the second Euclidean distance exceeding the preset threshold in the second cost matrix is updated to the first fixed value, so as to obtain the final second cost matrix. According to this second cost matrix, a global nearest neighbor association is performed to solve the second association result between each observed object under each camera belonging to the same category, and the associations in the second association result where the second Euclidean distance exceeds the preset threshold are filtered out to obtain the final second association result; for example, the observed object 1 under camera 1 is associated and matched with the observed object 1 under camera 2, which is the same object, and the observed object 2 under camera 1 is associated and matched with the observed object 2 under camera 2, which is the same object. Establishing the association of the same observed object under different cameras can provide multi-perspective image data support for the three-dimensional projection modeling of the subsequent observed object, and further improve the positioning accuracy.

[0067] In an alternative embodiment, the determining the observed pose of the observed object according to the grounding point position and pixel coordinates of the observed object under different cameras includes:

[0068] Step S131: Calculate the central pose of the observed object according to the grounding point position and pixel coordinates of the observed object under different cameras;

[0069] Further, the calculating the central pose of the observed object according to the grounding point position and pixel coordinates of the observed object under different cameras includes:

[0070] Perform an epipolar constraint on the observed object according to the grounding point position and pixel coordinates of the observed object under different cameras to obtain the central pixel coordinates of the 2D box of the observed object under different cameras;

[0071] Among them, the intersection of the epipolar constraints is the center of the 2D box of the multiple to-be-localized region images where the observed object is located; the 2D box is the circumscribed rectangle of the observed object in the corresponding to-be-localized region image.

[0072] According to the central pixel coordinates of the 2D boxes of the observed object under different cameras, the central pose of the observed object is obtained by triangulation.

[0073] As Figure 3 described, taking the center of the 2D boxes of the multiple to-be-localized region images where the observed object is located as the intersection of the epipolar constraints, and using the particle model to simplify each observed object into a feature point P in the to-be-localized region images under different cameras j，k , where P j，k represents the point of the j-th observed object in the to-be-localized region image under the k-th camera. Then, the epipolar constraint is solved to obtain the central pixel coordinates of the observed object. After that, taking the central pixel coordinates of the observed object as the initial value, triangulation is performed to obtain the central pose of the observed object. The specific triangulation process is as follows:

[0074] Let the homogeneous coordinates of the observed object be [x y z 1] T , then the projection of the homogeneous coordinates [x y z 1] T onto the to-be-localized region image is:

[0075]

[0076] Multiply both sides of the formula by μ on the left to get μ^PX = 0, and after expansion, it can be obtained:

[0077]

[0078] Among them, P 1 , P 2 , P 3 respectively represent the central pixel coordinates of the to-be-localized region images of the same observed object under three different cameras; λ represents the depth value; represents the transformation from the world coordinate system to the camera coordinate system. The value of X can be solved through the above formula, so as to obtain the central coordinates of the observed object. When the observed object is a vehicle, the solved central coordinates and the heading angle of the vehicle are used as the central pose of the vehicle. When the observed object is a pedestrian, the solved central coordinates are used as the central pose of the pedestrian.

[0079] Step S132: Perform three-dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object.

[0080] Further, when the observed object is a vehicle, the central pose includes the central coordinates of the vehicle and its heading angle;

[0081] Then, performing three-dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object includes:

[0082] Performing pose estimation through a preset cuboid projection model according to the central coordinates of the vehicle and its heading angle to obtain the observed pose of the vehicle.

[0083] As Figure 4 shown, performing cuboid projection on the observed object of the vehicle category through a preset cuboid projection model; the cuboid projection model is:

[0084]

[0085] wherein, w represents the width of the vehicle, h represents the height of the vehicle, and l represents the length of the vehicle, represents the camera model, represents the coordinates of the 4 vertices of the 3D Bounding Box of the vehicle in the vehicle coordinate system. For example, take Figure 4 the 3 vertices on the bottom surface and 1 vertex on the top surface marked in represents the coordinates of the corresponding 4 vertices of the vehicle in the camera coordinate system; (x car , y car , z car ) represents the central coordinates of the vehicle in the global coordinate system, that is, the coordinate value in X obtained by the above triangulation solution; u min represents the horizontal position of the left edge of the pixel of the 2D Bounding Box of the vehicle in the image of the area to be located, u max represents the horizontal position of the right edge of the pixel of the 2D Bounding Box of the vehicle in the image of the area to be located, v min represents the vertical position of the lower edge of the pixel of the 2D Bounding Box of the vehicle in the image of the area to be located, v max represents the vertical position of the upper edge of the pixel of the 2D Bounding Box of the vehicle in the image of the area to be located, u i represents the horizontal position of the vertex of the vehicle in the image of the area to be located, v i represents the upper edge of the pixel of the 2D BoundingBox of the vehicle in the image of the area to be located. K represents the transformation from the image coordinate system to the camera coordinate system, represents the transformation from the camera coordinate system to the vehicle coordinate system; θ represents the heading angle of the vehicle.

[0086] Through the above projection transformation, the positioning of the vehicle can be optimized to obtain a more accurate vehicle pose.

[0087] Further, when the observed object is a pedestrian, the central pose includes the central coordinates of the pedestrian.

[0088] Then, the three-dimensional projection of the observed object according to the central pose of the observed object to obtain the observed pose of the observed object includes:

[0089] According to the central coordinates of the pedestrian, position estimation is performed through a preset cylinder projection model to obtain the observed pose of the pedestrian.

[0090] As Figure 5 shown, a cylinder projection is performed on the observed object of the pedestrian category through a preset cylinder projection model; the cylinder projection model is:

[0091]

[0092] where represents the coordinates of the discrete points of the cylinder projection model of the pedestrian in the pedestrian coordinate system; r represents the radius of the cylinder projection model; (xped, yped, zped) represents the central coordinates of the vehicle in the global coordinate system, u up represents the horizontal pixel coordinate of the discrete point on the upper circle of the cylinder projection model; v up represents the vertical pixel coordinate of the discrete point on the upper circle of the cylinder projection model, u down represents the horizontal pixel coordinate of the discrete point on the lower circle of the cylinder projection model, v down represents the vertical pixel coordinate of the discrete point on the lower circle of the cylinder projection model; a min = min(u up , u down ), a max = max(u up , u down ), b min = min(v up , v down ), a b min = min(u up , u doon ), a max = max(u up , u down ).

[0093] Through the above projection transformation, the positioning of the pedestrian can be optimized to obtain a more accurate pedestrian position.

[0094] In an alternative embodiment, the use of the observed pose of the observed object for Kalman filter update to obtain the estimated pose of the observed object includes:

[0095] Obtain the filtered poses of each of the observed objects output by the Kalman filter at the current moment; wherein, the filtered pose of the observed object is estimated by the Kalman filter based on the filtered pose of the corresponding observed object at the previous moment and the corresponding motion model;

[0096] Exemplarily, the Kalman filter can estimate the moving speed of the observed object according to the position change of the observed object in the images of the area to be located in consecutive frames, so as to construct a corresponding motion model; then, based on the filtered pose output at the previous moment and the deduced motion model, further estimate the filtered pose of the observed object at the current moment.

[0097] Perform global nearest neighbor association on the observed poses of each of the observed objects and the filtered poses of each of the observed objects to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter;

[0098] Update the pose of each of the observed objects using the Kalman filter according to the first association result of each observed object and its observed pose to obtain the estimated pose of the observed object.

[0099] Further, the performing global nearest neighbor association on the observed poses of each of the observed objects and the filtered poses of each of the observed objects to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter includes:

[0100] Calculate the first Euclidean distance between the observed pose of each of the observed objects and the filtered pose of each of the observed objects;

[0101] Construct a first cost matrix based on the calculated first Euclidean distance;

[0102] Perform global nearest neighbor association solution according to the first cost matrix to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter.

[0103] Further, the performing global nearest neighbor association solution according to the first cost matrix to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter includes:

[0104] Update the first Euclidean distances in the first cost matrix that exceed a preset threshold to a first fixed value;

[0105] Perform global nearest neighbor association solution based on the updated first cost matrix to obtain the initial first association result between the observed poses of each of the observed objects and their filtered poses in the Kalman filter;

[0106] Perform filtering processing on the initial first association result between the observed poses of each of the observed objects and their filtered poses in the Kalman filter to obtain the final first association result; wherein, the first Euclidean distances in the final first association result do not exceed a preset threshold;

[0107] In the embodiment of the present invention, it is necessary to perform association matching between the observed poses of the observed objects captured by the camera and the filtered poses of the objects tracked by the Kalman filter. Specifically, the first Euclidean distance sqrt((x 1 -x 2 ) 2 +(y 1 -y 2 ) 2 ) can be calculated between the observed poses of m observed objects under the camera and the n filtered poses output by the Kalman filter, where (x 1 , y 1 ) represents the observed pose, and (x 2 , y 2)It represents the filtered pose; taking the calculated first Euclidean distance as matrix elements, a first cost matrix of m×n can be obtained; the element in the i-th row and j-th column of the first cost matrix represents the first Euclidean distance between the tracking object i of the Kalman filter and the observed object j of the camera. Among them, when it is necessary to associate the observed objects of the vehicle category, the first Euclidean distances related to the pedestrian category in the first cost matrix are set to the maximum value, and at the same time, the first Euclidean distances corresponding to the vehicles with yaw angles exceeding the preset angle threshold are also set to the maximum value. Among them, the maximum value can be preset to 100. For example, the first Euclidean distances between a certain pedestrian and a certain vehicle, and between a certain pedestrian and a certain pedestrian are set to 100; similarly, when it is necessary to associate the observed objects of the pedestrian category, the first Euclidean distances related to the vehicle category in the first cost matrix are set to the maximum value. For example, the first Euclidean distances between a certain pedestrian and a certain vehicle, and between a certain vehicle and a certain vehicle are set to 100. According to the above settings, the first cost matrix of the vehicle or pedestrian is obtained; then, a threshold judgment is performed on the first cost matrix, that is, the first Euclidean distances exceeding the preset threshold in the first cost matrix are updated to a first fixed value, and the first fixed value is 100, so as to obtain the final first cost matrix. According to this first cost matrix, a global nearest neighbor association is performed to solve the first association result between the observed objects belonging to the same category and the tracking objects of the Kalman filter, and the associations in the first association result with first Euclidean distances exceeding the preset threshold are filtered out to obtain the final first association result; for example, the observed object 1 is associated and matched with the tracking object 1 of the Kalman filter, which is the same object, the observed object 2 is associated and matched with the tracking object 2 of the Kalman filter, which is the same object, the observed object 3 is associated and matched with the tracking object 3 of the Kalman filter, which is the same object, and subsequently, Kalman filter updates are performed according to the observed poses and filtered poses corresponding to the associated and matched observed objects.

[0108] In an alternative embodiment, the step of using the Kalman filter to update the pose of each of the observed objects according to the first association result of each observed object and its observed pose to obtain the estimated pose of the observed object includes:

[0109] When the association result of the observed object is that the Kalman filter has pose tracking for the observed object, according to the observed pose of the observed object, the Kalman filter is used to update the pose of the corresponding observed object to obtain the estimated pose of the observed object;

[0110] When the association result of the observed object is that the Kalman filter does not have pose tracking for the observed object, the observed object is added to the Kalman filter, and its filtered pose in the Kalman filter is initialized according to the observed pose of the observed object.

[0111] Specifically, as Figure 6As shown, when the observed object is successfully associated and matched with the object tracked by the Kalman filter, Kalman filter update is performed based on the observed pose and the corresponding filtered pose of the observed object; otherwise, the observed pose of the observed object is used as its filtered pose in the Kalman filter, initialization processing is performed, and the observed object is added as an object tracked by the Kalman filter.

[0112] The working principle of the Kalman filter is as follows:

[0113] Pose prediction: X′ n+1 = A X n + BU;

[0114] Covariance prediction: P′ n+1 = FP n F T + Q;

[0115] Kalman gain calculation: S = P′ n+1 H(HP′ n+1 H T + R) -1 ;

[0116] Pose update: X n+1 = X′ n+1 + S(Z k - H × X′ n+1 );

[0117] Covariance update: P n+1 = (1 - SH)P′ n+1 ;

[0118] Where, X′ n+1 represents the filtered pose obtained by Kalman filter prediction update at the (n + 1)th moment; X n represents the estimated pose obtained by Kalman filter measurement update at the nth moment; X n+1 represents the estimated pose obtained by Kalman filter measurement update at the (n + 1)th moment; A represents the state transition matrix; B represents the control matrix; U represents the control state quantity; P′ n+1 represents the covariance prediction value obtained by Kalman filter prediction update at the (n + 1)th moment; P n+1 represents the covariance obtained by Kalman filter measurement update at the (n + 1)th moment; P n represents the covariance obtained by Kalman filter measurement update at the nth moment; Q represents the process covariance; S represents the Kalman gain; H represents the observation matrix, H T represents the transpose of H; R represents the observation covariance; Z k represents the matrix of the observed object (such as a vehicle, pedestrian) in the world coordinate system; F represents the Jacobian of the state transition matrix, F TDenotes the transpose of F. Exemplarily, the observation matrix H can be generated according to the observed pose of the observed object, and the above-mentioned observation covariance R can be obtained by calculating the covariance of the observed pose.

[0119] Initialize the Kalman filter: Initialize the variables, prediction update equation, and measurement update equation of the Kalman filter according to the initial filtering pose of the observed object's observed pose under multiple cameras and the preset initial parameters of the Kalman filter.

[0120] Iteration of the Kalman filter: At each iteration, obtain the estimated pose X n+1 , and the corresponding error covariance P k+1 . Each iteration period includes prediction update and measurement update.

[0121] Prediction update: According to the estimated pose X n at the previous moment, estimate the filtered pose X' n+1 at the current moment using the prediction update equation, and predict the predicted state error covariance P' k at the current moment according to the measurement error covariance P n+1 .

[0122] Measurement update: Based on the observed pose of the observed object at the current moment, determine the observation matrix H and the observation covariance R, and update the state X' n+1 →X n+1 and the state error covariance P' n+1 →P n+1 .

[0123] By continuously iterating the prediction update and the measurement update, as time goes by, the current moment becomes the previous moment, and a new current moment is re-estimated.

[0124] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: By using the images captured by the cameras installed on the roadside to locate vehicles and pedestrians, the cost can be reduced; at the same time, based on the multi-view images of the same area to be located under multiple cameras, the high computing power of the cloud server can be used to perform GNN association and projection optimization on the vehicles and pedestrians in the images of the area to be located, so as to achieve high-precision positioning of vehicles and pedestrians. Secondly, by associating and tracking the observed poses of vehicles and pedestrians with the Kalman filter, the positioning accuracy can be further improved.

[0125] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, many improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A field - end multi - camera fusion positioning method, characterized in that, it includes: Obtain the images of the area to be located recognized by multiple cameras at different positions at the current moment. The cameras are set at different positions on the roadside of the same area to be located; perform global nearest - neighbor association on the observed objects in the multiple images of the area to be located to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras; Calculate the central pose of the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras; The calculating the central pose of the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras includes: performing epipolar constraint on the observed object according to the grounding point positions and pixel coordinates of the observed object under different cameras to obtain the central pixel coordinates of the 2D box of the observed object under different cameras; wherein, the center of the 2D box of the multiple images of the area to be located where the observed object is located is the intersection point of the epipolar constraint; the 2D box is the circumscribed rectangle of the observed object in the corresponding image of the area to be located; calculate the central pose of the observed object by triangulation according to the central pixel coordinates of the 2D box of the observed object under different cameras; Determine the observed pose of the observed object according to the central pose of the observed object; Use the observed pose of the observed object for Kalman filter update to obtain the estimated pose of the observed object.

2. The field - end multi - camera fusion positioning method according to claim 1, characterized in that, the determining the observed pose of the observed object according to the central pose of the observed object includes: Performing three - dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object.

3. The field - end multi - camera fusion positioning method according to claim 2, characterized in that, When the observed object is a vehicle, the central pose includes the central coordinates and its heading angle of the vehicle; then, the performing three - dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object includes: estimating the pose according to the central coordinates and heading angle of the vehicle through a preset cuboid projection model to obtain the observed pose of the vehicle.

4. The field - end multi - camera fusion positioning method according to claim 2, characterized in that, When the observed object is a pedestrian, the central pose includes the central coordinates of the pedestrian; then, the performing three - dimensional projection on the observed object according to the central pose of the observed object to obtain the observed pose of the observed object includes: estimating the position according to the central coordinates of the pedestrian through a preset cylinder projection model to obtain the observed pose of the pedestrian.

5. The field - end multi - camera fusion positioning method according to claim 2, characterized in that, the using the observed pose of the observed object for Kalman filter update to obtain the estimated pose of the observed object includes: Obtain the filtered poses of each of the observed objects output by the Kalman filter at the current moment; wherein, the filtered pose of the observed object is estimated by the Kalman filter according to the filtered pose of the corresponding observed object at the previous moment and the corresponding motion model; perform global nearest neighbor association on the observed poses of each of the observed objects and the filtered poses of each of the observed objects to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter; according to the first association result of each observed object and its observed pose, use the Kalman filter to update the pose of each of the observed objects to obtain the estimated pose of the observed object.

6. The field-side multi-camera fusion positioning method according to claim 5, characterized in that the performing global nearest neighbor association on the observed poses of each of the observed objects and the filtered poses of each of the observed objects to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter includes: Calculate the first Euclidean distance between the observed pose of each of the observed objects and the filtered pose of each of the observed objects; construct a first cost matrix according to the calculated first Euclidean distance; perform global nearest neighbor association solution according to the first cost matrix to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter.

7. The field-side multi-camera fusion positioning method according to claim 6, characterized in that the performing global nearest neighbor association solution according to the first cost matrix to obtain a first association result between the observed pose of each observed object and its filtered pose in the Kalman filter includes: Update the first Euclidean distance exceeding a preset threshold in the first cost matrix to a first fixed value; perform global nearest neighbor association solution according to the updated first cost matrix to obtain an initial first association result between the observed pose of each of the observed objects and its filtered pose in the Kalman filter; perform filtering processing on the initial first association result between the observed pose of each of the observed objects and its filtered pose in the Kalman filter to obtain a final first association result; wherein, the first Euclidean distance in the final first association result does not exceed the preset threshold.

8. The field-side multi-camera fusion positioning method according to claim 5, characterized in that Updating the poses of the respective observed objects using the Kalman filter based on the first association results and the observed poses of the respective observed objects to obtain the estimated poses of the observed objects includes: when the association result of the observed object is that the Kalman filter is tracking the pose of the observed object, updating the pose of the corresponding observed object using the Kalman filter according to the observed pose of the observed object to obtain the estimated pose of the observed object; when the association result of the observed object is that the Kalman filter is not tracking the pose of the observed object, adding the observed object to the Kalman filter and initializing its filtering pose in the Kalman filter according to the observed pose of the observed object.

9. The field-end multi-camera fusion positioning method according to claim 1, characterized in that performing global nearest neighbor association on the observed objects in the multiple images of the areas to be positioned to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras, including: calculating the grounding point positions of the respective observed objects in the images of the areas to be positioned according to the pixel coordinates of the respective observed objects in the images of the areas to be positioned and the external parameters of the corresponding cameras; calculating the second Euclidean distances between the pixel coordinates and between the grounding point positions of the respective observed objects in any two images of the areas to be positioned; constructing a second cost matrix according to the calculated second Euclidean distances; and solving for global nearest neighbor association according to the second cost matrix to obtain the grounding point positions and pixel coordinates of the same observed object under different cameras.

Images