A visual perception positioning method and device, electronic equipment and storage medium

Abstract

The application discloses a visual perception positioning method and device, electronic equipment and storage medium. The visual perception positioning method comprises the following steps: inputting the relative coordinates of a tracking target in a current frame and the installation position of a camera into a Kalman filter to obtain a current frame estimation value corresponding to the tracking target; determining candidate three-dimensional information corresponding to the tracking target in the current frame according to the current frame estimation value, wherein the candidate three-dimensional information at least comprises the position, candidate speed and candidate heading angle of the tracking target; when the candidate three-dimensional information satisfies a set optimization condition, optimizing the candidate three-dimensional information according to a plurality of historical three-dimensional information to obtain target three-dimensional information, wherein the plurality of historical three-dimensional information comprises three-dimensional information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame; and when the candidate three-dimensional information does not satisfy the set optimization condition, determining the candidate three-dimensional information as the target three-dimensional information. The method can improve the accuracy of visual perception positioning of the tracking target.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of intelligent transportation technology, and in particular to a visual perception positioning method, device, electronic device and storage medium. Background Technology

[0002] In a roadside visual perception and positioning system, the combination of cameras and edge computing devices can be used to perceive, identify and locate tracking targets (including but not limited to motor vehicles, non-motor vehicles or pedestrians) within the perception range, and send the perception results to surrounding vehicles to improve driving safety and traffic efficiency.

[0003] In existing technologies, images captured by cameras can be processed using tracking algorithms to obtain two-dimensional information about the tracking target in the image. Then, a positioning algorithm can be used to locate the tracking target based on the two-dimensional information of the tracking target in the image. However, when locating the tracking target, the speed of the tracking target obtained by the positioning algorithm in existing technologies is unstable, and the heading angle of the tracking target is prone to jump, especially when the tracking target is running at low speed. This leads to a decrease in the accuracy of visual perception positioning of the tracking target. Summary of the Invention

[0004] This invention provides a visual perception positioning method, device, electronic device, and storage medium, which can improve the accuracy of visual perception positioning of tracked targets.

[0005] In a first aspect, embodiments of the present invention provide a visual perception localization method, comprising:

[0006] The relative coordinates of the tracking target and the camera's installation position in the current frame are input into the Kalman filter to obtain the estimated value of the tracking target in the current frame.

[0007] Based on the current frame estimation value, candidate 3D information corresponding to the tracked target in the current frame is determined, and the candidate 3D information includes at least the position, candidate velocity, and candidate heading angle of the tracked target;

[0008] When the candidate 3D information meets the set optimization conditions, the candidate 3D information is optimized based on multiple historical 3D information to obtain the target 3D information. The multiple historical 3D information includes the 3D information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame.

[0009] When the candidate 3D information does not meet the set optimization conditions, the candidate 3D information is determined as the target 3D information.

[0010] Secondly, embodiments of the present invention provide a visual perception positioning device, characterized in that it includes:

[0011] The input module is used to input the relative coordinates of the tracking target and the camera's installation position in the current frame into the Kalman filter to obtain the current frame estimate of the tracking target;

[0012] The first determining module is used to determine candidate three-dimensional information corresponding to the tracked target in the current frame based on the current frame estimation value, wherein the candidate three-dimensional information includes at least the position, candidate velocity and candidate heading angle of the tracked target;

[0013] The second determining module is used to optimize the candidate three-dimensional information according to multiple historical three-dimensional information to obtain target three-dimensional information when the candidate three-dimensional information meets the set optimization conditions. The multiple historical three-dimensional information includes the three-dimensional information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame.

[0014] The third determining module is used to determine the candidate three-dimensional information as the target three-dimensional information when the candidate three-dimensional information does not meet the set optimization conditions.

[0015] Thirdly, embodiments of the present invention provide an electronic device, including:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0020] The technical solution of this invention involves inputting the relative coordinates of the tracking target and the camera's installation position in the current frame into a Kalman filter to obtain an estimated value for the current frame corresponding to the tracking target. Based on this estimated value, candidate 3D information corresponding to the tracking target in the current frame is determined. This candidate 3D information includes at least the tracking target's position, candidate velocity, and candidate heading angle. When the candidate 3D information meets set optimization conditions, it is optimized based on multiple historical 3D information sets to obtain target 3D information. These historical 3D information sets include the 3D information corresponding to each frame in a sliding window of a set size adjacent to the current frame. When the candidate 3D information does not meet the set optimization conditions, it is determined as the target 3D information. This technical solution first obtains the estimated value of the current frame output by the Kalman filter, then determines the candidate 3D information of the tracking target using this estimated value. When the candidate 3D information meets the set optimization conditions, it optimizes the 3D information corresponding to each frame in the sliding window to obtain the target 3D information. This optimizes for phenomena such as unstable tracking target velocity or easily fluctuating tracking target heading angle, improving the accuracy of visual perception and positioning of the tracking target.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a visual perception positioning method provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a visual perception positioning method provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of a visual perception positioning device according to Embodiment 3 of the present invention;

[0026] Figure 4 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a visual perception positioning method according to Embodiment 1 of the present invention. This embodiment is applicable to situations requiring visual perception positioning. The method can be executed by a visual perception positioning device, which can be implemented in software and / or hardware and integrated into an electronic device. Further, the electronic device includes, but is not limited to, computers, laptops, smartphones, servers, etc. Figure 1 As shown, the method includes:

[0032] S110. Input the relative coordinates of the tracking target and the camera's installation position in the current frame into the Kalman filter to obtain the estimated value of the tracking target in the current frame.

[0033] The tracking target can refer to a target tracked through visual perception and positioning. There are no limitations on the tracking target; it can include, but is not limited to, motor vehicles, non-motor vehicles, or pedestrians. The camera can be any camera capable of capturing images of the tracking target. There are no restrictions on the camera's installation location; it can be determined based on the specific application requirements.

[0034] When performing roadside visual perception and localization, a combination of cameras and electronic devices can be used to perceive, identify, and locate the target within the perception range. For example, tracking algorithms can be used to process images captured by the camera to obtain two-dimensional information about the target, such as the target's identifier, its pixel coordinates, or its confidence level. Then, coordinate transformation can be performed on the target's pixel coordinates to determine its coordinates in the actual scene, such as the relative coordinates between the target and the camera's installation location. The target can then be located using these relative coordinates. The specific tracking algorithm is not limited, as long as it can process the image to obtain the two-dimensional information of the target.

[0035] In the images captured by the camera, there may be one or more tracking targets. One or more tracking targets can be placed in a tracking list. The tracking targets in the tracking list are updated by capturing multiple frames of images from the camera. In this embodiment of the invention, the visual perception and localization of a tracking target in the current frame image is described as an example.

[0036] The relative coordinates between the tracked target and the camera's mounting position in the current frame can refer to the relative coordinates between the position of the tracked target in the actual scene within the image captured in the current frame and the camera's mounting position. The method for determining the relative coordinates is not limited. For example, the image of the current frame captured by the camera can be processed using a tracking algorithm to obtain the pixel coordinates of the tracked target in the image. The obtained pixel coordinates can be transformed to determine the latitude and longitude of the tracked target in the world coordinate system corresponding to the pixel coordinates. Using the latitude and longitude of the camera's mounting position as the origin of the world coordinate system and the north direction as the vertical axis (i.e., the Y-axis) of the coordinate system, the relative coordinates between the latitude and longitude of the tracked target and the latitude and longitude of the camera's mounting position can be determined. This relative coordinate can be a relative position expressed as the distance between the tracked target and the camera, converted from the relative position of latitude and longitude.

[0037] The current frame estimate can be obtained by inputting the relative coordinates of the tracked target and the camera in the current frame into a Kalman filter. This estimate can be used to determine the 3D information of the tracked target in subsequent steps. The current frame estimate is not limited; it can include the lateral velocity component, longitudinal velocity component, lateral position coordinate, and longitudinal position coordinate of the tracked target. The lateral and longitudinal velocity components can be used to determine the speed and heading angle of the tracked target in the actual scene, while the lateral and longitudinal position coordinates can be used to determine the position of the tracked target in the actual scene, such as its latitude and longitude.

[0038] In a Kalman filter, the initial current frame estimate can be obtained from the previous frame estimate. Then, the relative coordinates of the tracked target and the camera's mounting position in the current frame are used as observations of the Kalman filter to update the initial current frame estimate, ultimately yielding the current frame estimate corresponding to the tracked target. Here, the previous frame estimate can be the Kalman filter output from the frame preceding the current frame; the initial current frame estimate can be the initial estimate obtained from the previous frame estimate using the Kalman filter; and the current frame estimate can be the final estimate obtained by updating the initial current frame estimate using the observations.

[0039] S120. Determine the candidate 3D information corresponding to the tracked target in the current frame based on the estimated value of the current frame. The candidate 3D information includes at least the position, candidate velocity, and candidate heading angle of the tracked target.

[0040] Among them, the candidate 3D information can be the candidate 3D information of the tracked target in the current frame. In the subsequent process, the target 3D information of the tracked target needs to be determined based on the candidate 3D information. The target 3D information is the final determined 3D information of the tracked target. The position, velocity and heading angle of the tracked target in the actual scene can be determined through the target 3D information.

[0041] The method for determining the candidate 3D information corresponding to the tracked target in the current frame based on the current frame estimate is not limited. For example, the candidate velocity included in the candidate 3D information can be determined based on the lateral and longitudinal velocity components included in the current frame estimate; the candidate heading angle included in the candidate 3D information can be determined based on the lateral and longitudinal velocity components included in the current frame estimate, and the candidate heading angle is the angle between the direction of the tracked target's movement and the due north direction; the position of the tracked target included in the candidate 3D information can be determined based on the lateral and longitudinal position coordinates included in the current frame estimate.

[0042] The horizontal and vertical position coordinates included in the current frame estimate can be the relative coordinates of the tracking target with respect to the camera's installation position. The position of the tracking target can be directly represented by the horizontal and vertical position coordinates, or the horizontal and vertical position coordinates can be converted into the latitude and longitude of the tracking target, and the position of the tracking target can be represented by the latitude and longitude of the tracking target. The specific method is not limited.

[0043] S130. When the candidate 3D information meets the set optimization conditions, the candidate 3D information is optimized based on multiple historical 3D information to obtain the target 3D information. The multiple historical 3D information includes the 3D information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame.

[0044] Setting optimization conditions refers to setting conditions that require optimization of the candidate 3D information. These optimization conditions can be determined based on actual application needs and are not specifically limited. In this embodiment of the invention, optimizing the candidate 3D information mainly involves optimizing the candidate velocity and candidate heading angle included in the candidate 3D information. The candidate 3D information includes the position, candidate velocity, and candidate heading angle of the tracking target; correspondingly, the target 3D information includes the position, target velocity, and target heading angle of the tracking target.

[0045] In one embodiment, setting the optimization conditions includes one or more of the following conditions:

[0046] The candidate velocity differs from the velocity of the tracked target in the previous frame by more than a first set threshold.

[0047] The candidate speed is less than the second set threshold;

[0048] The candidate speed is greater than the third set threshold;

[0049] The difference between the candidate heading angle and the heading angle of the tracked target in the previous frame is greater than the fourth set threshold;

[0050] The second set threshold is less than the third set threshold.

[0051] The first, second, third, and fourth set thresholds are all thresholds set according to actual needs. This invention does not limit them, as long as the second set threshold is less than the third set threshold.

[0052] The multiple historical 3D information includes the 3D information corresponding to each frame in a sliding window of a predetermined size that is adjacent to the current frame and precedes it. The size of the sliding window is predetermined and not limited. For example, the multiple historical 3D information includes the 3D information corresponding to each frame in the 30 frames adjacent to the current frame and precedes it, in which case the predetermined size is 30. The 3D information corresponding to each frame can include the position, velocity, and heading angle of the tracked target in each frame.

[0053] When the candidate 3D information meets the set optimization conditions, there is no limitation on the method of optimizing the candidate 3D information to obtain the target 3D information based on multiple historical 3D information. For example, when the candidate 3D information meets the set optimization conditions, the position of the tracking target included in the candidate 3D information is determined as the position of the tracking target included in the target 3D information; the average velocity of the tracking target included in the 3D information of each frame in the sliding window is determined as the target velocity; and the average heading angle of the tracking target included in the 3D information of each frame in the sliding window is determined as the target heading angle.

[0054] S140. When the candidate 3D information does not meet the set optimization conditions, the candidate 3D information is determined as the target 3D information.

[0055] When the candidate 3D information does not meet the set optimization conditions, it indicates that the current frame estimation value corresponding to the tracking target obtained by the Kalman filter is reliable. Therefore, the candidate 3D information corresponding to the tracking target in the current frame determined by the current frame estimation value is also reliable. Thus, the candidate 3D information can be directly determined as the target 3D information of the tracking target. Specifically, the position of the tracking target included in the candidate 3D information is determined as the position of the tracking target included in the target 3D information, the candidate velocity included in the candidate 3D information is determined as the target velocity included in the target 3D information, and the candidate heading angle included in the candidate 3D information is determined as the target heading angle included in the target 3D information.

[0056] The technical solution of this invention involves inputting the relative coordinates of the tracking target and the camera's installation position in the current frame into a Kalman filter to obtain an estimated value for the current frame corresponding to the tracking target. Based on this estimated value, candidate 3D information corresponding to the tracking target in the current frame is determined. This candidate 3D information includes at least the tracking target's position, candidate velocity, and candidate heading angle. When the candidate 3D information meets set optimization conditions, it is optimized based on multiple historical 3D information sets to obtain target 3D information. These historical 3D information sets include the 3D information corresponding to each frame in a sliding window of a set size adjacent to the current frame. When the candidate 3D information does not meet the set optimization conditions, it is determined as the target 3D information. This technical solution first obtains the estimated value of the current frame output by the Kalman filter, then determines the candidate 3D information of the tracking target using this estimated value. When the candidate 3D information meets the set optimization conditions, it optimizes the 3D information corresponding to each frame in the sliding window to obtain the target 3D information. This optimizes for phenomena such as unstable tracking target velocity or easily fluctuating tracking target heading angle, improving the accuracy of visual perception and positioning of the tracking target.

[0057] Example 2

[0058] Figure 2 This is a flowchart of a visual perception positioning method according to Embodiment 2 of the present invention. This embodiment, based on Embodiment 1, further refines the process of inputting the relative coordinates of the tracked target and the camera's installation position in the current frame into a Kalman filter to obtain the estimated value of the tracked target in the current frame; further refines the process of determining candidate 3D information corresponding to the tracked target in the current frame based on the estimated value of the current frame; and further refines the process of optimizing the candidate 3D information based on multiple historical 3D information to obtain the target's 3D information. For example... Figure 2 As shown, the method includes:

[0059] S111: Obtain the initial current frame estimate based on the previous frame estimate using a Kalman filter.

[0060] In a Kalman filter, the initial estimated value of the target at time t in the current frame can be predicted based on the estimated value of the previous frame at time t-1, and the estimation error covariance matrix of the system state at time t in the current frame can be determined. Specifically, this can be achieved using the following formula:

[0061]

[0062] P t - =AP t-1 A T +Q

[0063] in, This is the initial estimate for the current frame; A is the estimated value from the previous frame; B is the state transition matrix; C is the control matrix; D is the u-value. t-1 P is the control vector at time t-1; t - Let P be the covariance matrix of the estimation error of the system state at time t; t-1 Let be the estimation error covariance matrix of the system state at time t-1; Q is the process noise covariance matrix.

[0064] S112. Use the relative coordinates of the tracking target and the camera's installation position in the current frame as the observation value corresponding to the Kalman filter.

[0065] The relative coordinates between the tracked target and the camera's mounting position in the current frame can be determined in the following way:

[0066] The tracking algorithm is used to process the image of the current frame captured by the camera to obtain the pixel coordinates (u,v) of the tracked target in the current frame.

[0067] The obtained pixel coordinates are transformed using the following formula to determine the latitude and longitude (X,Y) of the tracking target in the world coordinate system.

[0068]

[0069] Where s is a scale factor of arbitrary size; H1 is the homography transformation matrix; s and H1 can be determined in advance according to actual needs, and there is no specific limitation.

[0070] Using the latitude and longitude (X0, Y0) of the camera installation location as the origin of the world coordinate system, and north as the vertical axis, the coordinate system is determined. Based on the latitude and longitude (X, Y) of the tracking target in the world coordinate system, the relative coordinates (x, y) between the tracking target and the camera installation location are determined, specifically through the following formula:

[0071] y = (Y - Y0) * D * 3600

[0072] x=(X-X0)*D*cos(Y)*3600

[0073] Where (X,Y) are the latitude and longitude of the tracked target in the world coordinate system; (X0,Y0) are the latitude and longitude of the camera installation position; D represents the distance (fixed value) corresponding to one second in m / s; (x,y) are the relative coordinates of the tracked target and the camera installation position. (x,y) is used as the observation value corresponding to the Kalman filter and input into the Kalman filter to update the initial current frame estimate.

[0074] S113. The initial current frame estimate is updated based on the observation value using a Kalman filter to obtain the current frame estimate corresponding to the tracked target.

[0075] The current frame estimate corresponding to the tracked target is obtained by updating the initial current frame estimate based on the observations using a Kalman filter, which can be achieved using the following formula:

[0076]

[0077]

[0078] P t =(IK t H2)P t -

[0079] Among them, K t P is the Kalman gain coefficient; t - Let H1 be the estimation error covariance matrix of the system state at time t; H2 is the observation matrix; and R is the observation noise covariance matrix. This is the estimated value for the current frame; Z is the initial estimate for the current frame; t For observations; P t This is the updated estimated error covariance matrix.

[0080] In one embodiment, the observation noise covariance matrix can be the following matrix:

[0081]

[0082] S121. Based on the horizontal and vertical velocity components included in the current frame estimate, determine the candidate velocities included in the candidate 3D information.

[0083] Candidate velocities can be determined using the following formula:

[0084]

[0085] Among them, v t For candidate velocities; v x The lateral velocity component; v y This represents the longitudinal velocity component.

[0086] S122. Based on the arctangent value corresponding to the ratio of the lateral velocity component and the longitudinal velocity component, determine the candidate heading angle included in the candidate three-dimensional information.

[0087] The candidate heading angle can be determined using the following formula:

[0088]

[0089] Where, θ t For candidate heading angles; v x The lateral velocity component; v y This represents the longitudinal velocity component.

[0090] S123. Convert the horizontal and vertical position coordinates included in the current frame estimate into latitude and longitude values. The horizontal and vertical position coordinates are the coordinates of the tracking target relative to the installation position of the camera.

[0091] The horizontal and vertical position coordinates included in the current frame estimate can be converted into latitude and longitude values ​​using the following formula:

[0092]

[0093]

[0094] Y = dcosθ ÷ D ÷ 3600 + Y0

[0095] X=dsinθ÷(D×cos(Y×π÷180))÷3600+X0

[0096] Where (X,Y) are the latitude and longitude of the tracked target in the world coordinate system, i.e., latitude and longitude values; (X0,Y0) are the latitude and longitude of the camera installation position; D represents the distance corresponding to one second (fixed value) in m / s; d is the distance from the tracked target to the camera; θ is the angle between the tracked target and due north; (x,y) are the relative coordinates of the tracked target and the camera installation position, where x is the horizontal position coordinate and y is the vertical position coordinate.

[0097] S124. Determine the latitude and longitude values ​​as the location of the tracking target included in the candidate 3D information.

[0098] S131. When the candidate three-dimensional information meets the set optimization conditions, the first mean value of the three-dimensional information corresponding to each frame in the sliding window is determined as the target velocity corresponding to the tracked target in the current frame. The first mean value is the mean value of the velocity of the tracked target included in each three-dimensional information.

[0099] S132. The second mean value of the three-dimensional information corresponding to each frame in the sliding window is determined as the target heading angle of the tracked target in the current frame. The second mean value is the mean value of the heading angle of the tracked target included in each three-dimensional information.

[0100] S133. The target velocity, target heading angle, and position of the tracked target are determined as the target three-dimensional information corresponding to the tracked target in the current frame.

[0101] Steps S131 to S133 can be combined and understood as follows: when the candidate 3D information meets the set optimization conditions, the position of the tracking target included in the candidate 3D information is determined as the position of the tracking target included in the target 3D information; the average value of the velocity of the tracking target included in the 3D information of each frame in the sliding window (i.e., the first average value) is determined as the target velocity included in the target 3D information; and the average value of the heading angle of the tracking target included in the 3D information of each frame in the sliding window (i.e., the second average value) is determined as the target heading angle included in the target 3D information.

[0102] S140. When the candidate 3D information does not meet the set optimization conditions, the candidate 3D information is determined as the target 3D information.

[0103] The technical solution of this invention, when the candidate 3D information meets the set optimization conditions, determines the average velocity of the tracked target included in the 3D information of each frame in the sliding window as the target velocity included in the target 3D information, and determines the average heading angle of the tracked target included in the 3D information of each frame in the sliding window as the target heading angle included in the target 3D information; when the candidate 3D information does not meet the set optimization conditions, the candidate 3D information is determined as the target 3D information. By optimizing the candidate 3D information to obtain the target 3D information, the phenomenon of unstable speed of the tracked target or easy jump in the heading angle of the tracked target is optimized, thereby improving the accuracy of visual perception and positioning of the tracked target.

[0104] In one embodiment, the current frame estimate includes a lateral velocity component, a longitudinal velocity component, lateral position coordinates, and longitudinal position coordinates. Correspondingly, after optimizing the candidate 3D information based on multiple historical 3D information to obtain the target 3D information, the method further includes:

[0105] The product of the sine values ​​of the first mean and the second mean is determined as the updated lateral velocity component;

[0106] The product of the cosine values ​​of the first mean and the second mean is determined as the updated longitudinal velocity component;

[0107] The sum of the lateral position coordinates and the first product is used to determine the updated lateral position coordinates. The first product is the product of the updated lateral velocity component and the duration of a single frame.

[0108] The sum of the longitudinal position coordinates and the second product is used to determine the updated longitudinal position coordinates. The second product is the product of the updated longitudinal velocity component and the duration of a single frame.

[0109] The updated lateral velocity component, the updated longitudinal velocity component, the updated lateral position coordinates, and the updated longitudinal position coordinates are determined as the updated current frame estimates for use in the next frame estimation.

[0110] After optimizing the candidate 3D information based on multiple historical 3D information to obtain the target 3D information, the current frame estimate output by the Kalman filter can be updated so that the updated current frame estimate can be used for the Kalman filter estimation in the next frame. This can be achieved through the following formula:

[0111] v' x =v mean sinθ mean

[0112] v' y =v mean cosθ mean

[0113] x t '=x t +v' x dt

[0114] y t '=y t +v' y dt

[0115] Among them, v' x For the updated lateral velocity component; v' y The updated longitudinal velocity component; v mean The first mean; θ mean The second mean; x t ' represents the updated horizontal position coordinates; x t y is the horizontal position coordinate; dt is the duration of a single frame, i.e., the time difference between two consecutive frames; t' represents the updated vertical position coordinates; y t The coordinates are the vertical position coordinates.

[0116] The updated lateral velocity component, updated longitudinal velocity component, updated lateral position coordinates, and updated longitudinal position coordinates are determined as the updated current frame estimate. The updated current frame estimate can be expressed as:

[0117] In one embodiment, after optimizing the candidate 3D information based on multiple historical 3D information to obtain the target 3D information, the method further includes:

[0118] Determine the first variance, which is the variance of the velocity of the tracked target included in the three-dimensional information corresponding to each frame in the sliding window;

[0119] Determine the second variance, which is the variance of the heading angle of the tracked target included in each three-dimensional information;

[0120] The first and second variances are weighted and summed.

[0121] The observation noise covariance matrix of the Kalman filter is updated using the weighted summation result for estimation in the next frame.

[0122] After optimizing the candidate 3D information based on multiple historical 3D information to obtain the target 3D information, the observation noise covariance matrix of the Kalman filter can be updated. The updated observation noise covariance matrix can then be used for the estimation of the Kalman filter in the next frame. This can be achieved through the following formula:

[0123] δ=w1v var +w2θ var

[0124]

[0125] Where R' is the updated observation noise covariance matrix; δ is a parameter in the observation noise covariance matrix that can be used for updating; v var θ is the first variance; var The second variance is represented by w1 and w2, which are two weights. The sum of w1 and w2 is 1, and the specific weights can be determined according to actual needs.

[0126] In one embodiment, before determining the relative coordinates between the tracking target and the camera's mounting position in the current frame, the camera can be calibrated first, i.e., the homography transformation matrix H1 from the camera to the world coordinate system can be determined, as follows:

[0127] 1) Data Acquisition

[0128] First, install a camera on any pole along the roadside, adjust the camera's angle and focal length, and set the resolution. Then, use the Global Positioning System (GPS) to capture multiple images within the camera's field of view and record the latitude and longitude (X, Y) data of the corresponding location at the time of image capture.

[0129] 2) Calibration point selection

[0130] Using a calibration tool, the pixel coordinates (u,v) of a specified point are obtained by manually tackling the points, and then the latitude and longitude data (X,Y) corresponding to the pixel coordinates of each specified point are determined.

[0131] 3) Calculate the homography transformation matrix H1 from the camera to the world coordinate system.

[0132] Through the above steps, the three-dimensional spatial coordinates (X,Y) have been obtained, which will be transformed into secondary coordinates (X,Y,1); the two-dimensional projection point coordinates (u,v) of the camera have been obtained, which will be transformed into secondary coordinates (u,v,1); given the perspective projection model, we can obtain:

[0133]

[0134]

[0135] Here, s is a scale factor of arbitrary size; given s, (X,Y,1) and (u,v,1), the homography transformation matrix H1 can be determined.

[0136] Specifically, by transforming the above formula, we can obtain:

[0137]

[0138]

[0139] The H1 matrix contains 9 unknowns, but due to the existence of the scaling factor, h can be set to... 33 =1, then the homography transformation matrix has only 8 degrees of freedom. Therefore, if there are 4 pairs of non-collinear corresponding points, a unique solution to the equation can be found. However, due to noise, more than 4 pairs of points are usually used to solve the problem. When the number of equations exceeds the number of unknowns, it becomes an overdetermined system of equations, which can be solved using least squares, i.e.:

[0140]

[0141] The above formula can also be expressed as:

[0142] Ah = 0

[0143] Where A is a 2N*9 matrix, N is the number of matched point pairs; h is the matrix to be solved; (x i ',y i (x, y) is a point (X, Y) on the three-dimensional plane Z = 0; i ,y i Let (u, v) be the pixel coordinates. The homogeneous system of equations can be solved using Singular Value Decomposition (SVD), and can be expressed as the following optimization problem:

[0144] min(||Ah|| 2 )

[0145] The derivation yields:

[0146] ||Ah|| 2 =||U∑VTh|| 2 =||∑VTh|| 2

[0147] In the above formula, let y = V T h, then we can obtain:

[0148] min||Ah|| 2 =min||∑y|| 2

[0149] We can obtain y = [0,...,1] T Therefore, h equals the last column of V, and thus the homography transformation matrix H1 can be determined.

[0150] In one embodiment, one or more tracking targets can be placed in a tracking list. For each tracking target in the tracking list, visual perception positioning can be performed using the visual perception positioning method provided in this embodiment of the invention. Each time visual perception positioning is required, the identifier of the tracking target determined by the tracking algorithm is used to query the tracking list. If the tracking target is not found, it is identified as a new tracking target and added to the tracking list. If the tracking target is found, its 3D information is updated. If the number of times a tracking target is lost exceeds a certain threshold, the tracking target is deleted from the tracking list.

[0151] Example 3

[0152] Figure 3 This is a schematic diagram of a visual perception positioning device according to Embodiment 3 of the present invention. This embodiment is applicable to situations involving visual perception positioning, such as... Figure 3 As shown, the specific structure of the device includes:

[0153] Input module 21 is used to input the relative coordinates of the tracking target and the camera's installation position in the current frame into the Kalman filter to obtain the current frame estimate of the tracking target;

[0154] The first determining module 22 is used to determine candidate three-dimensional information corresponding to the tracked target in the current frame based on the current frame estimation value. The candidate three-dimensional information includes at least the position, candidate velocity, and candidate heading angle of the tracked target.

[0155] The second determining module 23 is used to optimize the candidate three-dimensional information according to multiple historical three-dimensional information to obtain target three-dimensional information when the candidate three-dimensional information meets the set optimization conditions. The multiple historical three-dimensional information includes the three-dimensional information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame.

[0156] The third determining module 24 is used to determine the candidate three-dimensional information as the target three-dimensional information when the candidate three-dimensional information does not meet the set optimization conditions.

[0157] The visual perception positioning device provided in this embodiment first inputs the relative coordinates of the tracking target and the camera's installation position in the current frame into a Kalman filter via an input module to obtain an estimated value of the tracking target in the current frame. Then, a first determining module determines candidate 3D information corresponding to the tracking target in the current frame based on the estimated value of the current frame. The candidate 3D information includes at least the position, candidate velocity, and candidate heading angle of the tracking target. Then, a second determining module optimizes the candidate 3D information based on multiple historical 3D information when the candidate 3D information meets the set optimization conditions to obtain target 3D information. The multiple historical 3D information includes the 3D information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame. Finally, a third determining module determines the candidate 3D information as the target 3D information when the candidate 3D information does not meet the set optimization conditions.

[0158] Furthermore, the optimization conditions include one or more of the following conditions:

[0159] The difference between the candidate velocity and the velocity of the tracked target in the previous frame is greater than a first set threshold;

[0160] The candidate speed is less than a second preset threshold;

[0161] The candidate speed is greater than a third preset threshold;

[0162] The difference between the candidate heading angle and the heading angle of the tracked target in the previous frame is greater than the fourth set threshold.

[0163] Wherein, the second set threshold is less than the third set threshold.

[0164] Furthermore, the second determining module 23 is specifically used for:

[0165] The first mean value corresponding to the three-dimensional information of each frame in the sliding window is determined as the target velocity of the tracked target in the current frame. The first mean value is the mean value of the velocity of the tracked target included in each of the three-dimensional information.

[0166] The second mean value corresponding to the three-dimensional information of each frame in the sliding window is determined as the target heading angle of the tracked target in the current frame. The second mean value is the mean value of the heading angle of the tracked target included in each of the three-dimensional information.

[0167] The target velocity, the target heading angle, and the position of the tracked target are determined as the target three-dimensional information corresponding to the tracked target in the current frame.

[0168] Furthermore, the current frame estimate includes lateral velocity components, longitudinal velocity components, lateral position coordinates, and longitudinal position coordinates. Correspondingly, the device also includes:

[0169] The fourth determining module is used to determine the updated lateral velocity component by multiplying the sine of the first mean and the second mean after optimizing the candidate 3D information based on multiple historical 3D information to obtain the target 3D information.

[0170] The fifth determining module is used to determine the product of the cosine of the first mean and the second mean as the updated longitudinal velocity component;

[0171] The sixth determining module is used to determine the updated horizontal position coordinates by summing the horizontal position coordinates with the first product, where the first product is the product of the updated horizontal velocity component and the duration of a single frame.

[0172] The seventh determining module is used to determine the updated longitudinal position coordinates by summing the longitudinal position coordinates and the second product, where the second product is the product of the updated longitudinal velocity component and the duration of a single frame.

[0173] The eighth determining module is used to determine the updated lateral velocity component, the updated longitudinal velocity component, the updated lateral position coordinates, and the updated longitudinal position coordinates as the updated current frame estimate values ​​for use in the estimation of the next frame.

[0174] Furthermore, the device also includes:

[0175] The first variance determination module is used to determine the first variance after optimizing the candidate three-dimensional information based on multiple historical three-dimensional information to obtain the target three-dimensional information. The first variance is the variance of the velocity of the tracked target included in the three-dimensional information corresponding to each frame in the sliding window.

[0176] The second variance determination module is used to determine the second variance, which is the variance of the heading angle of the tracked target included in each of the three-dimensional information.

[0177] The weighted summation module is used to perform a weighted summation of the first variance and the second variance;

[0178] An update module is used to update the observation noise covariance matrix of the Kalman filter using the result of the weighted summation for estimation in the next frame.

[0179] Furthermore, the first determining module 22 is specifically used for:

[0180] Based on the lateral and longitudinal velocity components included in the current frame estimate, the candidate velocities included in the candidate 3D information are determined.

[0181] Based on the arctangent value corresponding to the ratio of the lateral velocity component and the longitudinal velocity component, the candidate heading angles included in the candidate three-dimensional information are determined.

[0182] The horizontal and vertical position coordinates included in the current frame estimate are converted into latitude and longitude values, where the horizontal and vertical position coordinates are the coordinates of the tracking target relative to the installation position of the camera;

[0183] The latitude and longitude values ​​are determined as the location of the tracking target included in the candidate three-dimensional information.

[0184] Furthermore, input module 21 is specifically used for:

[0185] The initial current frame estimate is obtained by using the Kalman filter based on the previous frame estimate;

[0186] The relative coordinates between the tracked target and the camera's installation position in the current frame are used as the observation values ​​corresponding to the Kalman filter;

[0187] The initial current frame estimate is updated using the Kalman filter based on the observations to obtain the current frame estimate corresponding to the tracked target.

[0188] The visual perception positioning device provided in the embodiments of the present invention can execute the visual perception positioning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0189] Example 4

[0190] Figure 4 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0191] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0192] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0193] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as visual perception localization methods.

[0194] In some embodiments, the visual perception localization method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the visual perception localization method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the visual perception localization method by any other suitable means (e.g., by means of firmware).

[0195] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0196] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0197] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0198] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0199] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0200] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0201] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0202] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A visual perception localization method, characterized in that, include: The relative coordinates of the tracking target and the camera's installation position in the current frame are input into the Kalman filter to obtain the estimated value of the tracking target in the current frame. Based on the current frame estimation value, candidate 3D information corresponding to the tracked target in the current frame is determined, and the candidate 3D information includes at least the position, candidate velocity, and candidate heading angle of the tracked target; When the candidate 3D information meets the set optimization conditions, the candidate 3D information is optimized based on multiple historical 3D information to obtain the target 3D information. The multiple historical 3D information includes the 3D information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame. When the candidate 3D information does not meet the set optimization conditions, the candidate 3D information is determined as the target 3D information; The process, after optimizing the candidate 3D information based on multiple historical 3D information to obtain the target 3D information, further includes: Determine the first variance, which is the variance of the velocity of the tracked target included in the three-dimensional information corresponding to each frame in the sliding window; Determine the second variance, which is the variance of the heading angle of the tracked target included in each of the three-dimensional information; The first variance and the second variance are weighted and summed. The observation noise covariance matrix of the Kalman filter is updated using the weighted summation result for estimation in the next frame.

2. The method according to claim 1, characterized in that, The optimization conditions include one or more of the following conditions: The difference between the candidate velocity and the velocity of the tracked target in the previous frame is greater than a first set threshold; The candidate speed is less than a second preset threshold; The candidate speed is greater than a third preset threshold; The difference between the candidate heading angle and the heading angle of the tracked target in the previous frame is greater than the fourth set threshold. Wherein, the second set threshold is less than the third set threshold.

3. The method according to claim 1, characterized in that, The target 3D information is obtained by optimizing the candidate 3D information based on multiple historical 3D information sources, including: The first mean value corresponding to the three-dimensional information of each frame in the sliding window is determined as the target velocity of the tracked target in the current frame. The first mean value is the mean value of the velocity of the tracked target included in each of the three-dimensional information. The second mean value corresponding to the three-dimensional information of each frame in the sliding window is determined as the target heading angle of the tracked target in the current frame. The second mean value is the mean value of the heading angle of the tracked target included in each of the three-dimensional information. The target velocity, the target heading angle, and the position of the tracked target are determined as the target three-dimensional information corresponding to the tracked target in the current frame.

4. The method according to claim 3, characterized in that, The current frame estimate includes lateral velocity components, longitudinal velocity components, lateral position coordinates, and longitudinal position coordinates. Correspondingly, after optimizing the candidate 3D information based on multiple historical 3D information to obtain the target 3D information, it also includes: The product of the first mean and the sine of the second mean is determined as the updated lateral velocity component; The product of the first mean and the cosine of the second mean is determined as the updated longitudinal velocity component; The sum of the lateral position coordinates and the first product is determined as the updated lateral position coordinates, where the first product is the product of the updated lateral velocity component and the duration of a single frame. The sum of the longitudinal position coordinates and the second product is determined as the updated longitudinal position coordinates, where the second product is the product of the updated longitudinal velocity component and the duration of a single frame. The updated lateral velocity component, the updated longitudinal velocity component, the updated lateral position coordinates, and the updated longitudinal position coordinates are determined as the updated current frame estimates for use in the estimation of the next frame.

5. The method according to claim 1, characterized in that, Based on the current frame estimation value, candidate 3D information corresponding to the tracked target in the current frame is determined, including: Based on the lateral and longitudinal velocity components included in the current frame estimate, the candidate velocities included in the candidate 3D information are determined. Based on the arctangent value corresponding to the ratio of the lateral velocity component and the longitudinal velocity component, the candidate heading angles included in the candidate three-dimensional information are determined. The horizontal and vertical position coordinates included in the current frame estimate are converted into latitude and longitude values, where the horizontal and vertical position coordinates are the coordinates of the tracking target relative to the installation position of the camera; The latitude and longitude values ​​are determined as the location of the tracking target included in the candidate three-dimensional information.

6. The method according to claim 1, characterized in that, The relative coordinates of the tracked target and the camera's mounting position in the current frame are input into a Kalman filter to obtain the estimated value of the tracked target in the current frame, including: The initial current frame estimate is obtained by using the Kalman filter based on the previous frame estimate; The relative coordinates between the tracked target and the camera's installation position in the current frame are used as the observation values ​​corresponding to the Kalman filter; The initial current frame estimate is updated using the Kalman filter based on the observations to obtain the current frame estimate corresponding to the tracked target.

7. A visual perception positioning device, characterized in that, include: The input module is used to input the relative coordinates of the tracking target and the camera's installation position in the current frame into the Kalman filter to obtain the current frame estimate of the tracking target; The first determining module is used to determine candidate three-dimensional information corresponding to the tracked target in the current frame based on the current frame estimation value, wherein the candidate three-dimensional information includes at least the position, candidate velocity, and candidate heading angle of the tracked target; The second determining module is used to optimize the candidate three-dimensional information based on multiple historical three-dimensional information to obtain target three-dimensional information when the candidate three-dimensional information meets the set optimization conditions. The multiple historical three-dimensional information includes the three-dimensional information corresponding to each frame in a sliding window of a set size adjacent to the current frame before the current frame. The third determining module is used to determine the candidate three-dimensional information as the target three-dimensional information when the candidate three-dimensional information does not meet the set optimization conditions; The device further includes: The first variance determination module is used to determine the first variance after optimizing the candidate three-dimensional information based on multiple historical three-dimensional information to obtain the target three-dimensional information. The first variance is the variance of the velocity of the tracked target included in the three-dimensional information corresponding to each frame in the sliding window. The second variance determination module is used to determine the second variance, which is the variance of the heading angle of the tracked target included in each of the three-dimensional information. The weighted summation module is used to perform a weighted summation of the first variance and the second variance; An update module is used to update the observation noise covariance matrix of the Kalman filter using the result of the weighted summation for estimation in the next frame.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images