A method and apparatus for determining a perspective matrix, an electronic device, and a storage medium

Abstract

The application discloses a perspective matrix determination method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining an initial perspective matrix between a video coordinate system and a radar coordinate system calibrated in advance; if it is determined that a target point meets a target tracking condition, determining a deviation value of the target point matched with the initial perspective matrix; recalibrating the initial perspective matrix according to the deviation value of each target point to obtain an intermediate perspective matrix; repeatedly performing the operation of determining the deviation value of the target point matched with the intermediate perspective matrix and recalibrating the intermediate perspective matrix according to the deviation value of each target point until it is determined that the intermediate perspective matrix meets a matrix precision condition, and taking the intermediate perspective matrix as a target perspective matrix. The application repeatedly calibrates the intermediate perspective matrix until the intermediate perspective matrix meets the matrix precision condition, and then determines the target perspective matrix, so that the calibration cost of the perspective matrix is reduced, and the accuracy of the perspective matrix is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for determining a perspective matrix. Background Technology

[0002] The advent of the radar-video integrated machine represents a significant breakthrough in road traffic management technology. This machine can fuse target information acquired by radar with that acquired through video. However, the target position coordinates acquired by radar and the target pixel coordinates detected by video are in two different coordinate systems. Therefore, to fuse these two pieces of information, a transformation relationship, or perspective matrix, needs to be determined.

[0003] In existing technologies, the perspective matrix is ​​obtained through calibration by determining several location points and the corresponding pixel coordinates in the video during the installation of the radar-video integrated machine. However, this method requires determining as many location points as possible, resulting in a large workload during calibration. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for determining a perspective matrix, so as to reduce the calibration cost of the perspective matrix and improve its accuracy.

[0005] In a first aspect, embodiments of the present invention provide a method for determining a perspective matrix, the method comprising:

[0006] Obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system;

[0007] If the target point is determined to meet the target tracking conditions, then the deviation value between the target point and the initial perspective matrix is ​​determined.

[0008] Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated to obtain the intermediate perspective matrix;

[0009] Repeatedly execute the operation of determining the deviation value between the target point and the intermediate perspective matrix, and recalibrating the intermediate perspective matrix according to the deviation value of each target point until it is determined that the intermediate perspective matrix meets the matrix accuracy condition, and then use the intermediate perspective matrix as the target perspective matrix.

[0010] Secondly, embodiments of the present invention also provide a device for determining a perspective matrix, the device comprising:

[0011] The first matrix determination module is used to obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system;

[0012] The deviation value determination module is used to determine the deviation value between the target point and the initial perspective matrix if the target point is determined to meet the target tracking conditions.

[0013] The second matrix determination module is used to recalibrate the initial perspective matrix based on the deviation values ​​of each target point to obtain the intermediate perspective matrix;

[0014] The target matrix determination module is used to repeatedly execute the operation of determining the deviation value between the target point and the intermediate perspective matrix, and recalibrating the intermediate perspective matrix according to the deviation value of each target point until it is determined that the intermediate perspective matrix meets the matrix accuracy condition, and then using the intermediate perspective matrix as the target perspective matrix.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for determining the perspective matrix as described in any of the embodiments of the present invention.

[0016] Fourthly, embodiments of the present invention also provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the method for determining a perspective matrix as described in any of the embodiments of the present invention.

[0017] The technical solution of this invention obtains an initial perspective matrix between a pre-calibrated video coordinate system and a radar coordinate system, then acquires target points and determines that the target points meet the target tracking conditions. Next, it determines the deviation value between the target points and the initial perspective matrix to ensure relatively stable target point tracking. Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated to obtain an intermediate perspective matrix. This ensures smaller and more accurate target point errors, reduces the number of calibrations and time required, and avoids inaccuracies in the intermediate perspective matrix due to the randomness of target points, ensuring a more accurate intermediate perspective matrix. Finally, the process of determining the deviation value between the target points and the intermediate perspective matrix and recalibrating the intermediate perspective matrix based on the deviation values ​​of each target point is repeated until the intermediate perspective matrix meets the matrix accuracy conditions. The intermediate perspective matrix is ​​then used as the target perspective matrix, thereby reducing the calibration cost of the perspective matrix and improving its accuracy.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a method for determining a perspective matrix provided in Embodiment 1 of the present invention;

[0021] Figure 2 This is a flowchart of a method for determining a perspective matrix provided in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of a perspective matrix determination device provided in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] Example 1

[0027] Figure 1This is a flowchart of a perspective matrix determination method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation where the transformation relationship (i.e., perspective matrix) obtained by automatically calibrating a target detected by video and a target detected by radar is accurately determined. This method can be executed by a perspective matrix determination device, which can be implemented in hardware and / or software. The perspective matrix determination device can be configured in an electronic device with a perspective matrix determination method.

[0028] like Figure 1 As shown, the method includes:

[0029] S110. Obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system.

[0030] The proposed solution utilizes a millimeter-wave radar-video integrated machine to simultaneously acquire video and radar information of the calibration point. The coordinate system of the calibration point in the video information is the video coordinate system, and the coordinate system in the radar information is the radar coordinate system. Therefore, in order to fuse the two types of information together, it is necessary to determine a transformation relationship, namely the perspective matrix.

[0031] Optionally, the initial perspective matrix is ​​pre-calibrated based on the radar coordinates and video coordinates of at least four calibration points.

[0032] For example, during the initial calibration, calibration points can be set at four points on the left and right sides of the installation site of the millimeter-wave radar video integrated camera, at a certain distance. For instance, calibration points could be located at 25 meters and 70 meters, with four positions on the left and right sides of the screen. The video coordinates and radar coordinates of these four calibration points can then be obtained. After calibrating these four points, the initial perspective matrix between the video coordinate system and the radar coordinate system can be obtained. The initial perspective matrix can be represented as:

[0033]

[0034] Where X, Y, and Z represent radar coordinates, and x and y represent video coordinates.

[0035] By transforming the above formula, we can obtain the following formula:

[0036]

[0037] Since c3 = 1, the perspective matrix has a total of 8 unknowns: a1, a2, a3, b1, b2, b3, c1, and c2. The solution to the initial perspective matrix can be obtained using four points (each point can form two equations).

[0038] S120. If it is determined that the target point meets the target tracking conditions, then determine the deviation value between the target point and the initial perspective matrix.

[0039] The target tracking conditions may include the following three conditions: the target point appears continuously in the video coordinate system for a preset number of frames; a position point that matches the target point can be determined in the radar coordinate system; and the deviation between the transformed coordinates of the target point after the video coordinate transformation and the radar coordinates of the position point conforms to a normal distribution.

[0040] In this process, video frames and radar frames are acquired synchronously, meaning they are captured at the same time. The number of consecutive frames in which a target point appears in the video coordinate system is limited. This is to ensure that a sufficient number of video coordinates of the target point are acquired, allowing for the determination of its position in the radar coordinate system through coordinate transformation and to determine the deviation value. The position point in the radar coordinate system that matches the target point is identified. This is because if the radar coordinates do not include a matching position point, it indicates a large error in the initial perspective matrix, and this anomaly needs to be ruled out. The deviation value is ensured to conform to a normal distribution. This is because if it does not conform to a normal distribution, it indicates unstable error, and the error of the perspective matrix must be kept within the range of random error.

[0041] Specifically, target points are acquired from multiple video frame images, and the target points are analyzed and processed. Target points that meet the target tracking conditions are retained. This is to ensure that the determined target points are accurate, so as to avoid deviations in subsequent calculations caused by inaccurate target points. In this way, the deviation value of the target point matching the initial perspective matrix can be accurately determined.

[0042] S130. Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated to obtain the intermediate perspective matrix.

[0043] Specifically, step 120 can determine the deviation values ​​of multiple target points, and then the deviation values ​​of these multiple points can be calculated and analyzed to recalibrate the initial perspective matrix.

[0044] Optionally, the deviation values ​​of multiple target points also conform to a normal distribution. For the deviation values ​​of target points that exceed the normal distribution μ±3σ, it is considered that their deviation values ​​have large errors. Therefore, the corresponding target points can be removed, and the retained target points can be used to recalibrate the initial perspective matrix.

[0045] In one feasible embodiment, optionally, the initial perspective matrix is ​​recalibrated based on the deviation values ​​of each target point, including steps A1-A2:

[0046] Step A1: Determine the standard deviation of the deviation values ​​of each target point based on the deviation values ​​of each target point.

[0047] Step A2: Recalibrate the initial perspective matrix based on the target points where the standard deviation of the deviation value is less than the preset standard deviation.

[0048] The preset standard deviation can be a standard deviation boundary value specifically set according to the actual situation in order to make the obtained target points more accurate.

[0049] Specifically, the deviation values ​​of multiple target points are obtained, and then the deviation values ​​of each target point are calculated to accurately determine the standard deviation of the deviation value of each target point. The standard deviation of the deviation value of each target point is compared with the preset standard deviation. Since the target points with a standard deviation of less than the preset standard deviation are more stable in tracking, the target points with a standard deviation of less than the preset standard deviation are retained to ensure that the selection of target points is more accurate, so that the recalibration of the initial perspective matrix using these target points can be more accurate.

[0050] This technical solution compares the preset standard deviation with the standard deviation of each target point, successfully selecting target points that meet the requirements for recalibration. This ensures that the target point error is smaller and more accurate, further guaranteeing that the initial perspective matrix can be correctly recalibrated using these points. This reduces the number of calibrations and the time required, avoids inaccuracies in the intermediate perspective matrix due to the randomness of the target points, and ensures that the obtained intermediate perspective matrix is ​​more accurate.

[0051] Optionally, after obtaining the target point for recalibration, the target point can be compared with the original calibration point, in the following two cases:

[0052] a. If the video coordinates of the target point are within the preset range of the video coordinates of the original calibration point, then replace the original calibration point with the target point and recalibrate to obtain the intermediate perspective matrix.

[0053] b. If the video coordinates of the target point are outside the preset range of the video coordinates of the original calibration point, the target point will be used as an extended calibration point and the intermediate perspective matrix will be calculated together with the original calibration point.

[0054] It is worth noting that, since the number of points used to calculate the intermediate perspective matrix has increased, the number of equations is greater than the number of unknowns, and a solution cannot be obtained directly, a solution needs to be fitted using the least squares method.

[0055] Assuming the overdetermined matrix Xθ = Y, the solution using the least squares method can be derived as follows:

[0056] θ=(X T X) -1 X T Y;

[0057] This embodiment does not limit the number of extended calibration points; alternatively, the total number of extended calibration points can be limited to 16.

[0058] S140. Repeatedly execute the operation of determining the deviation value between the target point and the intermediate perspective matrix, and recalibrating the intermediate perspective matrix according to the deviation value of each target point until it is determined that the intermediate perspective matrix meets the matrix accuracy condition, and use the intermediate perspective matrix as the target perspective matrix.

[0059] Among them, the matrix accuracy condition can be the condition that the perspective matrix does not need to be recalibrated again, that is, the perspective matrix has reached the required accuracy.

[0060] Specifically, after obtaining the intermediate perspective matrix, it's necessary to determine if it meets the matrix accuracy requirements. If it does, the intermediate perspective matrix is ​​sufficiently accurate, and it's used as the target perspective matrix, ending the calibration process. If it doesn't, new target points need to be obtained, and the process of determining the deviation value between the target point and the intermediate perspective matrix is ​​repeated. Based on the deviation value of each target point, the intermediate perspective matrix is ​​recalibrated until the matrix accuracy requirements are met. During this repeated process of determining the deviation value between the target point and the intermediate perspective matrix and recalibrating the intermediate perspective matrix based on the deviation value of each target point, target points need to be reselected for deviation value calculation. This ensures that the final matrix accuracy is independent of the overall deviation of the data in any given instance.

[0061] Optionally, the intermediate perspective matrix can be determined to meet the matrix accuracy condition in the following way: calculate the target value of the deviation between each target point and the previous intermediate perspective matrix of the current intermediate perspective matrix, and the difference between the target value of the deviation between each target point and the target value of the deviation between the target point and the current intermediate perspective matrix; if the difference is determined to be less than or equal to a preset difference, then the current intermediate perspective matrix is ​​determined to meet the matrix accuracy condition; wherein, the target value of the deviation includes the average value of the deviation and the standard deviation of the deviation.

[0062] In this embodiment, if the target deviation value of each target point in the current intermediate perspective matrix is ​​less than the target deviation value of each target point in the previous intermediate perspective matrix, and the target deviation values ​​of the perspective matrices obtained from the two rounds of calibration are relatively close, it indicates that the repeated calibration process is approaching convergence, the accuracy of the current intermediate perspective matrix is ​​already quite good, and repeated calibration can be stopped, and the current intermediate perspective matrix can be used as the final target perspective matrix.

[0063] Optionally, the solution in this application can be converted from radar coordinates to video coordinates, or vice versa. This application takes the conversion of video coordinates to radar coordinates as an example.

[0064] Optionally, taking the conversion of radar coordinates to video coordinates as an example, the process can be as follows: Obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system; determine the radar coordinates of the target point in the first number of consecutive radar frame images; perform coordinate transformation on the first number of radar coordinates of the target point according to the initial perspective matrix to obtain transformed coordinates that match each radar coordinate; determine the position points in each video frame image that match the transformed coordinates of the target point according to each transformed coordinate of the target point and each position point in the first number of video frame images; if the deviation values ​​between the first number of transformed coordinates of the target point and the first number of video coordinates of the position points that match the target point conform to a normal distribution, then the target point is determined to meet the target tracking condition; then determine the deviation value between the target point and the initial perspective matrix; recalibrate the initial perspective matrix according to the deviation values ​​of each target point to obtain an intermediate perspective matrix; repeat the operation of determining the deviation value between the target point and the intermediate perspective matrix and recalibrating the intermediate perspective matrix according to the deviation values ​​of each target point until the intermediate perspective matrix meets the matrix accuracy condition, and use the intermediate perspective matrix as the target perspective matrix.

[0065] The technical solution of this invention obtains an initial perspective matrix between a pre-calibrated video coordinate system and a radar coordinate system, then acquires target points and determines that the target points meet the target tracking conditions. Next, it determines the deviation value between the target points and the initial perspective matrix to ensure relatively stable target point tracking. Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated to obtain an intermediate perspective matrix. This ensures smaller and more accurate target point errors, reduces the number of calibrations and time required, and avoids inaccuracies in the intermediate perspective matrix due to the randomness of target points, ensuring a more accurate intermediate perspective matrix. Finally, the process of determining the deviation value between the target points and the intermediate perspective matrix and recalibrating the intermediate perspective matrix based on the deviation values ​​of each target point is repeated until the intermediate perspective matrix meets the matrix accuracy conditions. The intermediate perspective matrix is ​​then used as the target perspective matrix, thereby reducing the calibration cost of the perspective matrix and improving its accuracy.

[0066] Example 2

[0067] Figure 2 This is a flowchart of a method for determining a perspective matrix provided by an embodiment of the present invention. Based on the above embodiments, the present invention further specifies the determination of the deviation value between the target point and the initial perspective matrix if the target point is determined to meet the target tracking conditions.

[0068] like Figure 2 As shown, the method includes:

[0069] S210. Obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system.

[0070] The specific process of calibrating to obtain the initial perspective matrix has been described in the above embodiments, and will not be repeated here.

[0071] S220. Determine that the target point meets the target tracking conditions.

[0072] Specifically, a first number of consecutive video frames are acquired, and the video coordinates of the target point in the first number of consecutive video frames are determined. Based on the initial perspective matrix, the first number of video coordinates of the target point are transformed to obtain transformed coordinates that match each video coordinate. At the same time, based on each transformed coordinate of the target point and each position point in the first number of radar frames, the position points in each radar frame that match the transformed coordinates of the target point are determined. Finally, if the deviation values ​​between the first number of transformed coordinates of the target point and the first number of radar coordinates of the position points that match the target point conform to a normal distribution, then the target point is determined to meet the target tracking condition.

[0073] In a feasible embodiment, determining the location points in each radar frame image that match the transformed coordinates of the target point based on the transformed coordinates of the target point and the location points in a first number of radar frame images may include steps B1-B2:

[0074] Step B1: Among the location points in the radar frame image that matches the target video frame image, determine the location points whose distance to the transformed coordinates corresponding to the target point in the target video frame image is less than or equal to a preset distance.

[0075] Step B2: If it is determined that the number of times the target location point appears in the first number of radar frame images meets the confidence condition, then the target location point is taken as the location point that matches the target point.

[0076] The confidence condition can be that the target location point appears most frequently in the first number of radar frame images.

[0077] Specifically, the video coordinates of the target point in the first number of video frames are obtained, and the first number of transformed coordinates of the target point are determined through an initial perspective matrix. Simultaneously, the position points in the first number of radar frames are obtained. Then, the distance between each position point in the radar frame image matching the target video frame image and the transformed coordinates of the target point in the corresponding target video frame image is determined. Position points with distances less than or equal to a preset distance are identified. These position points may exist one, multiple, or zero times. Because there may be one, multiple, or zero position points, there may be multiple different position points. Therefore, if the number of times the target position point appears in the first number of radar frames meets the confidence condition (i.e., the target position point appears most frequently in the first number of radar frames), then this target position point is taken as the position point matching the target point. This method ensures a more accurate matching position point and avoids errors in calculating the target point deviation due to the randomness of the position point.

[0078] Optionally, if multiple target locations all satisfy the confidence condition and appear the same number of times in the first number of radar frames, then the target location closest to the target point's transformed coordinates is selected as the matching location. The closest distance to the target point's transformed coordinates can be either the smallest average distance between the target location's coordinates and the target point's transformed coordinates in the first number of radar frames, or the smallest minimum distance between the target location's coordinates and the target point's transformed coordinates in the first number of radar frames. This embodiment does not impose any limitations on this.

[0079] Optionally, if multiple consecutive target points fail to be tracked during pre-calibration, i.e., the corresponding position point cannot be matched during the confirmation of the target point matching position point, it is considered that there is an anomaly in the calibration of the initial perspective matrix. In this case, the initial perspective matrix needs to be re-calibrated.

[0080] This technical solution, by identifying locations in radar frames that match the target video frame image and whose distance to the transformed coordinates corresponding to the target video frame image is less than or equal to a preset distance, can accurately filter out locations that may meet the target point matching criteria. Then, in the first number of radar frames, the frequency of occurrence of the target location point is determined. When the frequency of occurrence meets the confidence condition, i.e., the frequency of occurrence is the highest, it indicates that a location point matching the target point has been found. This achieves accurate determination of the location point matching the target point and avoids inaccurate matching due to the randomness of the location point.

[0081] S230. Determine the deviation value between the target point and the initial perspective matrix.

[0082] Specifically, the first number of transformed coordinates of the target point and the first number of radar coordinates of the location point matching the target point are obtained. Then, based on the deviation between the first number of transformed coordinates of the target point and the first number of radar coordinates of the location point matching the target point, the average deviation value of the target point is determined.

[0083] Because the deviation values ​​follow a normal distribution, if any deviation value of the target point exceeds the normal distribution range of μ±3σ, these deviation values ​​are discarded, and the average deviation value of the target point is recalculated to reduce the impact of gross errors on the deviation values. Finally, among the deviation values ​​after discarding those exceeding the normal distribution range of μ±3σ, the deviation value that meets the condition of the average deviation value is determined as the deviation value to be matched with the initial perspective matrix. The condition of the average deviation value can be that the difference between the deviation value and the average deviation value is minimized within the normal distribution. That is, the deviation value closest to the average deviation value is taken as the deviation value for matching the target point with the initial perspective matrix.

[0084] S240. Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated to obtain the intermediate perspective matrix.

[0085] In the above embodiments, the process of determining the target point for recalibration based on the deviation value of each target point, and the process of obtaining the perspective matrix by recalibration based on each target point for recalibration have been described, and will not be repeated here.

[0086] S250. Repeatedly execute the operation of determining the deviation value between the target point and the intermediate perspective matrix, and recalibrating the intermediate perspective matrix according to the deviation value of each target point until it is determined that the intermediate perspective matrix meets the matrix accuracy condition, and use the intermediate perspective matrix as the target perspective matrix.

[0087] The technical solution of this invention obtains an initial perspective matrix between a pre-calibrated video coordinate system and a radar coordinate system. Then, based on the transformed coordinates of the target point and the position points in a first number of radar frame images, it determines the position points in each radar frame image that match the transformed coordinates of the target point. This achieves accurate determination of the position points that match the target point, avoiding inaccuracies caused by the randomness of the position points. Furthermore, by determining that the deviation values ​​between the first number of transformed coordinates of the target point and the first number of radar coordinates of the position points that match the target point conform to a normal distribution, it is determined that the target point meets the target tracking conditions. Then, it determines the deviation value between the target point and the initial perspective matrix to ensure that the tracking of the target point is relatively stable. Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated to obtain an intermediate perspective matrix. Finally, the operation of determining the deviation value between the target point and the intermediate perspective matrix and recalibrating the intermediate perspective matrix based on the deviation values ​​of each target point is repeated until the intermediate perspective matrix meets the matrix accuracy conditions. The intermediate perspective matrix is ​​then used as the target perspective matrix, thereby reducing the calibration cost of the perspective matrix and improving its accuracy.

[0088] Example 3

[0089] Figure 3 This is a schematic diagram of a perspective matrix determination device provided in an embodiment of the present invention. Figure 3 As shown, the device includes:

[0090] The first matrix determination module 310 is used to obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system;

[0091] The deviation value determination module 320 is used to determine the deviation value between the target point and the initial perspective matrix if the target point is determined to meet the target tracking conditions.

[0092] The second matrix determination module 330 is used to recalibrate the initial perspective matrix based on the deviation values ​​of each target point to obtain the intermediate perspective matrix;

[0093] The target matrix determination module 340 is used to repeatedly perform the operation of determining the deviation value between the target point and the intermediate perspective matrix, and recalibrating the intermediate perspective matrix according to the deviation value of each target point, until the intermediate perspective matrix is ​​determined to meet the matrix accuracy condition, and then the intermediate perspective matrix is ​​used as the target perspective matrix.

[0094] Optionally, the initial perspective matrix is ​​pre-calibrated based on the radar coordinates and video coordinates of at least four calibration points.

[0095] Optionally, the deviation value determination module includes a first condition determination unit, specifically used for:

[0096] Determine the video coordinates of the target point in a first number of consecutive video frames;

[0097] Based on the initial perspective matrix, the first number of video coordinates of the target point are transformed to obtain the transformed coordinates that match each video coordinate.

[0098] Based on the transformed coordinates of the target point and the position points in the first number of radar frame images, determine the position points in each radar frame image that match the transformed coordinates of the target point;

[0099] If the deviation values ​​between the first number of transformed coordinates of the target point and the first number of radar coordinates of the position point matching the target point conform to a normal distribution, then the target point is determined to meet the target tracking condition.

[0100] Optionally, the condition determination unit includes a location point determination unit, specifically used for:

[0101] Among the locations of radar frame images that match the target video frame image, determine the locations where the distance between the transformed coordinates corresponding to the target video frame image and the location is less than or equal to a preset distance.

[0102] If it is determined that the number of times the target location point appears in the first number of radar frame images meets the confidence condition, then the target location point is taken as the location point that matches the target point.

[0103] Optionally, the deviation value determination module includes a deviation value determination unit, specifically used for:

[0104] The average deviation value of the target point is determined based on the first number of transformed coordinates of the target point and the first number of radar coordinates of the position point that matches the target point.

[0105] Among the various deviation values ​​of the target point, the deviation value that meets the condition of the average deviation value is determined as the deviation value that matches the initial perspective matrix.

[0106] Optional, the second matrix determination module is specifically used for:

[0107] Determine the standard deviation of the deviation values ​​for each target point based on the deviation values ​​of each target point;

[0108] The initial perspective matrix is ​​recalibrated based on the target point where the standard deviation of the deviation value is less than the preset standard deviation.

[0109] Optionally, the target matrix determination module includes a second condition determination unit, specifically used for:

[0110] Calculate the target value of the deviation between each target point and the previous perspective matrix of the intermediate perspective matrix, and the difference between the target value of the deviation between each target point and the intermediate perspective matrix.

[0111] If the difference is determined to be less than or equal to a preset difference, then the intermediate perspective matrix is ​​determined to satisfy the matrix precision condition.

[0112] The target deviation value includes the average deviation value and the standard deviation of the deviation value.

[0113] The perspective matrix determination device provided in the embodiments of the present invention can execute the perspective matrix determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0114] Example 4

[0115] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0116] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0117] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0118] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the method for determining the perspective matrix.

[0119] In some embodiments, the method for determining the perspective matrix may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for determining the perspective matrix described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the method for determining the perspective matrix by any other suitable means (e.g., by means of firmware).

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0124] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0125] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0126] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining a perspective matrix, characterized in that, include: Obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system; If the target point is determined to meet the target tracking conditions, then the deviation value between the target point and the initial perspective matrix is ​​determined. Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated to obtain the intermediate perspective matrix; Repeat the process of determining the deviation value between the target point and the intermediate perspective matrix, and recalibrating the intermediate perspective matrix according to the deviation value of each target point until the intermediate perspective matrix meets the matrix accuracy condition, and then use the intermediate perspective matrix as the target perspective matrix. The determination of the deviation value between the target point and the initial perspective matrix includes: Obtain a first number of transformed coordinates of the target point, determine the position points in each radar frame image that match the transformed coordinates of the target point, and obtain a first number of radar coordinates of the position points that match the target point; Based on a first number of transformed coordinates of the target point and a first number of radar coordinates of the position point matching the target point, the deviation value between the target point and the initial perspective matrix is ​​determined.

2. The method according to claim 1, characterized in that, The initial perspective matrix is ​​obtained by pre-calibrating based on the radar coordinates and video coordinates of at least four calibration points.

3. The method according to claim 1, characterized in that, Determine if the target point meets the target tracking conditions, including: Determine the video coordinates of the target point in a first number of consecutive video frames; Based on the initial perspective matrix, the first number of video coordinates of the target point are transformed to obtain the transformed coordinates that match each video coordinate. Based on the transformed coordinates of the target point and the position points in the first number of radar frame images, determine the position points in each radar frame image that match the transformed coordinates of the target point; If the deviation values ​​between the first number of transformed coordinates of the target point and the first number of radar coordinates of the position point matching the target point conform to a normal distribution, then the target point is determined to meet the target tracking condition.

4. The method according to claim 3, characterized in that, Based on the transformed coordinates of the target point and the position points in the first number of radar frame images, determine the position points in each radar frame image that match the transformed coordinates of the target point, including: Among the locations of radar frame images that match the target video frame image, determine the locations where the distance between the transformed coordinates corresponding to the target video frame image and the location is less than or equal to a preset distance. If it is determined that the number of times the target location point appears in the first number of radar frame images meets the confidence condition, then the target location point is taken as the location point that matches the target point.

5. The method according to claim 3, characterized in that, Determining the deviation value between the target point and the initial perspective matrix includes: The average deviation value of the target point is determined based on the first number of transformed coordinates of the target point and the first number of radar coordinates of the position point that matches the target point. Among the various deviation values ​​of the target point, the deviation value that meets the condition of the average deviation value is determined as the deviation value that matches the initial perspective matrix.

6. The method according to claim 1, characterized in that, Based on the deviation values ​​of each target point, the initial perspective matrix is ​​recalibrated, including: Determine the standard deviation of the deviation values ​​for each target point based on the deviation values ​​of each target point; The initial perspective matrix is ​​recalibrated based on the target point where the standard deviation of the deviation value is less than the preset standard deviation.

7. The method according to claim 1, characterized in that, Determining that the intermediate perspective matrix satisfies the matrix precision condition includes: Calculate the target value of the deviation between each target point and the previous perspective matrix of the intermediate perspective matrix, and the difference between the target value of the deviation between each target point and the intermediate perspective matrix. If the difference is determined to be less than or equal to a preset difference, then the intermediate perspective matrix is ​​determined to satisfy the matrix precision condition. The target deviation value includes the average deviation value and the standard deviation of the deviation value.

8. A device for determining a perspective matrix, characterized in that, include: The first matrix determination module is used to obtain the initial perspective matrix between the pre-calibrated video coordinate system and the radar coordinate system; The deviation value determination module is used to determine the deviation value between the target point and the initial perspective matrix if the target point is determined to meet the target tracking conditions. The second matrix determination module is used to recalibrate the initial perspective matrix based on the deviation values ​​of each target point to obtain the intermediate perspective matrix; The target matrix determination module is used to repeatedly execute the operation of determining the deviation value between the target point and the intermediate perspective matrix, and recalibrating the intermediate perspective matrix according to the deviation value of each target point until the intermediate perspective matrix meets the matrix accuracy condition, and then using the intermediate perspective matrix as the target perspective matrix. The deviation value determination module is used to: obtain a first number of transformed coordinates of the target point, determine the position points in each radar frame image that match the transformed coordinates of the target point, and obtain a first number of radar coordinates of the position points that match the target point; and determine the deviation value between the target point and the initial perspective matrix based on the first number of transformed coordinates of the target point and the first number of radar coordinates of the position points that match the target point.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for determining the perspective matrix as described in any one of claims 1-7.

10. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the method for determining the perspective matrix as described in any one of claims 1-7.

Images