An image live broadcast method and device based on trajectory information verification

Abstract

The present invention provides an image live broadcast method and device based on trajectory information verification. The method includes: solving the coordinates of a real target in the image coordinate system to establish a real target motion trajectory data set C1; establishing an identified target motion trajectory data set C2 based on the position coordinates of the target identified by using a target recognition model; obtaining a real target motion trajectory A from C1 and an identified target motion trajectory B from C2, where the category of the identified target belongs to the derivative category set of the real target and the video recording devices are the same; scoring the video image corresponding to each motion trajectory B based on the similarity between the motion trajectory A and each motion trajectory B, and switching the played video image to the video image with the highest score. By performing image optimization and ranking based on the similarity score of motion trajectories, the present invention can improve the accuracy and robustness of the output result compared with the optimization method based on single-frame images and data.

Description

Technical Field

[0001] The present invention belongs to the technical field of image live broadcast, and particularly relates to an image live broadcast method and device based on trajectory information verification. Background Art

[0002] In a television live broadcast task, multiple camera devices are generally used for shooting, so that the program has the characteristics of multiple angles and multiple scenes, and is presented more comprehensively and beautifully. The live broadcast system is responsible for selecting one of the multiple video and audio signals and broadcasting it to a specified channel, which is the core part of completing the live broadcast task. With the rapid development of radio and television technology and the gradual improvement of information communication means, as well as the popularization of monitoring and portable camera devices, the information sources of live broadcast tasks have been greatly enriched. At the same time, it has also brought great challenges to live broadcast personnel. Especially in the case of a large number of live images, many shooting targets, fast moving speeds, and short event times, the live broadcast work is extremely heavy, which is likely to lead to phenomena such as image switching disconnection and incomplete display of key frame images.

[0003] There are currently two live broadcast modes: one is the mainstream manual live broadcast work mode, in which operators conduct repeated and detailed deductions and drills according to the pre-specified live broadcast plan and pre-plan, continuously optimize the control steps, run-in the cooperation process, and solve various problems to present a relatively smooth live broadcast effect. This mode has the following problems: slow response speed, and it is easy to have a live broadcast disconnection in the case of rapid scene or target transformation and movement; high cost, and in the case of a large number of video images, the method of stacking manpower and material resources is used to increase efficiency and empower; the other is the live broadcast work mode based on computer vision technology. Using computer vision technology, a training set of target images is established in advance, and by selecting a suitable machine learning training algorithm, a trained classifier model is generated to identify the targets in the video images, and the multi-channel videos are preferably sorted and live broadcasted according to the recognition results. This mode has the following problems: when the target in the image is partially blocked by an obstacle, multiple targets have the same appearance and close positions, or there are new targets that have not been recognized and trained in the image, the target recognition accuracy is still uncontrollable. Also, due to the small error tolerance space in the live broadcast task, in important events such as sports competitions, performances, and exhibitions, even if there is only one switching error within a few hours, it is a live broadcast accident for the television station. Therefore, the live broadcast work mode based on computer vision technology has low reliability and is mostly used for auxiliary or display under specific conditions, and the actual use scenarios are limited. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides an image live broadcast method and device based on trajectory information verification.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions.

[0006] In a first aspect, the present invention provides an image live broadcast method based on trajectory information verification, including the following steps:

[0007] Based on the position coordinates of the real target and the position coordinates of the video recording device obtained in real time, solve the coordinates of the real target in the image coordinate system, and establish a real target motion trajectory data set C1;

[0008] Use the target detection model to perform target recognition and tracking on the video images captured by each video recording device in real time, and establish an identified target motion trajectory data set C2 based on the position coordinates of the identified targets;

[0009] Obtain a real target motion trajectory A from C1 and an identified target motion trajectory B from C2, where the category of the identified target belongs to the derivative category set of the real target and the video recording devices are the same;

[0010] Based on the similarity between the motion trajectory A and each motion trajectory B, score the video image corresponding to each motion trajectory B, and switch the played video image to the video image with the highest score.

[0011] Further, the method for solving the coordinates of the real target in the image coordinate system includes:

[0012] Establish a measurement coordinate system ENU with the video recording device as the origin, and calculate the coordinates of the real target in the measurement coordinate system (E 0 , N 0 , U 0 );

[0013] Based on (E 0 , N 0 , U 0 ), calculate the azimuth angle and pitch angle of the line connecting the real target and the video recording device, and calculate the differences Δα and Δβ between the azimuth angle and pitch angle and the azimuth angle and pitch angle read from the scale of the video recording device respectively. Set Δα to be positive when the real target is on the right side of the optical axis of the video recording device, and Δβ to be positive when the real target is above the optical axis of the video recording device, and the absolute values of Δα and Δβ are both less than π;

[0014] Establish an image coordinate system with the upper left vertex of the image as the origin, the horizontal right direction as the positive x-axis, and the vertical downward direction as the positive y-axis. Calculate the coordinates of the real target in the image coordinate system according to the following formula:

[0015] x = Δα / A R + 0.5 (1)

[0016] y = 0.5 - Δβ / E R (2)

[0017] In the formula, A R , E RThey are the horizontal viewing angle and the vertical viewing angle at the current focal length of the video recording device, where x and y are the abscissa and ordinate respectively, and 0 ≤ x ≤ 1, 0 ≤ y ≤ 1.

[0018] Furthermore, the data in datasets C1 and C2 both include the following fields: timestamp, target ID, video recording device ID, coordinates x and y in the image coordinate system, speed v, acceleration a, and target category class; the data in C2 also includes the width and height of the image and the confidence level p of the recognized target.

[0019] Even further, the calculation formula for scoring the video image corresponding to each motion trajectory B is:

[0020] S i = k 1 S i1 + k 2 S i2 + k 3 S i3 (3)

[0021] In the formula, S i is the total score of the video image corresponding to the i-th motion trajectory B, S i1 is the trajectory similarity score between motion trajectory A and the i-th motion trajectory B, S i2 is the position deviation score between the real target and the recognized target in the last frame data of motion trajectory A and the i-th motion trajectory B, S i3 is the similarity score between the real target category of motion trajectory A and the recognized target category of the i-th motion trajectory B, and k 1 , k 2 , k 3 are weighting coefficients; where i = 1, 2,......, n, and n is the number of motion trajectories B.

[0022] Even further, the calculation formula for score S i1 is:

[0023] S i1 = max(1 - D i / max(D 1 , D 2 ,......, D n ), 10 -3 ) (4)

[0024] D i = D ix + D iy + D iv + D ia (5)

[0025] In the formula, D iis the DTW distance between the motion trajectory A and the i-th motion trajectory B, D ix 、D iy 、D iv 、D ia are the DTW distances of the x, y, v, and a components respectively, and i = 1, 2,......, n.

[0026] Furthermore, before calculation, D i corrects the motion trajectory according to the following method:

[0027] Calculate the mean difference of the coordinates of the motion trajectory A and the motion trajectory B. The formula is as follows:

[0028]

[0029]

[0030] In the formula, dx and dy are the mean differences of the abscissa and ordinate respectively, x Ai 、y Ai and x Bi 、y Bi are the i-th coordinates of the motion trajectory A and the motion trajectory B respectively, and m A 、m B are the numbers of coordinates of the motion trajectory A and the motion trajectory B respectively;

[0031] Correct the coordinates of the motion trajectory B according to the following formula:

[0032] x′ Bi =x Bi +dx(8)

[0033] y′ Bi =y Bi +dy(9)

[0034] In the formula, x′ Bi 、y′ Bi are the corrected coordinates, and i = 1, 2,......, m B .

[0035] Furthermore, the calculation formula of the score S i2 is:

[0036]

[0037] In the formula, x A 、y A are the coordinates in the last frame of data of the motion trajectory A, and x iB 、y iB are the coordinates in the last frame of data of the i-th motion trajectory B.

[0038] Furthermore, the score S i3 is calculated by the following formula:

[0039] S i3 = (1 - (id i - 1) / M) × p i (11)

[0040] wherein, p i is the confidence of the recognition target of the i-th motion trajectory B, id i is the serial number of the recognition target category of the i-th motion trajectory B sorted according to the degree of association in its derived category set, id i = 1, 2,......, M, and M is the number of elements in the derived category set.

[0041] Furthermore, the method further includes filtering the data in C1, and the method is as follows:

[0042] If the coordinates x and y in the data satisfy: max(|2x - 1|, |2y - 1|) > 1, it means that the real target is outside the visible angle of the shooting device, and the data is deleted.

[0043] In a second aspect, the present invention provides an image live broadcast device based on trajectory information verification, including:

[0044] A C1 establishment module, configured to solve the coordinates of the real target in the image coordinate system based on the position coordinates of the real target and the position coordinates of the shooting device obtained in real time, and establish a real target motion trajectory data set C1;

[0045] A C2 establishment module, configured to perform target recognition and tracking on the video images captured by each shooting device in real time by using a target detection model, and establish a recognition target motion trajectory data set C2 based on the position coordinates of the recognized targets;

[0046] A trajectory matching module, configured to obtain a real target motion trajectory A from C1 and a recognition target motion trajectory B from C2, where the category of the recognition target belongs to the derived category set of the real target and the shooting devices are the same;

[0047] An image scoring module, configured to score the video image corresponding to each motion trajectory B based on the similarity between the motion trajectory A and each motion trajectory B, and switch the played video image to the video image with the highest score.

[0048] Compared with the prior art, the present invention has the following beneficial effects.

[0049] The present invention solves the coordinates of a real target in the image coordinate system, establishes a real target motion trajectory dataset C1, establishes an identified target motion trajectory dataset C2 based on the position coordinates of the target identified by using a target detection model, obtains a real target motion trajectory A from C1, obtains an identified target motion trajectory B from C2, where the category of the identified target belongs to the derivative category set of the real target and the recording devices are the same, scores the video image corresponding to each motion trajectory B based on the similarity between the motion trajectory A and each motion trajectory B, and switches the played video image to the video image with the highest score, realizing image live broadcast based on trajectory information verification. The present invention completes image optimization and sorting based on motion trajectory data composed of multiple frames of images and data. Compared with the optimization method based on single-frame images and data, it can improve the accuracy and robustness of the output results and reduce the false alarm rate caused by situations such as close target positions, partial visibility, occlusion, repetition, and poor visibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flowchart of a method for image live broadcast based on trajectory information verification according to an embodiment of the present invention.

[0051] Figure 2 It is a schematic diagram of an image live broadcast system.

[0052] Figure 3 It is a block diagram of a composition of an image live broadcast device based on trajectory information verification according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0054] Figure 1 It is a flowchart of a method for image live broadcast based on trajectory information verification according to an embodiment of the present invention, including the following steps:

[0055] Step 101: Based on the position coordinates of the real target obtained in real time and the position coordinates of the recording device, solve the coordinates of the real target in the image coordinate system, and establish a real target motion trajectory dataset C1;

[0056] Step 102: Use a target detection model to perform target recognition and tracking on the video images captured by each recording device in real time, and establish an identified target motion trajectory dataset C2 based on the position coordinates of the identified target;

[0057] Step 103: Obtain a real target motion trajectory A from C1 and an identified target motion trajectory B from C2, where the category of the identified target belongs to the derivative category set of the real target and the recording devices are the same;

[0058] Step 104: Score the video images corresponding to each motion trajectory B based on the similarity between the motion trajectory A and each motion trajectory B, and switch the played video image to the video image with the highest score.

[0059] An image live broadcast method based on trajectory information verification in this embodiment is applied to a live broadcast system. The composition of the live broadcast system is as Figure 2 shown. Figure 2 The network device in it is a device that provides wired and wireless communication for information transmission; the time synchronization device is a device responsible for providing unified standard time signals and standard frequency signals for correcting the time of other devices; the target positioning device is a GPS or Beidou positioning device installed on the real target, which can transmit the position information of the real target back in real time through the network device; the recording device is a device such as a camera or webcam with photography and videography functions, equipped with a GPS locator, and the device base is equipped with azimuth angle, pitch angle drive devices and scale disks for obtaining the azimuth angle and pitch angle information of the device, and can transmit the position, azimuth angle, pitch angle and focal length of the device back in real time through the network device; the image selection and switching device is a device with video and audio signal selection and switching functions such as a video and audio matrix or a distributed codec, which is used to complete the selection and switching of multi-channel video and audio information, send the images recorded by multiple recording devices to the image acquisition device, receive and execute the optimal selection instruction issued by the software operation server, and send the optimal output to the main program of the TV or the large screen; the image acquisition device is a device that converts the image of the analog electrical signal into a digital image signal, such as an acquisition card, an encoder, etc.; the software operation server is a computer hardware environment for running software such as an image recognition model, data reception, optimal sorting, etc., composed of multiple servers, which is used for multi-channel image recognition, sending and receiving image recognition results, receiving data transmitted back by the target and the recording device, data alignment, data caching, sorting calculation, controlling the image selection and switching device to complete image selection, etc.

[0060] In this embodiment, step 101 is mainly used to establish a real target motion trajectory dataset C1. As described above, positioning devices are installed on both the real target and the video recording device. An azimuth angle, pitch angle driving device and scale are also installed on the base of the video recording device, and the azimuth angle and pitch angle information of the video recording device can be obtained. By collecting the above data information in real time, the position data of the real target at different acquisition times can be obtained, and thus the real target motion trajectory data can be obtained. The motion trajectory data of each real target is saved to obtain the dataset C1. The end point of the motion trajectory is the current time t. If the time length of the trajectory is set to T, the starting point of the trajectory is t - T. Invalid data can be filtered by setting the maximum time interval Δt between adjacent data. When the data interval exceeds Δt, the data is not credible and is discarded. For the convenience of use and query, the data in C1 is saved by classification. For example, C1(i, j, k) represents the data corresponding to the k-th time stamp (acquisition time) of the j-th video recording device of the i-th real target.

[0061] In this embodiment, step 102 is mainly used to identify the target motion trajectory dataset C2. This embodiment involves two concepts: real target and identified target. The real target refers to physical entities such as airplanes and ships to be photographed; the identified target is the target identified from the input video image using a target recognition model. To distinguish it from the real target, it is called the identified target. In this embodiment, the trained target recognition model is used to identify and track the target in the input image, and data such as the position coordinates, speed, acceleration and confidence of the identified target are obtained. Based on the above data at different times, the motion trajectory data of the identified target can be obtained. The motion trajectory data of the identified target obtained from the video images captured by each video recording device is saved to obtain the dataset C2. To facilitate the comparison of the similarity between the real target motion trajectory and the identified target motion trajectory, their time lengths, start and end times, etc. are the same. Similarly, invalid data in C2 can be filtered by setting the maximum time interval. The data in C2 is also saved by classification. For example, C2(i, j, k) represents the data corresponding to the k-th time stamp of the j-th identified target of the i-th video recording device.

[0062] In this embodiment, step 103 is mainly used for motion trajectory matching. In this embodiment, the video images are optimized by comparing the similarity between the real target motion trajectory and the recognized target motion trajectory. Since the recording devices are not unique and one recording device may correspond to multiple recognized targets, it is not possible to compare any real target motion trajectory in C1 with any recognized target motion trajectory in C2. Instead, matching is required. For a single real target motion trajectory A, the recognized target motion trajectory B that matches it should have the same recording device, and the recognized target belongs to the derivative category set of the real target. For example, if the real target category is an airplane, the recognized target category can be an airplane, or it can be a wing, a wheel, etc. The derivative category is the category presented by the real target under different recording conditions. For example, the real target is an airplane. When the airplane is partially blocked, only the wing or a set of wheels may be shown; the image recorded from the rear during flight may be the tail flame or the wake; what is seen in an infrared camera is a light mass; when the distance is far, the image is just a point; what is seen in the image recorded at night is the indicator light on the fuselage; these are not airplanes during image recognition, but they are the presentations of the airplane in images under different recording conditions. Therefore, the derivative category of the airplane should be In order to compare the similarity between the real target category and the recognized target category, it is also necessary to arrange the derivative categories in descending order of relevance. For example, the derivative category sorting of the airplane is airplane, wing, wheel, wake, tail flame, light, and point.

[0063] In this embodiment, step 104 is mainly used for switching the played video image based on the score of the video image. In this embodiment, the video images corresponding to each motion trajectory B are scored based on the similarity between the motion trajectory A and each motion trajectory B, and a ranking from high to low according to the scores is obtained. Then, the played video image is switched to the video image with the highest score. The similarity of the motion trajectories takes into account the similarity of all data frames on the motion trajectories. Compared with the optimization method based on single-frame images and data, it can improve the accuracy and robustness of the output results and reduce the false alarm rate caused by situations such as close target positions, partial visibility, occlusion, repetition, and poor visibility. When switching images based on the score ranking, in order to avoid the phenomenon of repeated image switching causing discomfort to the viewer, the preferred ranking range threshold r num , the maximum switching time threshold r max and the minimum switching time threshold r min can be set, and timers t1 and t2 are set. When the serial number i of the currently played image in the sorting result is i = 1, the values of the timers t1 and t2 are cleared; if i ≠ 1 and i <= r num , the timer t2 is cleared, and the cumulative timing t1 is performed. When the timing time t1 > r max , an image switching instruction for the sorting of 1 is issued, and at the same time the timer t1 is cleared; if i > r num , the cumulative timing t2 is performed. When the timing time t2 > rmin Send an image instruction to switch the sorting to 1 when

[0064] As an optional embodiment, the method for solving the coordinates of the real target in the image coordinate system includes:

[0065] Establish a measurement coordinate system ENU with the video recording device as the origin, and calculate the coordinates of the real target in the measurement coordinate system (E 0 , N 0 , U 0 );

[0066] Based on (E 0 , N 0 , U 0 ), calculate the azimuth angle and elevation angle of the line connecting the real target and the video recording device, and calculate the differences Δα and Δβ between the azimuth angle and elevation angle and the azimuth angle and elevation angle read from the scale disk of the video recording device respectively. Set Δα to be positive when the real target is on the right side of the optical axis of the video recording device, and Δβ to be positive when it is above the optical axis of the video recording device, and the absolute values of Δα and Δβ are both less than π;

[0067] Establish an image coordinate system with the upper left vertex of the image as the origin, the horizontal right direction as the positive direction of the x-axis, and the vertical downward direction as the positive direction of the y-axis, and calculate the coordinates of the real target in the image coordinate system according to the following formula:

[0068] x = Δα / A R +0.5 (1)

[0069] y = 0.5 - Δβ / E R (2)

[0070] In the formula, A R , E R are the horizontal viewing angle and elevation viewing angle at the current focal length of the video recording device respectively, x and y are the abscissa and ordinate respectively, and 0 ≤ x ≤ 1, 0 ≤ y ≤ 1.

[0071] This embodiment provides a technical solution for solving the coordinates of the real target in the image coordinate system. First, establish a measurement coordinate system with the video recording device as the origin, that is, the east-north-up coordinate system ENU, and calculate the coordinates of the real target in the measurement coordinate system (E 0 , N 0 , U 0 ), and the formula is as follows:

[0072] E 0 = -(X - X 0 ) × sinL + (Y - Y 0 ) × cosL

[0073] N 0 = -(X - X 0)×sinB×cosL-(Y - Y 0 )×sinB×sinL+(Z - Z 0 )×cosB

[0074] U 0 =(X - X 0 )×cosB×cosL+(Y - Y 0 )×cosB×sinL+(Z - Z 0 )×sinB

[0075] In the formula, (X, Y, Z) and (X 0 , Y 0 , Z 0 ) are the geocentric space rectangular coordinate system coordinates of the target and the center point of the video recording device respectively, and (L, B, H) are the geodetic rectangular coordinate system coordinates of the center point of the video recording device.

[0076] Then, based on (E 0 , N 0 , U 0 ), calculate the azimuth angle α and pitch angle β of the line connecting the real target and the video recording device. The formulas are as follows:

[0077]

[0078]

[0079] Then, calculate the differences between the azimuth angle α, pitch angle β and the azimuth angle α 0 and pitch angle β 0 of the optical axis of the video recording device: Δα = α - α 0 , Δβ = β - β 0 , and make the absolute values of Δα and Δβ both less than π.

[0080] Next, establish an image coordinate system with the upper left vertex of the image as the origin, the horizontal right direction as the positive x-axis, and the vertical downward direction as the positive y-axis. Calculate the coordinates x and y of the real target in the image coordinate system according to formulas (1) and (2), satisfying 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1.

[0081] It should be noted that the image coordinate system in this embodiment is just an approximate name because the real target does not appear in the image, and its coordinates x and y are also normalized, that is, the maximum values of x and y are both 1. The coordinates of the recognized target are the coordinates in the image coordinate system, generally in units of the number of pixels. The maximum values of the horizontal and vertical coordinates are the number of pixels included in the width and height of the image. In order not to affect the comparison result of the real target trajectory and the recognized target trajectory, the coordinates of the recognized target should also be normalized, that is, the horizontal and vertical coordinates are divided by their maximum values respectively to transform the value range to between 0 and 1.

[0082] As an optional embodiment, the data in datasets C1 and C2 both include the following fields: timestamp, target ID, video recording device ID, coordinates x, y in the image coordinate system, speed v, acceleration a, target category class; the data in C2 further includes the width and height of the image and the confidence level p of the recognized target.

[0083] This embodiment gives the field content included in the data of datasets C1 and C2. The fields that the data in C1 and C2 both contain are timestamp, target ID, video recording device ID, coordinates x, y, speed v, acceleration a, target category class. The timestamp is the data acquisition moment. The speed and acceleration of the real target can be calculated based on the coordinates of several adjacent frames of data; the speed and acceleration of the recognized target are directly obtained by the target recognition model using the tracking algorithm. The target category of the recognized target can be a derivative category of the real target category. Different from C1, the data in C2 further includes the width and height of the image and the confidence level p of the recognized target. The confidence level p is also directly output by the target recognition model, and the confidence level p is used when calculating the similarity between the real target category and the recognized target category.

[0084] As an optional embodiment, the calculation formula for scoring the video image corresponding to each motion trajectory B is:

[0085] S i =k 1 S i1 +k 2 S i2 +k 3 S i3 (3)

[0086] In the formula, S i is the total score of the video image corresponding to the i-th motion trajectory B, S i1 is the trajectory similarity score between motion trajectory A and the i-th motion trajectory B, S i2 is the position deviation score between the real target and the recognized target in the last frame of data of motion trajectory A and the i-th motion trajectory B, S i3 is the similarity score between the real target category of motion trajectory A and the recognized target category of the i-th motion trajectory B, k 1 、k 2 、k 3 are weighting coefficients; where i = 1, 2,......, n, and n is the number of motion trajectories B.

[0087] This embodiment provides a calculation formula for video image scoring. The scoring formula is shown in Figure (3). The total score is equal to the weighted sum of three scoring items. The first scoring item is the similarity score of the motion trajectory. The similarity is calculated based on the position coordinates, velocity, and acceleration components of each data point. The greater the similarity, the higher the score. The similarity score of the motion trajectory is the most important one among the three scoring items, so its weighting coefficient k 1 is greater than k 2 and k 3 . The second scoring item is the position deviation score between the real target and the recognized target in the last frame data of the two trajectories. The greater the deviation, the lower the score. The third scoring item is the similarity score between the real target category and the recognized target category. The greater the similarity, the higher the score.

[0088] As an alternative embodiment, the calculation formula for the score S i1 is:

[0089] S i1 = max(1 - D i / max(D 1 , D 2 ,......, D n ), 10 -3 ) (4)

[0090] D i = D ix + D iy + D iv + D ia (5)

[0091] In the formula, D i is the DTW distance between the motion trajectory A and the i-th motion trajectory B. D ix , D iy , D iv , D ia are the DTW distances of the x, y, v, and a components respectively, and i = 1, 2,......, n.

[0092] This embodiment provides a calculation formula for the similarity score S i1 of the motion trajectory. This embodiment calculates the similarity of the motion trajectory based on the DTW distance (Dynamic Time Warping) algorithm. The core of the DTW algorithm is to calculate the optimal matching path between two sequences. The dynamic programming technique is used to find a path with the minimum cumulative distance. This path represents the best match between the two time series. Calculate the distance between each corresponding point on the optimal matching path and sum up these distances to obtain a total distance. This total distance is the similarity measure between the two time series, that is, the DTW distance. Compare the coordinate x, y, velocity v, and acceleration a components in the two trajectory data according to the time series to obtain the DTW distance Di = D ix + D iy + D iv + D ia , where D ix , D iy , D iv , D ia are the DTW distances of the x, y, v, and a components respectively. The scoring formula is as shown in formula (4). Actually, it calculates the score based on the relative magnitudes of the DTW distance D i between the motion trajectory A and each motion trajectory B. The larger D i is, the lower the score. Formula (4) can limit the minimum value of the score S i1 to 10 -3 , which can prevent the value of S i1 from being 0 (which is significantly different from other non-zero values).

[0093] As an alternative embodiment, before the calculation, D i corrects the motion trajectory according to the following method:

[0094] Calculate the mean difference of the coordinates of the motion trajectory A and the motion trajectory B. The formula is as follows:

[0095]

[0096]

[0097] In the formula, dx and dy are the mean differences of the abscissa and ordinate respectively, x Ai , y Ai and x Bi , y Bi are the i-th coordinates of the motion trajectory A and the motion trajectory B respectively, and m A , m B are the numbers of coordinates of the motion trajectory A and the motion trajectory B respectively;

[0098] Correct the coordinates of the motion trajectory B according to the following formula:

[0099] x′ Bi = x Bi + dx(8)

[0100] y′ Bi = y Bi + dy(9)

[0101] In the formula, x′ Bi , y′ Bi are the corrected coordinates, and i = 1, 2,......, m B .

[0102] This embodiment gives the method of calculating the DTW distance Di A technical solution for trajectory correction before. Due to factors such as the installation position of the positioning device and positioning errors, the target trajectory obtained based on the position coordinates may not necessarily be the trajectory of the target center point, and it is necessary to eliminate the influence brought by the trajectory distance difference through trajectory correction. In this embodiment, the mean difference between the coordinates of two trajectories (Equations (6) and (7)) is calculated, and the coordinates of one of the trajectories are corrected using the mean difference (Equations (8) and (9)). The specific method is as above and will not be elaborated in detail here.

[0103] As an optional embodiment, the score S i2 The calculation formula is:

[0104]

[0105] In the formula, x A , y A are the coordinates in the last frame of data of the motion trajectory A, and x iB , y iB are the coordinates in the last frame of data of the i-th motion trajectory B.

[0106] This embodiment gives the calculation formula of the score S i2 . S i2 is the position deviation score between the real target and the recognized target in the last frame of data of the two trajectories. Therefore, the position deviation can be represented by the distance between the position points represented by the two coordinates in the last frame of data. The larger the distance, the larger the position deviation and the lower the score. The scoring formula is as in Equation (10). As described above, the coordinates x iB , y iB of the recognized target should be the normalized coordinates, that is, the coordinate value ranges are all between 0 and 1. Since the maximum distance is the square root of 2, in order to avoid negative scores, the sum of squares term inside the square root in Equation (10) is divided by 2.

[0107] As an optional embodiment, the calculation formula of the score S i3 is:

[0108] S i3 =(1 - (id i - 1) / M)×p i (11)

[0109] In the formula, p i is the confidence of the recognized target of the i-th motion trajectory B, and id i is the serial number of the recognized target category of the i-th motion trajectory B sorted according to the degree of association in its derived category set, and id i = 1, 2,......, M, where M is the number of elements in the derived category set.

[0110] This embodiment gives the calculation formula of the score S i3 The score S i3 is the similarity score between the true target category and the recognized target category. Since it is limited that the recognized target category must belong to the derivative category set of the true target category when performing category matching, this embodiment scores based on the ranking of the association degree of the recognized target category in the derivative category set. The higher the ranking, the higher the association degree and the higher the score. In addition, the score value is not only related to the ranking of the association degree, but also related to the confidence level p i The higher the confidence level, the higher the score. The scoring formula is shown in formula (11).

[0111] As an optional embodiment, the method further includes filtering the data in C1, and the method is as follows:

[0112] If the coordinates x and y in the data satisfy: max(|2x - 1|, |2y - 1|) > 1, then the true target is outside the visible angle of the shooting device, and the data is deleted.

[0113] This embodiment gives a technical solution for filtering the data in C1. In order to eliminate the influence of invalid data, this embodiment screens the true target data in C1 by judging whether the center point of the true target is within the visible angle of the shooting device. According to the previous embodiment, when the center point of the true target is within the visible angle of the shooting device, its position coordinates satisfy 0 ≤ x ≤ 1, 0 ≤ y ≤ 1. Therefore, as long as one of x and y is not within 0 to 1, the center point of the true target is not within the visible angle of the shooting device. That any one of x and y is not within 0 to 1 is equivalent to the inequality max(|2x - 1|, |2y - 1|) > 1. Therefore, this embodiment screens the data according to whether this inequality holds.

[0114] Figure 3 It is a schematic diagram of the composition of an image live broadcast device based on trajectory information verification according to an embodiment of the present invention. The device includes:

[0115] A C1 establishment module 11, configured to solve the coordinates of the true target in the image coordinate system based on the position coordinates of the real-time acquired true target and the position coordinates of the shooting device, and establish a true target motion trajectory data set C1;

[0116] A C2 establishment module 12, configured to perform target recognition and tracking on the video images captured by each shooting device in real time by using a target detection model, and establish a recognized target motion trajectory data set C2 based on the position coordinates of the recognized targets;

[0117] A trajectory matching module 13 is configured to obtain a real target motion trajectory A from C1 and an identified target motion trajectory B from C2, where the category of the identified target belongs to the set of derivative categories of the real target and the recording devices are the same;

[0118] An image scoring module 14 is configured to score each video image corresponding to the motion trajectory B based on the similarity between the motion trajectory A and each motion trajectory B, and switch the played video image to the video image with the highest score.

[0119] The device of this embodiment can be used to execute Figure 1 the technical solutions of the method embodiment shown. The implementation principle and technical effects are similar and will not be elaborated here. The same applies to the subsequent embodiments and will not be further described.

[0120] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. An image directing method based on trajectory information verification, characterized in that: The following steps are involved: Based on the real target position coordinates and the position coordinates of the recording device acquired in real time, the coordinates of the real target in the image coordinate system are solved to establish the real target motion trajectory data set C1; The target detection model is used to identify and track the video images captured by each camera in real time, and a target motion trajectory dataset C2 is established based on the position coordinates of the identified targets. Obtain a real target motion track A from C1, and obtain a recognized target motion track B from C2, wherein the category of the recognized target belongs to the derived category set of the real target and the recording equipment is the same; Based on the similarity between the motion trajectory A and each motion trajectory B, the video image corresponding to each motion trajectory B is scored, and the playing video image is switched to the video image with the highest score; The data in datasets C1 and C2 include the following fields: timestamp, target ID, camera ID, coordinates x and y in the image coordinate system, velocity v, acceleration a, and target category class; the data in C2 also includes the width and height of the image and the confidence p of the identified target; The calculation formula for scoring the video image corresponding to each motion trajectory B is: S i =k1S i1 +k2S i2 +k3S i3 (3) In the formula, S i is the total score of the video image corresponding to the i-th motion trajectory B, S i1 Score the similarity between motion trajectory A and the i-th motion trajectory B, S i2 Score the position deviation between the real target and the recognized target in the last frame of the motion trajectory A and the i-th motion trajectory B, S i3 Score the similarity between the real target category of motion trajectory A and the identified target category of the i-th motion trajectory B, k1, k2, k3 weighting coefficients; where i = 1, 2, ..., n, n is the number of motion trajectories B; Score: S i1 The calculation formula is: S i1 =max(1-D i / max(D1,D2,......,D n ),10 -3 ) (4) D i =D ix +D iy +D iv +D ia (5) Where D i is the DTW distance between trajectory A and the i-th trajectory B, D ix , D iy , D iv , D ia are the DTW distances of the x, y, v, and a components respectively, i=1,2,...,n.

2. The image directing method based on trajectory information verification according to claim 1 is characterized in that: Methods for solving the coordinates of the real target in the image coordinate system include: Establish a measurement coordinate system ENU with the recording device as the origin, and calculate the coordinates of the real target in the measurement coordinate system (E0, N0, U0); Based on (E0, N0, U0), the azimuth and elevation angles of the line connecting the real target and the video recording device are calculated, and the differences Δα and Δβ between the azimuth and elevation angles and the azimuth and elevation angles read by the scale of the video recording device are calculated respectively. It is set that when the real target is located to the right of the optical axis of the video recording device, Δα is positive, and when the real target is located above the optical axis of the video recording device, Δβ is positive, and the absolute values ​​of Δα and Δβ are both less than π; Establish an image coordinate system with the upper left vertex of the image as the origin, the horizontal right as the positive x-axis, and the vertical downward as the positive y-axis. Calculate the coordinates of the real target in the image coordinate system as follows: x=Dα / A R +0.5 (1) y=0.5-Δβ / E R (2) In the formula, A R 、E R They are the horizontal viewing angle and pitch viewing angle of the camera at the current focal length. x and y are the horizontal and vertical coordinates respectively, 0≤x≤1, 0≤y≤1.

3. The image directing method based on trajectory information verification according to claim 1 is characterized in that: Before calculating D i Correct the motion trajectory as follows: Calculate the mean difference between the coordinates of motion trajectory A and motion trajectory B. The formula is as follows: In the formula, dx and dy are the mean difference of the horizontal axis and the mean difference of the vertical axis respectively, and x Ai ,y Ai and x Bi ,y Bi are the i-th coordinates of motion trajectory A and motion trajectory B, respectively, m A 、m B are the number of coordinates of motion trajectory A and motion trajectory B respectively; Correct the coordinates of the motion trajectory B according to the following formula: x′ Bi =x Bi +dx(8)y′ Bi =y Bi +dy(9)In the formula, x′ Bi , y′ Bi is the corrected coordinate, i=1,2,......,m B .

4. The image directing method based on trajectory information verification according to claim 1, characterized in that: Score: S i2 The calculation formula is: In the formula, x A ,y A is the coordinate of the last frame of motion trajectory A, x iB ,y iB is the coordinate of the last frame of data of the i-th motion trajectory B.

5. The image directing method based on trajectory information verification according to claim 1 is characterized in that: Score: S i3 The calculation formula is: S i3 =(1-(id i -1) / M)×p i (11) In the formula, p i is the confidence of the target identified by the i-th motion trajectory B, id i is the sequence number of the identified target category of the i-th motion trajectory B in the derived category set to which it belongs, sorted by the degree of association, id i =1,2,......,M, where M is the number of elements in the derived category set.

6. The image directing method based on trajectory information verification according to claim 2 is characterized in that: The method further includes filtering the data in C1 as follows: If the coordinates x and y in the data satisfy: max(|2x-1|,|2y-1|)>1, the real target is outside the visual angle of the video recording device, and the data is deleted.

7. An image directing device based on trajectory information verification, characterized in that: include: The C1 establishment module is used to solve the coordinates of the real target in the image coordinate system based on the position coordinates of the real target and the position coordinates of the recording device acquired in real time, and establish the real target motion trajectory data set C1; C2 establishment module, used to use the target detection model to identify and track the video images captured by each camera in real time, and to establish the target motion trajectory data set C2 based on the position coordinates of the identified target; A trajectory matching module is used to obtain a real target motion trajectory A from C1 and obtain a recognized target motion trajectory B from C2, wherein the category of the recognized target belongs to the derived category set of the real target and the recording device is the same; An image scoring module, used to score the video image corresponding to each motion trajectory B based on the similarity between the motion trajectory A and each motion trajectory B, and switch the playing video image to the video image with the highest score; The data in datasets C1 and C2 include the following fields: timestamp, target ID, camera ID, coordinates x and y in the image coordinate system, velocity v, acceleration a, and target category class; the data in C2 also includes the width and height of the image and the confidence p of the identified target; The calculation formula for scoring the video image corresponding to each motion trajectory B is: S i =k1S i1 +k2S i2 +k3S i3 (3) In the formula, S i is the total score of the video image corresponding to the i-th motion trajectory B, S i1 Score the similarity between motion trajectory A and the i-th motion trajectory B, S i2 Score the position deviation between the real target and the recognized target in the last frame of the motion trajectory A and the i-th motion trajectory B, S i3 Score the similarity between the real target category of motion trajectory A and the identified target category of the i-th motion trajectory B, k1, k2, k3 weighting coefficients; where i = 1, 2, ..., n, n is the number of motion trajectories B; Score: S i1 The calculation formula is: S i1 =max(1-D i / max(D1,D2,......,D n ),10 -3 ) (4) D i =D ix +D iy +D iv +D ia (5) Where D i is the DTW distance between trajectory A and the i-th trajectory B, D ix , D iy , D iv , D ia are the DTW distances of the x, y, v, and a components respectively, i=1,2,...,n.

Images