Panoramic video target tracking method, electronic device, and storage medium

Abstract

The application provides a panoramic video target tracking method, an electronic device and a storage medium, and relates to the technical field of visual target tracking. The panoramic video target tracking method comprises the following steps: first, obtaining information of a target to be tracked in a current sampling frame of a target video, wherein the target video is obtained by planarizing a panoramic video; then, calculating an optical flow of the target to be tracked between the current sampling frame and a next sampling frame according to the information of the target to be tracked; further, determining a reference coordinate matrix of the target to be tracked in the next sampling frame according to the optical flow; and finally, correcting the reference coordinate matrix to obtain an effective coordinate matrix of the target to be tracked in the next sampling frame. The planar video generated by the panoramic video is taken as a processing object, so that the frequent 2D rendering process is avoided, the time consumption of the target tracking process is effectively reduced, and the efficiency of the target tracking is improved.

Description

[Technical Field]

[0001] This application relates to the field of visual target tracking technology, and in particular to a panoramic video target tracking method, electronic device, and storage medium. [Background Technology]

[0002] Visual target tracking is an important research direction in the field of computer vision. It refers to the detection and tracking of moving targets in image sequences to obtain motion parameters such as the position and trajectory of the moving targets.

[0003] Currently, common methods for target tracking in panoramic videos are typically based on spherical coordinate systems. Since a single frame cannot cover the entire content of a panoramic video in spherical coordinates, this method requires continuously rendering local regions of consecutive sampled frames to obtain local 2D images, and then performing target tracking within these local 2D images. However, the multiple rendering processes are very time-consuming, resulting in low execution efficiency for current target tracking methods. [Summary of the Invention]

[0004] This application provides a panoramic video target tracking method, electronic device, and storage medium to reduce the time consumption of panoramic video target tracking and improve the efficiency of panoramic video target tracking.

[0005] In a first aspect, embodiments of this application provide a panoramic video target tracking method, the method comprising: acquiring information of a target to be tracked in a current sampling frame of a target video; wherein the target video is obtained by planarization processing of a panoramic video; calculating the optical flow of the target to be tracked between the current sampling frame and the next sampling frame based on the information of the target to be tracked; determining a reference coordinate matrix of the target to be tracked in the next sampling frame based on the optical flow; and correcting the reference coordinate matrix to obtain an effective coordinate matrix of the target to be tracked in the next sampling frame.

[0006] In one possible implementation, calculating the optical flow of the target to be tracked between the current sampling frame and the next sampling frame based on the information of the target to be tracked includes: determining each pixel point contained in the target to be tracked based on the information of the target to be tracked; calculating the optical flow of each pixel point between the current sampling frame and the next sampling frame respectively; and determining the average value of the optical flow of each pixel point as the optical flow of the target to be tracked.

[0007] In one possible implementation, the optical flow includes a horizontal optical flow and a vertical optical flow; determining the reference coordinate matrix of the target to be tracked in the next sampling frame based on the optical flow includes: determining the horizontal optical flow as the horizontal displacement value of the target to be tracked; determining the vertical optical flow as the vertical displacement value of the target to be tracked; and determining the reference coordinate matrix of the target to be tracked in the next sampling frame based on the horizontal displacement value and the vertical displacement value.

[0008] In one possible implementation, the reference coordinate matrix is ​​modified by: offsetting the reference coordinate matrix according to a preset offset vector to obtain several offset coordinate matrices of the target to be tracked; and modifying the reference coordinate matrix according to the image features of the target to be tracked at each offset coordinate matrix.

[0009] In one possible implementation, the reference coordinate matrix is ​​corrected based on the image features of the target to be tracked at each offset coordinate matrix, including: calculating the feature vectors of the target to be tracked at each offset coordinate matrix; and correcting the reference coordinate matrix based on each feature vector.

[0010] In one possible implementation, the feature vectors of the target to be tracked at each offset coordinate matrix are calculated, including: calculating the color histogram and gradient histogram of the target to be tracked at each offset coordinate matrix; and determining the feature vectors of the target to be tracked at each offset coordinate matrix based on the color histogram and the gradient histogram.

[0011] In one possible implementation, the reference coordinate matrix is ​​modified according to each of the feature vectors, including: calculating the similarity between each of the feature vectors and the target feature vector, wherein the target feature vector is calculated based on the information of the target to be tracked in the current sampling frame; and modifying the reference coordinate matrix according to the similarity.

[0012] In one possible implementation, the reference coordinate matrix is ​​corrected based on the similarity, including: determining whether the maximum value among the various similarities is greater than a first threshold; if it is greater than the first threshold, then determining the offset coordinate matrix corresponding to the feature vector with the largest similarity as the effective coordinate matrix of the target to be tracked; if it is not greater than the first threshold, then determining the offset coordinate matrix corresponding to the feature vector with the largest similarity as the corrected reference coordinate matrix; and repeatedly correcting the corrected reference coordinate matrix until the maximum value among the obtained similarities is greater than the first threshold.

[0013] In one possible implementation, the method further includes: converting the effective coordinate matrix into an effective spherical coordinate matrix; and determining the spherical distance between the target to be tracked in the next sampling frame and the current sampling frame based on the effective spherical coordinate matrix.

[0014] In one possible implementation, the method further includes: after determining that the spherical distance is greater than a second threshold, performing 2D rendering on the target to be tracked.

[0015] Secondly, embodiments of this application provide a panoramic video target tracking device, the device comprising: an acquisition module, configured to acquire information about a target to be tracked in a current sampling frame of a target video; wherein the target video is obtained by planarizing a panoramic video; a calculation module, configured to calculate the optical flow of the target to be tracked between the current sampling frame and the next sampling frame based on the information of the target to be tracked; a determination module, configured to determine a reference coordinate matrix of the target to be tracked in the next sampling frame based on the optical flow; and an execution module, configured to correct the reference coordinate matrix to obtain an effective coordinate matrix of the target to be tracked in the next sampling frame.

[0016] In one possible implementation, the calculation module is specifically used to: determine each pixel point contained in the target to be tracked based on the information of the target to be tracked; calculate the optical flow of each pixel point between the current sampling frame and the next sampling frame; and determine the average value of the optical flow of each pixel point as the optical flow of the target to be tracked.

[0017] In one possible implementation, the optical flow includes a horizontal optical flow and a vertical optical flow; the determining module is specifically used to determine the horizontal optical flow as the horizontal displacement value of the target to be tracked; determine the vertical optical flow as the vertical displacement value of the target to be tracked; and determine the reference coordinate matrix of the target to be tracked in the next sampling frame based on the horizontal displacement value and the vertical displacement value.

[0018] In one possible implementation, the execution module is specifically used to: offset the reference coordinate matrix according to a preset offset vector to obtain several offset coordinate matrices of the target to be tracked; and correct the reference coordinate matrix according to the image features of the target to be tracked at each offset coordinate matrix.

[0019] In one possible implementation, the execution module is specifically used to calculate the feature vectors of the target to be tracked at each offset coordinate matrix; and to correct the reference coordinate matrix based on each feature vector.

[0020] In one possible implementation, the execution module is specifically used to calculate the color histogram and gradient histogram of the target to be tracked at each offset coordinate matrix; and to determine the feature vector of the target to be tracked at each offset coordinate matrix based on the color histogram and the gradient histogram.

[0021] In one possible implementation, the execution module is specifically used to calculate the similarity between each of the feature vectors and the target feature vector, wherein the target feature vector is calculated based on the information of the target to be tracked in the current sampling frame; and to correct the reference coordinate matrix based on the similarity.

[0022] In one possible implementation, the execution module is specifically used to determine whether the maximum value among the various similarities is greater than a first threshold; if it is greater than the first threshold, the offset coordinate matrix corresponding to the feature vector with the largest similarity is determined as the effective coordinate matrix of the target to be tracked; if it is not greater than the first threshold, the offset coordinate matrix corresponding to the feature vector with the largest similarity is determined as the corrected reference coordinate matrix; the corrected reference coordinate matrix is ​​repeatedly corrected until the maximum value among the obtained similarities is greater than the first threshold.

[0023] In one possible implementation, the execution module is further configured to convert the effective coordinate matrix into an effective spherical coordinate matrix; and determine the spherical distance between the target to be tracked in the next sampling frame and the current sampling frame based on the effective spherical coordinate matrix.

[0024] In one possible implementation, the execution module is further configured to perform 2D rendering on the target to be tracked after determining that the spherical distance is greater than the second threshold.

[0025] Thirdly, embodiments of this application provide an electronic device, including: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method as described in the first aspect by invoking the program instructions.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that cause the computer to perform the method described in the first aspect.

[0027] The above technical solution uses the planar video generated from the panoramic video as the processing object, avoiding frequent 2D rendering processes, effectively reducing the time consumption of the target tracking process, and improving the efficiency of target tracking. [Attached Image Description]

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of a panoramic video target tracking method provided in this application embodiment;

[0030] Figure 2 A flowchart illustrating another panoramic video target tracking method provided in this application embodiment;

[0031] Figure 3 A graph showing the correspondence between quantized angle values ​​and weights provided in an embodiment of this application;

[0032] Figure 4 A schematic diagram of the structure of a panoramic video target tracking device provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application.

Detailed Implementation Methods

[0034] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0035] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0036] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0037] Figure 1 A flowchart illustrating a panoramic video target tracking method provided in an embodiment of this application. Figure 1 As shown, the above panoramic video target tracking method may include:

[0038] Step 101: Obtain information about the target to be tracked in the current sampling frame of the target video. The target video is obtained by planarizing the panoramic video.

[0039] In this embodiment, before performing target tracking on the panoramic video, the acquired panoramic video can be converted into a planar video. Therefore, the panoramic video target tracking method provided in this application can use the planar video as the processing object to achieve target tracking. In this application, the planar video obtained after planarizing the panoramic video is referred to as the target video.

[0040] First, the target video can be sampled according to a set sampling rate to obtain multiple sampled frames of the target video. It should be noted that the effectiveness of the method provided in this application embodiment is less affected by the sampling rate. Therefore, even in the case of sparse sampling, reliable tracking results can still be obtained.

[0041] Understandably, to achieve target tracking, it is only necessary to obtain the coordinate parameters of the target in each sampling frame. Since the target is usually composed of multiple pixels, the coordinate parameters of the target can be represented as a coordinate matrix. For example, the target o i = [x, y, w, h]. Where (x, y) represents the target to be tracked. i The starting position of the minimum bounding matrix; w represents the target o to be tracked. i The width; h represents the target to be tracked. i The height.

[0042] In this embodiment of the application, the current sampling frame can be the first sampling frame obtained from the target video, or any intermediate sampling frame.

[0043] If the current sampling frame is the first sampling frame, the information of the target to be tracked can be obtained by using a deep network model to obtain the information of the target to be tracked contained therein.

[0044] When the current sampling frame is any one of the middle sampling frames, the target tracking result of the sampling frame can be obtained by using the panoramic video target tracking method provided in this application based on the information of the target to be tracked in the first sampling frame, that is, the information of the target to be tracked contained therein.

[0045] The information about the targets to be tracked may include: the number of targets to be tracked, the coordinate parameters of each target, etc.

[0046] When the current sampling frame contains multiple targets to be tracked, the target tracking method for each target is the same. For ease of description and understanding, this application embodiment only uses one target to be tracked as an example to illustrate the panoramic video target tracking method provided in this application. When there are multiple targets to be tracked, the same operation can be performed for each target.

[0047] Step 102: Calculate the optical flow of the target to be tracked between the current sampling frame and the next sampling frame based on the information of the target to be tracked.

[0048] First, the current sampled frame can be converted from an RGB image to a grayscale image.

[0049] If the information of the target to be tracked includes the coordinate parameters of the target, the individual pixels contained in the target can be determined based on the coordinate parameters of the target.

[0050] Then, the optical flow of each pixel between the current sampling frame and the next sampling frame can be calculated to obtain the displacement of each pixel. Taking any pixel A as an example, assuming its coordinates are (x, y), then F x (i,j) can be used to represent the horizontal displacement of pixel A; F y (i,j) can be used to represent the displacement of pixel A in the vertical direction.

[0051] Finally, the average optical flow of each pixel can be calculated to obtain the optical flow of the target to be tracked. Taking the target to be tracked as an example... i For example, given a value of [x,y,w,h], the target to be tracked is o. i The optical flow in the horizontal direction can be expressed as: The optical flow in the vertical direction can be expressed as:

[0052] Step 103: Determine the reference coordinate matrix of the target to be tracked in the next sampling frame based on the optical flow.

[0053] Since optical flow can represent the displacement of the target being tracked, the horizontal displacement value of the target can be obtained based on the optical flow in the horizontal direction, and the vertical displacement value of the target can be obtained based on the optical flow in the vertical direction.

[0054] Furthermore, based on the horizontal and vertical displacement values, a coordinate transformation is performed on the target to be tracked in the current sampling frame to obtain the reference coordinate matrix p = [f] of the target to be tracked in the next sampling frame. x +x,f y +y,w,h).

[0055] Step 104: Correct the reference coordinate matrix to obtain the effective coordinate matrix of the target to be tracked in the next sampling frame.

[0056] Due to unavoidable errors, the reference coordinate matrix obtained in step 103 may not accurately represent the position of the target to be tracked in the next sampling frame. Therefore, it is necessary to correct the reference coordinate matrix to obtain the effective coordinate matrix of the target to be tracked in the next sampling frame.

[0057] In this embodiment, the effective coordinate matrix is ​​used to represent the accurate position of the target to be tracked in the next sampling frame. It should be noted that the accurate position described here refers to relative accuracy within a reasonable error range and should not be construed as absolute accuracy.

[0058] First, the reference coordinate matrix can be offset according to the preset offset vector to obtain several offset coordinate matrices of the target to be tracked.

[0059] There can be multiple preset offset vectors, each with a different direction. This allows us to obtain the offset coordinate matrices of the reference coordinate matrix in various directions.

[0060] In one feasible implementation, nine preset offset vectors can be set. The nine preset offset vectors are as follows: as well as By offsetting the reference coordinate matrix using nine preset offset vectors, nine offset coordinate matrices in different directions can be obtained. The preset offset vectors... The corresponding offset coordinate matrix overlaps with the original reference coordinate matrix.

[0061] Then, the reference coordinate matrix can be corrected based on the image features of the target at each offset coordinate matrix.

[0062] It is understandable that the same target to be tracked has the same image features in different sampling frames. Therefore, in this embodiment, the reference coordinate matrix can be corrected according to the image features of the target to be tracked at each offset coordinate matrix.

[0063] Specifically, the image features of the target at each offset coordinate matrix can be compared with the target image features. The target image features refer to the image features of the target in the current sampling frame. Then, the offset coordinate matrix corresponding to the comparison result with the highest matching degree can be determined as the effective coordinate matrix of the target in the next sampling frame.

[0064] Based on obtaining the effective coordinate matrix of the target in the next sampling frame, the position of the target in subsequent sampling frames can be tracked according to the panoramic video target tracking method provided in this application embodiment. This process can be repeated in this manner, without further elaboration.

[0065] Furthermore, since the obtained effective coordinate matrix is ​​in a planar coordinate system, it can be transformed to a spherical coordinate system to obtain an effective spherical coordinate matrix.

[0066] Still awaiting target tracking iFor example, suppose its effective coordinate matrix is ​​(x′, y′, w, h), and the length and width of each sampling frame are W and H, respectively. Then, the corresponding effective spherical coordinate matrix b o for:

[0067] b o =[cos(φ)×cos(θ), cos(φ)×sin(θ), sin(φ)]

[0068] Among them, θ=(x′+w / 2)×(2π) / W, φ=0.5π-(y′+h / 2)×π / H.

[0069] Correspondingly, the coordinate parameters of the target to be tracked in the current sampling frame can also be converted into a spherical coordinate matrix b using the same calculation method. g Furthermore, based on the two spherical coordinate matrices, the spherical distance γ of the target to be tracked between the current sampling frame and the next sampling frame can be obtained:

[0070] γ=arccos(b o (0)×b g (0)+b o (1)×b g (1)+b o (2)×b g (2))×180 / π

[0071] The above technical solution uses the two-dimensional projection image of panoramic video as the processing object, avoiding frequent rendering and coordinate transformation processes, thus effectively improving the efficiency of panoramic video target tracking. Furthermore, since the above solution first determines the reference coordinate matrix of the target to be tracked through optical flow, even under sparse sampling, tracking will not fail due to excessive movement of the target. This reduces the impact of the sampling rate on the tracking effect and improves the reliability of target tracking.

[0072] Building upon the aforementioned embodiments, and further, in panoramic video, targets with large angular changes are typically those captured in a surrounding shot, resulting in a significant visual impact. Therefore, 2D rendering can be performed on the tracked targets with large changes in spherical distance to obtain compelling video clips.

[0073] In one implementation, the spherical distance between the target to be tracked and two adjacent frames (e.g., the current sampling frame and the next sampling frame) can be determined. If the spherical distance is greater than a set second threshold, then the target to be tracked can be rendered in 2D.

[0074] Alternatively, in another implementation, the cumulative spherical distance of the target to be tracked over multiple frames can be determined. When the cumulative spherical distance exceeds a set third threshold, the target to be tracked is rendered in 2D.

[0075] The above method allows for the selection of targets with large angle changes from multiple trackable targets for rendering, resulting in more ideal video effects.

[0076] In another embodiment of this application, the implementation method of the aforementioned step 104 is further described.

[0077] Figure 2 A flowchart illustrating a panoramic video target tracking method provided in an embodiment of this application. Figure 2 As shown, after obtaining the offset coordinate matrices of the target to be tracked based on the preset offset vector, the reference coordinate matrix can be corrected according to the following process.

[0078] Step 201: Calculate the feature vector of the target at each offset coordinate matrix based on the image features of the target at each offset coordinate matrix.

[0079] In this embodiment of the application, image features may include color features and shape features. Color features can be represented using a color histogram, and shape features can be represented using a gradient histogram.

[0080] Based on the above, the color histogram and gradient histogram of the target to be tracked at each offset coordinate matrix can be calculated respectively.

[0081] When calculating the color histogram, the calculation method for the region defined by each offset coordinate matrix is ​​the same. This application's embodiment uses the region defined by any one offset coordinate matrix as an example for illustration.

[0082] First, you can convert an RGB image to a grayscale image.

[0083] Then, the region defined by the offset coordinate matrix can be divided into u×v rectangular blocks. For any rectangular block, the pixel value of each pixel contained within it can be calculated.

[0084] In this embodiment, to appropriately improve the fault tolerance, the pixel value of each pixel can be quantized. Specifically, the pixel value range of the grayscale image is between 0 and 255. The pixel value of each pixel can be quantized to the range [0, N], then the quantization scale s c = [256 / N]. Assuming pixel value A is I(i, j), then the quantized pixel value is...

[0085] Based on the quantized pixel values ​​of each pixel, the color histogram of each rectangular block can be obtained. Where r0, r1, c0, and c1 represent the starting row, ending row, starting column, and ending column of each rectangular block, respectively. At that time, Hc (i, j, k) = 1; otherwise H c (i, j, k) = 0.

[0086] Finally, H of each rectangular block c (k) Concatenation yields a one-dimensional vector d of length u×v×N. c That is, the color histogram of the target to be tracked at that offset coordinate matrix.

[0087] When calculating the gradient histogram, the region defined by each offset coordinate matrix is ​​calculated in the same way. This application's embodiment uses the region defined by any one offset coordinate matrix as an example for illustration.

[0088] First, you can convert an RGB image to a grayscale image.

[0089] Then, the region defined by the offset coordinate matrix can be divided into u×v rectangular blocks. For any rectangular block, the Sobel operator S can be used. x Calculate the horizontal gradient G of each pixel. x Using the Sobel operator S y Calculate the vertical gradient G of each pixel. y .

[0090] in, Horizontal gradient G x and the vertical gradient G y For specific calculation methods, please refer to existing technologies.

[0091] Furthermore, the magnitude U of the gradient and the angle O can be calculated using the following formulas:

[0092]

[0093]

[0094] To appropriately improve the fault tolerance, the angle value of each pixel can be quantized. Specifically, the angle value of each pixel can be quantized to the range [0, M], and the quantization scale s is... g = [360 / M]. In this embodiment, for any pixel, angle quantization can be performed by rounding down and rounding up. That is, assuming the angle value of pixel A is O(i,j), the quantized angle values ​​are respectively as well as

[0095] like Figure 3 As shown, based on the quantized angle value With angle value O(i,j) / s gThe relationship between them can be used to assign different weights to the two quantized angle values.

[0096] Specifically, angle value The corresponding weight values ​​can be: Angle value The corresponding weight values ​​can be: That is, the angle value obtained by rounding up is the same as the angle value O(i,j) / s. g The smaller the difference between the two values, the smaller the weight of the angle value obtained by taking the value downwards. Conversely, the angle value O(i,j) / s g The smaller the difference between the angle value obtained by rounding down and the angle value obtained by rounding up, the smaller the weight of the angle value obtained by rounding up.

[0097] Based on the quantized angle values ​​of each pixel, the gradient histogram of each rectangular block can be obtained.

[0098] Where r0, r1, c0, and c1 represent the starting row, ending row, starting column, and ending column of each rectangular block, respectively. At that time, H g (i,j,k)=w0*U(i,j); when At that time, H g (i,j,k)=w1*U(i,j); when and At that time, H g (i,j,k)=0.

[0099] Finally, H of each rectangular block g (k) Concatenation yields a one-dimensional vector d of length u×v×M. g That is, the gradient histogram of the target to be tracked at that offset coordinate matrix.

[0100] Furthermore, based on the color histogram d c and gradient histogram d g Calculate the feature vector D of the target to be tracked at each offset coordinate matrix. i .

[0101] Specifically, at any offset coordinate matrix, the feature vector D of the target to be tracked... i It can be obtained from the following formula:

[0102]

[0103] in,

[0104] Step 202: Based on each feature vector, correct the reference coordinate matrix of the target to be tracked in the next sampling frame.

[0105] In this embodiment of the application, firstly, the similarity between each of the aforementioned feature vectors and the target feature vector can be calculated respectively.

[0106] Here, the target feature vector refers to the feature vector of the target to be tracked in the current sampling frame. Its specific calculation method is the same as that used in step 201 above to calculate the feature vector of the region determined by any offset coordinate matrix.

[0107] Then, the reference coordinate matrix can be corrected based on the calculated similarity scores.

[0108] Specifically, the maximum value among the various similarities can be selected, and it can be determined whether the maximum value is greater than the set first threshold.

[0109] If the value is greater than the first threshold, it indicates that the image features of the target to be tracked at that offset coordinate matrix are highly similar to the image features of the target to be tracked in the current sampling frame. In this case, it can be considered that the target to be tracked in the next sampling frame has been found. Therefore, the offset coordinate matrix corresponding to the feature vector with the highest similarity can be determined as the effective coordinate matrix of the target to be tracked in the next sampling frame.

[0110] If the similarity is not greater than the first threshold, it indicates that the image features of the target to be tracked at that offset coordinate matrix are not very similar to the image features of the target to be tracked in the current sampling frame. In this case, the offset coordinate matrix corresponding to the feature vector with the highest similarity can be determined as the corrected reference coordinate matrix. Then, based on the above correction method, the corrected reference coordinate matrix can be repeatedly corrected until the maximum value of the obtained similarity is greater than the aforementioned first threshold.

[0111] It should be noted that if the preset offset vector corresponding to the offset coordinate matrix with the highest similarity is... This means the reference coordinate matrix itself is the position closest to the target being tracked. In this case, during repeated corrections, the size of the preset offset vector can be reduced by a set proportion to obtain a new offset coordinate matrix. For example, the new preset offset vector could be:

[0112] The above technical solution allows for repeated correction of the reference coordinate matrix of the target in the next sampling frame based on its color and shape features. This improves the accuracy of target tracking.

[0113] Figure 4 This is a schematic diagram of the structure of a panoramic video target tracking device provided in an embodiment of this application. Figure 4As shown, the panoramic video target tracking device provided in this application embodiment may include: an acquisition module 41, a calculation module 42, a determination module 43, and an execution module 44.

[0114] The acquisition module 41 is used to acquire information about the target to be tracked in the current sampling frame of the target video; wherein the target video is obtained by planarizing a panoramic video.

[0115] The calculation module 42 is used to calculate the optical flow of the target to be tracked between the current sampling frame and the next sampling frame based on the information of the target to be tracked.

[0116] The determination module 43 is used to determine the reference coordinate matrix of the target to be tracked in the next sampling frame based on the optical flow.

[0117] The execution module 44 is used to correct the reference coordinate matrix to obtain the effective coordinate matrix of the target to be tracked in the next sampling frame.

[0118] In a specific implementation, the calculation module 42 is specifically used to: determine each pixel point contained in the target to be tracked based on the information of the target to be tracked; calculate the optical flow of each pixel point between the current sampling frame and the next sampling frame; and determine the average value of the optical flow of each pixel point as the optical flow of the target to be tracked.

[0119] In a specific implementation, optical flow includes horizontal optical flow and vertical optical flow; the determination module 43 is specifically used to determine the horizontal optical flow as the horizontal displacement value of the target to be tracked; determine the vertical optical flow as the vertical displacement value of the target to be tracked; and determine the reference coordinate matrix of the target to be tracked in the next sampling frame based on the horizontal displacement value and the vertical displacement value.

[0120] In a specific implementation process, the execution module 44 is specifically used to offset the reference coordinate matrix according to the preset offset vector to obtain several offset coordinate matrices of the target to be tracked; and to correct the reference coordinate matrix according to the image features of the target to be tracked at each offset coordinate matrix.

[0121] In a specific implementation, the execution module 44 is specifically used to calculate the feature vectors of the target to be tracked at each offset coordinate matrix; and to correct the reference coordinate matrix based on each feature vector.

[0122] In a specific implementation, the execution module 44 is specifically used to calculate the color histogram and gradient histogram of the target to be tracked at each offset coordinate matrix; and to determine the feature vector of the target to be tracked at each offset coordinate matrix based on the color histogram and the gradient histogram.

[0123] In a specific implementation process, the execution module 44 is specifically used to calculate the similarity between each feature vector and the target feature vector, the target feature vector being calculated based on the information of the target to be tracked in the current sampling frame; and to correct the reference coordinate matrix based on the similarity.

[0124] In a specific implementation process, the execution module 44 is specifically used to determine whether the maximum value among the various similarities is greater than the first threshold; if it is greater than the first threshold, the offset coordinate matrix corresponding to the feature vector with the largest similarity is determined as the effective coordinate matrix of the target to be tracked; if it is not greater than the first threshold, the offset coordinate matrix corresponding to the feature vector with the largest similarity is determined as the corrected reference coordinate matrix; the corrected reference coordinate matrix is ​​repeatedly corrected until the maximum value among the obtained similarities is greater than the first threshold.

[0125] In a specific implementation, the execution module 44 is also used to convert the effective coordinate matrix into an effective spherical coordinate matrix; and to determine the spherical distance between the target to be tracked in the next sampling frame and the current sampling frame based on the effective spherical coordinate matrix.

[0126] In a specific implementation, the execution module 44 is also used to perform 2D rendering of the target to be tracked after determining that the spherical distance is greater than the second threshold.

[0127] The above technical solution uses the planar video generated from the panoramic video as the processing object, avoiding frequent 2D rendering processes, effectively reducing the time consumption of the target tracking process, and improving the efficiency of target tracking.

[0128] Figure 5 A schematic diagram of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device may include at least one processor; and at least one memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor can execute the panoramic video target tracking method provided in the embodiments of this application by calling the program instructions.

[0129] The aforementioned electronic device can be a panoramic video target tracking device, and this embodiment does not limit the specific form of the aforementioned electronic device.

[0130] Figure 5 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown. Figure 5 The electronic device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 5As shown, the electronic device is represented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, memory 430, communication interface 420, and communication bus 440 connecting different system components (including memory 430 and processor 410).

[0132] Communication bus 440 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0133] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0134] Memory 430 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media). In these cases, each drive can be connected to the communication bus 440 via one or more data media interfaces. The memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0135] A program / utility having a set (at least one) of program modules can be stored in memory 430. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0136] The electronic device can also communicate with one or more external devices (such as a keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the electronic device, and / or with any device that enables the electronic device to communicate with one or more other computing devices (such as a network card, modem, etc.). This communication can be performed through communication interface 420. Furthermore, the electronic device can also communicate through a network adapter ( Figure 5 (Not shown) communicates with one or more networks (e.g., Local Area Network (LAN), Wide Area Network (WAN), and / or public networks, such as the Internet). The aforementioned network adapter can communicate with other modules of the electronic device via the communication bus 440. It should be understood that, although... Figure 5 Not shown, other hardware and / or software modules can be used in conjunction with electronic devices, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Drives (RAID) systems, tape drives, and data backup storage systems.

[0137] The processor 410 executes various functional applications and data processing by running programs stored in the memory 430, such as implementing the panoramic video target tracking method provided in the embodiments of this application.

[0138] This application also provides a computer-readable storage medium storing computer instructions that cause the computer to execute the panoramic video target tracking method provided in this application.

[0139] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0140] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0141] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0142] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0144] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0146] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0147] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A panoramic video target tracking method, characterized in that, include: Information about the target to be tracked in the current sampling frame of the target video is obtained; wherein, the target video is obtained by planarizing a panoramic video; Based on the information of the target to be tracked, calculate the optical flow of the target to be tracked between the current sampling frame and the next sampling frame; Based on the optical flow, determine the reference coordinate matrix of the target to be tracked in the next sampling frame; The reference coordinate matrix is ​​corrected to obtain the effective coordinate matrix of the target to be tracked in the next sampling frame; The correction of the reference coordinate matrix includes: The reference coordinate matrix is ​​offset according to a preset offset vector to obtain several offset coordinate matrices of the target to be tracked; The reference coordinate matrix is ​​corrected based on the image features of the target to be tracked at each offset coordinate matrix.

2. The method according to claim 1, characterized in that, Based on the information of the target to be tracked, the optical flow of the target to be tracked between the current sampling frame and the next sampling frame is calculated, including: Based on the information of the target to be tracked, determine each pixel point contained in the target to be tracked; Calculate the optical flow of each pixel between the current sampling frame and the next sampling frame; The average value of the optical flow of each pixel is determined as the optical flow of the target to be tracked.

3. The method according to claim 1, characterized in that, The optical flow includes horizontal optical flow and vertical optical flow; based on the optical flow, determining the reference coordinate matrix of the target to be tracked in the next sampling frame includes: The horizontal optical flow is determined as the horizontal displacement value of the target to be tracked; The vertical optical flow is determined as the vertical displacement value of the target to be tracked; Based on the horizontal displacement value and the vertical displacement value, the reference coordinate matrix of the target to be tracked in the next sampling frame is determined.

4. The method according to claim 1, characterized in that, Based on the image features of the target to be tracked at each offset coordinate matrix, the reference coordinate matrix is ​​corrected, including: Calculate the feature vector of the target to be tracked at each offset coordinate matrix; The reference coordinate matrix is ​​corrected based on each of the aforementioned feature vectors.

5. The method according to claim 4, characterized in that, Calculate the feature vector of the target to be tracked at each offset coordinate matrix, including: Calculate the color histogram and gradient histogram of the target to be tracked at each offset coordinate matrix; Based on the color histogram and the gradient histogram, the feature vectors of the target to be tracked at each offset coordinate matrix are determined.

6. The method according to claim 4, characterized in that, The reference coordinate matrix is ​​corrected based on each of the aforementioned feature vectors, including: The similarity between each of the aforementioned feature vectors and the target feature vector is calculated, wherein the target feature vector is calculated based on the information of the target to be tracked in the current sampling frame; The reference coordinate matrix is ​​corrected based on the similarity.

7. The method according to claim 6, characterized in that, Based on the similarity, the reference coordinate matrix is ​​corrected, including: Determine whether the maximum value among all similarities is greater than the first threshold; If the similarity is greater than the first threshold, the offset coordinate matrix corresponding to the feature vector with the highest similarity is determined as the effective coordinate matrix of the target to be tracked. If the value is not greater than the first threshold, the offset coordinate matrix corresponding to the feature vector with the highest similarity is determined as the corrected reference coordinate matrix. The modified reference coordinate matrix is ​​repeatedly modified until the maximum value of the obtained similarity is greater than the first threshold.

8. The method according to claim 1, characterized in that, The method further includes: Convert the effective coordinate matrix into an effective spherical coordinate matrix; Based on the effective spherical coordinate matrix, the spherical distance of the target to be tracked between the next sampling frame and the current sampling frame is determined.

9. The method according to claim 8, characterized in that, The method further includes: After determining that the spherical distance is greater than the second threshold, 2D rendering is performed on the target to be tracked.

10. An electronic device, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 9 by calling the program instructions.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1 to 9.

Images