Target tracking method, electronic equipment and storage medium
By acquiring the motion vector information of historical images, the mapping relationship between images is constructed, and the search area is adjusted, and the search area failure problem caused by image offset during target tracking is solved, and the accurate target locking of the servo system is achieved.
Patent Information
- Application Number
- CN202411188767.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-26
AI Technical Summary
During the target tracking process, the search area failure caused by image offset, resulting in the servo system being unable to lock the target normally. The prior art can easily cause tracking errors and jumps by expanding the search area or adjusting the direction.
By obtaining the motion vector information of several frames of historical images, the background offset between the current frame image and the previous frame image is determined, and the mapping relationship is constructed, and the search area in the previous frame image is mapped to the current frame image to adjust Search the area to avoid the search area failure caused by image offset.
It realizes that without affecting the internal principles and effects of the tracking algorithm, effectively adjusts the search area to avoid targets from tracking, ensures that the servo system accurately locks the target, and solves the tracking failure problem caused by image drift.
Smart Images

Figure CN119027453A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target tracking, and in particular to a target tracking method, electronic equipment and storage medium. Background Art
[0002] For the control products of the tracker-linked servo system, when the user selects a tracking target, the tracker will use its own search area to locate the tracking target, and send corresponding information to the servo system to link the servo system to lock the selected target. Therefore, it is necessary to ensure that the tracking algorithm of the tracker can always track the target correctly to ensure that the servo system accurately locks the target. However, in actual applications, due to the diversity of the location and scale of the selected target, when the servo system quickly locks the selected tracking target, a huge offset will occur between the two adjacent frames of the image, which will cause the target to directly deviate from the search area of the tracking algorithm in the tracker, and the subsequent tracking tasks cannot be carried out normally.
[0003] Therefore, it is necessary to adjust the search area of the tracking algorithm in the tracker during the process of locking the target by the linkage servo system, so that the target selected by the user is always in the search area by adjusting the search area, so as to track the selected target normally. However, the current methods such as expanding the search area and expanding the search area in the corresponding direction according to the position of the target in the image will directly affect the effect of the tracking algorithm itself in the tracker, and will directly lead to the jump between similar targets during the tracking process, and it is easy to track the wrong target. Summary of the invention
[0004] The purpose of the present invention is to provide a target tracking method, an electronic device and a storage medium, which maps the search area of a tracker in a previous frame image to a current frame image according to a mapping relationship, realizes a process in which the search area is adjusted as the image shifts, and avoids the situation in which the search area of the tracker cannot normally capture the target object due to image shift.
[0005] In order to solve the above technical problems, the present invention provides a target tracking method, comprising:
[0006] By encoding and decoding the original video before the current frame image, motion vector information of several frames of historical images before the current frame image is obtained;
[0007] Determine the background offset between the current frame image and the previous frame image based on motion vector information of several frames of historical images;
[0008] Constructing a mapping relationship between the current frame image and its previous frame image based on the background offset;
[0009] The mapping relationship is used to map the position of the search area of the tracker in the previous frame image to the current frame image, so that the tracker tracks the target object based on the mapped search area.
[0010] Optionally, the step of performing encoding and decoding on the original video before the current frame image to obtain motion vector information of several frames of historical images before the current frame image includes:
[0011] Get the original video before the current frame image;
[0012] Encoding the original video to obtain an encoded compressed video;
[0013] The compressed video is decoded by a hardware accelerator to extract motion vector information of several frames of historical images.
[0014] Optionally, encoding the original video includes:
[0015] Determining a frame type of each frame of the original video, and calculating a prediction error and motion vector information of each frame of the original video according to the frame type;
[0016] Converting the prediction error of each frame of the original video from the spatial domain to the frequency domain;
[0017] Determine the transformation coefficients in the process of converting each frame of the image, and quantize the transformation coefficients;
[0018] The quantized transform coefficients and the motion vector information of each frame of image are compressed by entropy coding to obtain compressed video.
[0019] Optionally, the determining the background offset between the current frame image and the previous frame image based on the motion vector information of several frames of historical images includes:
[0020] Determine the image offset between each historical image frame and its own reference frame based on motion vector information of several frames of historical images;
[0021] The background offset between the current frame image and the previous frame image is determined according to the image offset of each frame of historical images.
[0022] Optionally, before determining the image offset between each historical image frame and its own reference frame based on the motion vector information of the plurality of historical image frames, the method further includes:
[0023] Determine a reference frame of each historical image frame from the reference frame indexes of the plurality of historical image frames;
[0024] Filtering out motion vector information corresponding to a specific historical image as target motion vector information; the specific historical image is a historical image whose reference frame is a previous frame image of the image itself;
[0025] The determining of the image offset between each historical image frame and its own reference frame based on the motion vector information of the plurality of historical image frames comprises:
[0026] An image offset between a historical image corresponding to the target motion vector information and its own reference frame is determined based on the target motion vector information.
[0027] Optionally, after filtering out the motion vector information corresponding to the specific historical image, the method further includes:
[0028] Filter out the motion vector information corresponding to the specific historical image, which has the same macroblock size;
[0029] A preset number of motion vector information is retained from the motion vector information with the same macroblock size as the target motion vector information.
[0030] Optionally, determining the background offset between the current frame image and its previous frame image according to the image offset of each frame of historical images includes:
[0031] For any frame of historical image, determine the horizontal offset and vertical offset between itself and the previous frame of image;
[0032] respectively collecting statistics on the horizontal offset and the vertical offset;
[0033] The horizontal offset that appears most frequently in the statistical results is used as the horizontal offset in the background offset between the current frame image and the previous frame image;
[0034] The vertical offset that appears most frequently in the statistical results is used as the vertical offset in the background offset between the current frame image and the previous frame image.
[0035] Optionally, constructing a mapping relationship between the current frame image and its previous frame image based on the background offset includes:
[0036] Taking the background offset as a reference, determining a number of previous frame positions in a previous frame image corresponding to a number of current positions in a current frame image;
[0037] Establishing a linear equation system according to the corresponding relationship between the current position and the previous frame position;
[0038] A mapping relationship between the current frame image and its previous frame image is obtained based on the linear equation group.
[0039] In order to solve the above technical problems, the present invention further provides an electronic device, comprising:
[0040] Memory for storing computer programs;
[0041] A processor is used to implement the steps of the target tracking method as described above.
[0042] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the target tracking method as described above are implemented.
[0043] The present invention provides a target tracking method, which first obtains motion vector information of several frames of historical images, and then determines the image offset between the current frame image and the previous frame image according to the motion vector information of the historical images, thereby determining the mapping relationship between the current frame image and the previous frame image, so as to map the search area of the tracker in the previous frame image to the current frame image according to the mapping relationship, and realizes the process of adjusting the search area with the image offset, thereby avoiding the situation that the search area of the tracker cannot normally capture the target object due to the image offset, and solves the problem of direct tracking failure due to image drift in the tracking target locking process without significantly increasing the algorithm time consumption, and will not affect the internal principle and effect of the tracking algorithm, and effectively realizes the point selection target locking tracking process of the tracker linkage servo control.
[0044] The present invention also provides an electronic device and a computer-readable storage medium, which have the same beneficial effects as the above target tracking method. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the prior art and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 A flow chart of a target tracking method provided by the present invention;
[0047] Figure 2 A schematic diagram of a video encoding process provided by the present invention;
[0048] Figure 3 A schematic diagram of a flow chart of a motion vector screening process provided by the present invention;
[0049] Figure 4 A schematic diagram of a flow chart of another target tracking method provided by the present invention;
[0050] Figure 5 A schematic diagram of target tracking comparison before and after the search area is adjusted provided by the present invention;
[0051] Figure 6 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION
[0052] The core of the present invention is to provide a target tracking method, an electronic device and a storage medium, which maps the search area of the tracker in the previous frame image to the current frame image according to a mapping relationship, and realizes the process of adjusting the search area as the image shifts, thereby avoiding the situation where the search area of the tracker cannot capture the target object normally due to image shift.
[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In order to solve the above technical problems, the present invention provides a target tracking method. Figure 1 As shown, including:
[0055] S11: Obtain motion vector information of several frames of historical images before the current frame image by encoding and decoding the original video before the current frame image;
[0056] It is not difficult to understand that if you want to adjust the search area of the tracker in the current frame, you need to first determine the specific offset of the entire image between the current frame and the previous frame. At this time, you can determine the motion trajectory and motion direction of the lens based on several frames of historical images before the current frame image, so as to predict the background offset between the current frame image and the previous frame image. Therefore, you can first obtain the motion vector information of several frames of historical images. The acquisition of the motion vector information of the historical images can be obtained by encoding and decoding the original video before the current frame image. For any frame image, the motion vector information specifically includes the reference frame index, the reference frame position and the current frame position. The specific content and implementation method of the motion vector information are not specifically limited in this application. The process of obtaining the motion vector information can be specifically implemented by encoding and decoding the video. The application does not make any special restrictions here. The specific number of frames of the historical images obtained is not specifically limited in this application, and can be selected according to the actual operation requirements of the lens.
[0057] S12: determining a background offset between a current frame image and a previous frame image based on motion vector information of several frames of historical images;
[0058] It is not difficult to understand that by obtaining the motion vector information of the historical images, the approximate background offset between two adjacent frames of images during the lens movement can be roughly determined, and thus the background offset between the current frame image and its previous frame image can be statistically summarized through the motion vector information of several frames of historical images, so as to compensate the search area later. The process of determining the background offset between the current frame image and its previous frame image based on the motion vector information of several frames of historical images specifically includes determining the image offset between each frame of historical images and its own reference frame based on the motion vector information of several frames of historical images; and determining the background offset between the current frame image and its previous frame image according to the image offset of each frame of historical images.
[0059] It is understandable that by comparing the reference frame position and the current frame position in the motion vector information, the image offset between any frame of historical image and its reference frame can be determined, thereby determining the motion of the lens corresponding to each frame of historical image. The image offset specifically includes the horizontal offset of the image in the horizontal direction and the vertical offset of the image in the vertical direction. The image offset can also be directly expressed by coordinates or other methods, and this application does not make any special restrictions here.
[0060] Furthermore, after determining the image offset between each historical image frame and its own reference frame, the motion trajectory of the current shot can be roughly determined based on the image offset of all historical images, thereby estimating the background offset between the current frame image and its previous frame image, that is, predicting the background offset between the current frame image and the previous frame image. Specifically, the effective motion vectors in each historical image frame can be counted, and the data with the largest statistics can be used as the offset between the current frame and its reference frame.
[0061] S13: constructing a mapping relationship between the current frame image and its previous frame image based on the background offset;
[0062] It is not difficult to understand that the background offset between the current frame image and its previous frame image is also the background offset between two adjacent frames, which can determine the position change between two adjacent frames, and then construct a mapping relationship between two adjacent frames; the mapping relationship between the current frame image and its previous frame image refers to the position mapping relationship between the two frames, that is, a certain position (coordinate point) in the previous frame image will have a change in position coordinates due to the offset captured by the lens. The present application predicts the change in this position coordinate by constructing a mapping relationship between the current frame image and its previous frame image, and adjusts the search area of the tracker based on this, thereby avoiding the influence of the background image offset caused by lens capture on the target tracking process.
[0063] S14: Mapping the position of the search area of the tracker in the previous frame image to the current frame image by using the mapping relationship, so that the tracker tracks the target object based on the mapped search area.
[0064] It can be understood that after the mapping relationship between the current frame image and its previous frame image is constructed, the search area of the tracker in the previous frame image can be mapped to the current frame image based on this mapping relationship. Specifically, several reference position points in the search area can be determined, and the target position points of each reference position point in the current frame image can be determined based on the mapping relationship, and then the corresponding search area can be generated according to the determined target position point. At this time, the mapped search area is the search area after the position adjustment is made after considering the image offset caused by the lens or other problems in the current frame. The tracker can track the target object according to the adjusted search area, thereby avoiding the situation where the target object is out of the original search area due to the image offset caused by the servo system during the lens locking process.
[0065] It should be noted that after the mapping relationship is determined, the coordinates of the target object in the previous frame image in the tracking algorithm can be mapped to the corresponding position in the current frame image through the mapping relationship, and then the search area of the tracker corresponding to the target object in the previous frame image is also synchronously mapped along with the mapping of the coordinates of the target object. The position of the search area is adjusted by mapping the search area, so that the position adjustment process is used to compensate for the offset of the background between frames caused by the servo quickly locking the target.
[0066] It is not difficult to understand that the target tracking method provided by the present application is a pre-compensation process of the search area in the current frame performed in real time when each frame image is acquired. Since what is finally determined is the mapping relationship between the current frame image and its previous frame image, for the current frame whose reference frame is the previous frame, its encoding process is completed with reference to the previous frame image, so the offset between itself and the previous frame image can be accurately determined based on this mapping relationship; however, if the current frame is an independently coded frame, it does not rely on other frames for encoding. At this time, when adjusting the search area, the mapping relationship corresponding to the previous frame image can be directly used to adjust the search area.
[0067] As a specific embodiment, see Figure 5 As shown, Figure 5 The left half is a brief schematic diagram of two adjacent frames of images of the tracker and the servo system in the process of tracking the target object without using the target tracking method provided by the present application, and the right half is a brief schematic diagram of two adjacent frames of images of the tracker and the servo system in the process of tracking the target object with the target tracking method provided by the present application; the image in the upper half is the previous frame image, the image in the lower half is the current frame image, the box is a schematic diagram of the search area of the tracker, the car is the target object to be tracked, and the tree is a schematic diagram of the image background; it is not difficult to find that, without using the target tracking method provided by the present application, The image background has a certain offset, and due to this offset, the target object has left the search area of the tracker, causing the tracker to lose tracking of the target object; when the target tracking method provided by the present application is adopted, the search area of the tracker is adjusted according to the background offset, that is, it is adjusted to the left from the position of the dotted frame to the position of the solid frame, and the translation amount of the search area to the left is consistent with the offset of the background tree from the previous frame position to the current frame position, and the offset of the background tree from the previous frame position to the current frame position is also the background offset between the current frame image and its previous frame image determined in the present application. This keeps the target object within the search area of the tracker, avoids the loss of the target object, and ensures the accurate tracking process of the target object by the tracker.
[0068] The present invention designs a method for adjusting and compensating the search area of the tracker based on video coding and decoding, so that the target can be correctly locked in the process of the tracker linkage servo system control, and the motion vector information of each frame can be directly output by hard decoding, which can solve the problem of direct tracking failure due to drift in the tracking target locking process without increasing the algorithm time consumption, and does not affect the internal principle and effect of the algorithm, and effectively realizes the point-selected target locking and tracking of the tracker linkage servo control. It replaces the existing technology of changing the internal logic of the tracking algorithm, and the adjustment process of the entire search area is independent of the tracking algorithm and servo control, realizing the process of effectively locking the target without affecting the overall product control. In the process of servo fast locking the target, it is ensured that the tracking algorithm can effectively and correctly track the target object, and control the servo system to accurately and stably lock the target.
[0069] The present invention provides a target tracking method, which first obtains motion vector information of several frames of historical images, and then determines the image offset between the current frame image and the previous frame image according to the motion vector information of the historical images, thereby determining the mapping relationship between the current frame image and the previous frame image, so as to map the search area of the tracker in the previous frame image to the current frame image according to the mapping relationship, and realizes the process of adjusting the search area with the image offset, thereby avoiding the situation that the search area of the tracker cannot normally capture the target object due to the image offset, and solves the problem of direct tracking failure due to image drift in the tracking target locking process without significantly increasing the algorithm time consumption, and will not affect the internal principle and effect of the tracking algorithm, and effectively realizes the point selection target locking tracking process of the tracker linkage servo control.
[0070] Based on the above embodiments:
[0071] As an optional embodiment, obtaining motion vector information of several frames of historical images before the current frame image includes:
[0072] Get the original video before the current frame image;
[0073] Encode the original video to obtain an encoded compressed video;
[0074] The compressed video is decoded by a hardware accelerator to extract motion vector information of several frames of historical images.
[0075] It is not difficult to understand that the process of extracting motion vector information of several frames of historical images in the original video can be realized by multiplexing the video encoding process in the video display process and adding the hardware decoding process of the compressed video. The video captured by the lens will be displayed on the display device through the process of software encoding and software decoding. Therefore, after the lens or movement imaging of the product captures the original video of the target scene, the video is first encoded and processed by the video stream compression method in the video display process, and then the encoded video stream is decoded and processed by the hardware accelerator, so as to directly obtain the motion vector and reference frame information of each frame image in the video stream except the independently encoded frame, and the motion vector information can be further screened and processed according to the macroblock size and reference frame index, etc., to obtain effective motion vector prediction information. The specific type and implementation method of the hardware accelerator are not particularly limited in this application.
[0076] Specifically, the software encoding process of the video display process can be reused, and the process of hardware decoding the encoded compressed video can be added to directly extract the motion vector information of each frame of the image through hard decoding, thereby reducing the time consumption of the entire process. The process of using hard decoding of the video encoding and decoding information to accurately compensate for the inter-frame background offset caused by the servo system quickly locking the target is realized, thereby ensuring the real-time performance of the tracking algorithm.
[0077] As an optional embodiment, encoding the original video includes:
[0078] Determine the frame type of each frame image in the original video, and calculate the prediction error and motion vector information of each frame image according to the frame type;
[0079] Convert the prediction error of each frame in the original video from the spatial domain to the frequency domain;
[0080] Determine the transformation coefficients in the conversion process of each frame of the image and quantize the transformation coefficients;
[0081] The quantized transform coefficients and motion vector information of each frame of image are compressed through entropy coding to obtain compressed video.
[0082] Understandably, see Figure 2As shown, after obtaining the raw data corresponding to the raw video of the scene captured by the imaging lens or the movement, the software decoding process will encode the captured video. At this time, the frames in the video stream are divided into three types: I frame (independently encoded frame, not dependent on other frames), P frame (depending on the previous I frame or P frame for predictive encoding, that is, the reference frame is the previous frame of itself), B frame (depending on the previous and next I frame or P frame for bidirectional predictive encoding, that is, the reference frame is the previous frame or the next frame of itself). For different types of frames, the calculation methods of their prediction error and motion vector information are different. For I frame, the prediction error and motion vector information need to be calculated by intra-frame prediction. For P frame and B frame, the prediction error and motion vector information need to be calculated by inter-frame prediction using their reference frames.
[0083] Furthermore, in order to reduce invalid information in the encoded video and compress the subsequent calculation amount, after calculating the prediction error and motion vector information of each frame image, it is necessary to transform each prediction error, convert the prediction error in the spatial domain to the frequency domain, and quantize the transformation coefficients in this process to reduce data redundancy. The quantized transformation coefficients, motion vector information and other encoding parameters will be further compressed by entropy coding, and the encoded data will be used to realize the encoding process of the original video. It is also possible to further filter in the process of compressing data to reduce data redundancy. The encoder will also manage the reference frame list to ensure the validity and timely update of the reference frames of each frame image.
[0084] It should be noted that see Figure 3 As shown, the encoded and compressed video stream will be hard-decoded by the hardware accelerator. The hardware accelerator directly reads the entropy-coded data from the video stream and performs entropy decoding. After entropy decoding, it can directly output the quantized transform coefficients, motion vector information and other encoding parameters. At this time, the frame number, frame type and motion vector information of each frame image can be directly extracted, where the motion vector information includes the reference frame index, macroblock size, reference frame position (src_x, src_y) and current frame position (dst_x, dst_y).
[0085] Specifically, the present application directly encodes and decodes the video stream of the servo locking process, and by obtaining the inter-frame offset information of the video and combining it with the tracking algorithm, it can directly and effectively solve the offset problem that occurs in the locking process of such products. By solving the decoded information of the video stream, it directly compensates for the large inter-frame background offset caused by the servo quickly locking the target, thereby realizing a compensation method that will not affect the tracking algorithm and the operation of the servo itself.
[0086] As an optional embodiment, before determining the image offset between each historical image frame and its own reference frame based on the motion vector information of the plurality of historical image frames, the method further includes:
[0087] Determine a reference frame of each historical image frame from the reference frame indexes of the plurality of historical image frames;
[0088] Filtering out motion vector information corresponding to a specific historical image as target motion vector information; the specific historical image is a historical image whose reference frame is a previous frame image of the image itself;
[0089] The image offset between each historical image frame and its own reference frame is determined based on motion vector information of several frames of historical images, including:
[0090] An image offset between a historical image corresponding to the target motion vector information and its own reference frame is determined based on the target motion vector information.
[0091] It is not difficult to understand that in this application, the background offset between the previous frame image and the current frame needs to be used to adjust the search area in the current frame image. Therefore, for all image frames in the original video, only the specific historical image whose reference frame is its previous frame image can provide effective and accurate prediction information for the construction process of the mapping relationship between the current frame image and its previous frame image. Therefore, only the motion vector information corresponding to the specific historical image is the effective target motion vector information. Figure 3 As shown, at this time, the reference frame index of each frame image obtained during the video decoding process can be used to determine which frames have a reference frame as the previous frame image, and select them as specific historical images. The process of determining the image offset between the historical image corresponding to the target motion vector information and its own reference frame based on the target motion vector information can be directly calculated and implemented through the reference frame position and the current frame position in the target motion vector information, and this application does not make any special restrictions here.
[0092] Specifically, the present application needs to adjust the position of the search area through the mapping relationship between the current frame and the previous frame. Therefore, when predicting the position adjustment result of the search area of the current frame, only the specific historical image whose reference frame is its own previous frame image can provide effective prediction information. Therefore, it is only necessary to filter out the motion vector information corresponding to the specific historical image to determine the background offset, thereby further reducing the subsequent calculation amount.
[0093] As an optional embodiment, after the motion vector information corresponding to the specific historical image is screened out, the method further includes:
[0094] Filter out the motion vector information corresponding to the specific historical image, which has the same macroblock size;
[0095] A preset number of motion vector information is retained from the motion vector information with the same macroblock size as the target motion vector information.
[0096] It is not difficult to understand that, see Figure 3 As shown, on the basis of filtering the motion vector information of the previous frame with the reference frame as the reference frame index, the motion vector information of uniform size, that is, the motion vector information corresponding to the macroblock of the same size, can be further filtered according to the macroblock size, and the number of the filtered motion vector information is randomly sampled with certain restrictions, and only a fixed number of valid motion vector information is retained as the target motion vector information. The specific value of the preset number and the selection method are not particularly limited in this application. The process of retaining the preset number is only to reduce the number of motion vector information, which can be achieved by random selection and other methods.
[0097] Specifically, the amount of final target motion vector information can be further reduced according to the macroblock size or a preset number of random samples, thereby further simplifying the calculation, improving the efficiency of the entire algorithm process, and reducing time consumption.
[0098] As an optional embodiment, determining the background offset between the current frame image and the previous frame image according to the image offset of each frame of historical images includes:
[0099] For any frame of historical image, determine the horizontal offset and vertical offset between itself and the previous frame of image;
[0100] Count the horizontal offset and vertical offset respectively;
[0101] The horizontal offset that appears most frequently in the statistical results is used as the horizontal offset in the background offset between the current frame image and the previous frame image;
[0102] The vertical offset that appears most frequently in the statistical results is used as the vertical offset in the background offset between the current frame image and the previous frame image.
[0103] It can be understood that the background offset between the current frame and the reference frame can be solved based on the filtered effective target motion vector information, the horizontal offset is the difference between dst_x and src_x, and the vertical offset is the difference between dst_y and src_y, and then the solved horizontal offset and vertical offset are respectively counted, and the solved offsets of the horizontal offset and vertical offset with the largest number, that is, the most frequent occurrences in the statistical process are selected as the overall background offset between the current frame and the reference frame. The present application does not make any special restrictions on the specific method for determining the background offset between the current frame and the reference frame, and is not limited to the process of defining the offset with the most frequent statistics as the offset between the current frame and the reference frame in this embodiment, and the statistical results can also be processed by weighted averaging to determine the offset between the current frame and the reference frame.
[0104] Specifically, the background offset between the current frame image and its previous frame image can be directly determined by statistically analyzing the offset of historical images, which is simple, effective, and easy to implement, and is conducive to the simple implementation of the entire target tracking method.
[0105] As an optional embodiment, constructing a mapping relationship between a current frame image and a previous frame image based on the background offset includes:
[0106] Based on the background offset, determine a number of previous frame positions in the previous frame image corresponding to a number of current positions in the current frame image;
[0107] Establish a linear equation system based on the corresponding relationship between the current position and the position of the previous frame;
[0108] The mapping relationship between the current frame image and its previous frame image is obtained based on the linear equation group.
[0109] It is not difficult to understand that after determining the overall background offset between the current frame image and its previous frame image, several groups of previous frame positions and current frame positions can be selected based on this background offset, and a linear equation system can be established based on these groups of position relationships, and finally the linear equation system is solved to obtain the mapping relationship between the current frame image and its previous frame image. This application does not make any special restrictions on the specific selection basis and implementation method of the several corresponding positions required when establishing the linear equation system, and can be selected according to the actual application situation and implemented by determining a special iconic position in the background image.
[0110] Specifically, the mapping relationship between the current frame image and its previous frame image can be directly solved by establishing a linear equation group of corresponding positions. This is simple, effective, and easy to implement, which is conducive to the simple implementation of the entire target tracking method.
[0111] As an optional embodiment, obtaining a mapping relationship between a current frame image and a previous frame image based on a linear equation group includes:
[0112] The singular value decomposition method is used to solve the linear equations to obtain the homography matrix between the current frame image and its previous frame image;
[0113] The homography matrix is determined as the mapping relationship between the current frame image and its previous frame image.
[0114] It is not difficult to understand that, see Figure 4 As shown, after the above-mentioned encoding, hard decoding, screening of effective motion vectors, and statistical calculation of horizontal and vertical offsets of the original video, a linear equation group can be established according to the selection of several groups of reference frame positions and current frame positions corresponding to the background offset, and then the singular value decomposition method is used to solve the linear equation group to obtain the homography matrix of the current frame image. The solved homography matrix can be directly determined as the mapping relationship between the current frame image and its previous frame image, and the homography matrix is used to directly map the corresponding position; the coordinates of the target object in the previous frame image in the tracking algorithm are mapped to the corresponding position in the current frame image through the homography matrix, and the corresponding search area corresponding to the target object in the previous frame image is also mapped synchronously with the mapping of the coordinates of the target object, thereby realizing the compensation process for the background offset.
[0115] It should be noted that, for P frames and B frames, their previous frame images can be used as their reference frames. Therefore, if the current frame image is a P frame or a B frame, the mapping relationship between the current frame image and its previous frame image is also the mapping relationship between the current frame image and its reference frame image; however, for I frames, their encoding process is independent. Therefore, if the current frame image is an I frame, the mapping relationship between the current frame image and its previous frame image needs to be implemented using the mapping relationship of its previous frame image, that is, the homography matrix of the independently encoded frame directly uses the homography matrix of the previous frame.
[0116] Specifically, by solving the video stream decoding information, it is possible to directly compensate for the large background offset between frames caused by the servo quickly locking the target. The search area in the previous frame image is mapped to the corresponding position of the current frame image through the homography matrix, and the background offset between frames caused by the servo fast locking process is offset through decoding compensation.
[0117] In order to solve the above technical problems, the present invention also provides a target tracking system, comprising:
[0118] A motion vector acquisition unit, used to acquire motion vector information of several frames of historical images before the current frame image;
[0119] A historical image offset determination unit, used to determine the image offset between each historical image frame and its own reference frame based on motion vector information of a plurality of historical image frames;
[0120] A current frame offset determination unit, used to determine the background offset between the current frame image and the previous frame image according to the image offset of each frame of historical images;
[0121] A mapping relationship determining unit, configured to construct a mapping relationship between the current frame image and its previous frame image based on the background offset;
[0122] The search area adjustment unit is used to map the position of the search area of the tracker in the previous frame image to the current frame image by using the mapping relationship, so that the tracker tracks the target object based on the mapped search area.
[0123] For an introduction to a target tracking system provided by the present invention, please refer to the above-mentioned embodiment of the target tracking method, and the present invention will not be repeated here.
[0124] See also Figure 6 To solve the above technical problems, the present invention further provides an electronic device, including:
[0125] A memory 21, used for storing computer programs;
[0126] The processor 22 is used to implement the steps of the target tracking method as described above.
[0127] Among them, the processor 22 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 22 can be implemented in at least one hardware form of DSP (Digital Signal Processor), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may integrate a GPU (graphics processing unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 22 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0128] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the method for determining the state of charge of a single cell disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, data of the method for determining the state of charge of a single cell, etc.
[0129] In some embodiments, the electronic device may further include a display screen, an input and output interface, a communication interface, a power supply, and a communication bus.
[0130] It can be understood by those skilled in the art that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure.
[0131] For an introduction to an electronic device provided by the present invention, please refer to the above-mentioned embodiment of the target tracking method, and the present invention will not be described in detail here.
[0132] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the target tracking method as described above are implemented.
[0133] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present application. Specifically, the computer-readable storage medium may include but is not limited to any type of disk, including a floppy disk, an optical disk, and a mobile hard disk, or any type of medium or device suitable for storing instructions, data, etc., and the present application does not make any special restrictions here.
[0134] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above-mentioned embodiment of the target tracking method, and the present invention will not be repeated here.
[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0136] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0137] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0138] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A target tracking method, characterized in that: include: By encoding and decoding the original video before the current frame image, motion vector information of several frames of historical images before the current frame image is obtained; Determine the background offset between the current frame image and the previous frame image based on motion vector information of several frames of historical images; Constructing a mapping relationship between the current frame image and its previous frame image based on the background offset; The mapping relationship is used to map the position of the search area of the tracker in the previous frame image to the current frame image, so that the tracker tracks the target object based on the mapped search area.
2. The target tracking method according to claim 1, characterized in that: The step of obtaining motion vector information of several frames of historical images before the current frame image by encoding and decoding the original video before the current frame image includes: Get the original video before the current frame image; Encoding the original video to obtain an encoded compressed video; The compressed video is decoded by a hardware accelerator to extract motion vector information of several frames of historical images.
3. The target tracking method according to claim 2, characterized in that: The encoding of the original video comprises: Determining a frame type of each frame of the original video, and calculating a prediction error and motion vector information of each frame of the original video according to the frame type; Converting the prediction error of each frame of the original video from the spatial domain to the frequency domain; Determine the transformation coefficients in the process of converting each frame of the image, and quantize the transformation coefficients; The quantized transform coefficients and the motion vector information of each frame of image are compressed by entropy coding to obtain compressed video.
4. The method for tracking a target in a video according to claim 1, wherein: The determining the background offset between the current frame image and the previous frame image based on the motion vector information of several frames of historical images includes: Determine the image offset between each historical image frame and its own reference frame based on motion vector information of several frames of historical images; The background offset between the current frame image and the previous frame image is determined according to the image offset of each frame of historical images.
5. The target tracking method according to claim 4, characterized in that: Before determining the image offset between each historical image frame and its own reference frame based on the motion vector information of the plurality of historical image frames, the method further includes: Determine a reference frame of each historical image frame from the reference frame indexes of the plurality of historical image frames; Filtering out motion vector information corresponding to a specific historical image as target motion vector information; the specific historical image is a historical image whose reference frame is a previous frame image of the image itself; The determining of the image offset between each historical image frame and its own reference frame based on the motion vector information of the plurality of historical image frames comprises: An image offset between a historical image corresponding to the target motion vector information and its own reference frame is determined based on the target motion vector information.
6. The target tracking method according to claim 5, characterized in that: After the motion vector information corresponding to the specific historical image is screened out, the method further includes: Filter out the motion vector information corresponding to the specific historical image, which has the same macroblock size; A preset number of motion vector information is retained from the motion vector information with the same macroblock size as the target motion vector information.
7. The target tracking method according to claim 4, characterized in that: The determining of the background offset between the current frame image and the previous frame image according to the image offset of each frame of historical images includes: For any frame of historical image, determine the horizontal offset and vertical offset between itself and the previous frame of image; respectively collecting statistics on the horizontal offset and the vertical offset; The horizontal offset that appears most frequently in the statistical results is used as the horizontal offset in the background offset between the current frame image and the previous frame image; The vertical offset that appears most frequently in the statistical results is used as the vertical offset in the background offset between the current frame image and the previous frame image.
8. The target tracking method according to any one of claims 1 to 7, characterized in that: The step of constructing a mapping relationship between the current frame image and its previous frame image based on the background offset includes: Taking the background offset as a reference, determining a number of previous frame positions in a previous frame image corresponding to a number of current positions in a current frame image; Establishing a linear equation system according to the corresponding relationship between the current position and the previous frame position; A mapping relationship between the current frame image and its previous frame image is obtained based on the linear equation group.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the target tracking method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target tracking method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Method for implementing machine-carried video compression and target tracking unitedly
CN101511022A
Target tracking method and system based on prior information
CN108932731A
FPGA-based unmanned aerial vehicle image processing method and device, and computer equipment
CN118354086A
Electronic image stabilization method, device and equipment and storage medium
CN118521615A
Cited By
Target tracking method, device, equipment and medium
CN119292288A
A target tracking method, apparatus, device and medium
CN119292288B
Unmanned aerial vehicle tracking method and device, photoelectric integrated equipment and medium
CN119559213A
Image inter-frame displacement detection method and device, storage medium and program product
CN121458999A
