Multi-modal target tracking method and related apparatus

Abstract

The application discloses a multi-modal target tracking method and related device, comprising: acquiring a current frame of a target object, the current frame comprising a visible light image and an infrared light image; obtaining a first target response map according to the visible light image and a first position filter; obtaining a second target response map according to the infrared light image and a second position filter; when the maximum energy value of any one of the first target response map and the second target response map is greater than a preset threshold, obtaining a visible light confidence and an infrared light confidence according to the first target response map and the second target response map; determining the position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence and the infrared light confidence. The application also discloses a tracking device. The target tracking method of the application can reduce the calculation amount, facilitate the application of the target algorithm to embedded devices, and meet the demand of real-time tracking target.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a multi-modal target tracking method and related device. BACKGROUND

[0002] In a battlefield environment, a scout often uses a single soldier helmet to scout a target, thereby tracking the target. The definition of target tracking is to estimate the position, size, and other states of the target in each subsequent frame given the target template in the initial frame. Most existing target tracking methods are applied to image sequences in a visible light scene, which conforms to human visual perception and contains rich texture information. However, when the target is in a weak light environment or under some extreme conditions such as fog and heavy rain, the imaging effect based on the visible light modality is poor, and the corresponding target tracking algorithm cannot obtain good results. Thermal infrared image sequence information is based on temperature imaging and is not sensitive to light. In night scenes, fog, and other scenes where visible light cannot be effectively adapted, thermal infrared image sequence information can also provide thermal information of the target. Therefore, multi-modal target tracking based on RGB-T combines information from two different modalities of visible light and infrared images, which has a complementary effect and improves the robustness of tracking. At the same time, since the algorithm is applied to an embedded device, it needs to achieve efficient target tracking under limited computing resources.

[0003] Currently, the multi-modal target tracking method based on RGB-T introduces a reverse tracking mechanism to track T frames in reverse after tracking T frames. The confidence is obtained by weighting the average overlap rate (the ratio of the intersection area to the union area of two rectangles) of the target frame obtained by reverse tracking and the original target frame and the average distance of the center points, and then the modality with higher confidence is selected as the tracking result. However, this method needs to track T frames in reverse for both modalities, which increases the computational load of the algorithm. Since the algorithm needs to be transplanted into an embedded device, there is a certain requirement for lightweight algorithms, so it cannot meet the real-time tracking requirement. SUMMARY

[0004] The embodiments of the present application provide a multi-modal target tracking method and related device. The multi-modal target tracking method of the present application can reduce the computational load, facilitate the application of the target algorithm to an embedded device, and meet the real-time tracking requirement.

[0005] In a first aspect, the embodiments of the present application provide a multi-modal target tracking method, comprising:

[0006] obtain a current frame of a target object, the current frame comprising a visible light image and an infrared light image; obtain a first target response map according to the visible light image and a first position filter; obtain a second target response map according to the infrared light image and a second position filter; the first position filter is obtained based on a visible light image of a historical frame, and the second position filter is obtained based on an infrared light image of the historical frame; when a maximum energy value of any one of the first target response map and the second target response map is greater than a preset threshold, obtain a visible light confidence and an infrared light confidence according to the first target response map and the second target response map; and determine a position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence and the infrared light confidence.

[0007] In a second aspect, an embodiment of the present application provides a tracking device, comprising:

[0008] an obtaining unit, configured to obtain a current frame of a target object, the current frame comprising a visible light image and an infrared light image;

[0009] a calculating unit, configured to obtain a first target response map according to the visible light image and a first position filter; obtain a second target response map according to the infrared light image and a second position filter; the first position filter is obtained based on a visible light image of a historical frame, and the second position filter is obtained based on an infrared light image of the historical frame; when a maximum energy value of any one of the first target response map and the second target response map is greater than a preset threshold, obtain a visible light confidence and an infrared light confidence according to the first target response map and the second target response map;

[0010] a determining unit, configured to determine a position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence and the infrared light confidence.

[0011] In a third aspect, an embodiment of the present application further provides a tracking device, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is configured to store program code, and the processor is configured to call the program code to execute part or all of the method in the first aspect.

[0012] In a fourth aspect, an embodiment of the present application further provides a chip system applied to an electronic device; the chip system comprises one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a circuit; the interface circuit is configured to receive a signal from a memory of the electronic device and send the signal to the processor, and the signal comprises computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes part or all of the method in the first aspect.

[0013] In a fifth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement part or all of the method in the first aspect.

[0014] In a sixth aspect, the embodiments of the present application further provide a computer program, which is executed to implement part or all of the method in the first aspect.

[0015] It can be seen that, in the scheme of the present application, the confidence of different modalities is calculated, the negative influence of the modality with large tracking error on the tracking effect is adaptively reduced, and thus the success rate of tracking is improved. Meanwhile, the Kalman filter is added to predict and estimate the position of the target object, the position of the target object is determined by the Kalman filter in the case where the position filter is unreliable, and thus the error accumulation of the position filter is avoided, so as to improve the robustness of the algorithm. In addition, the scheme of the present application does not need to repeatedly calculate the past image frames, and the computational complexity is effectively reduced.

[0016] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0018] Figure 1 A flowchart of a multi-modal target tracking method provided by an embodiment of the present application;

[0019] Figure 2 A flowchart of another multi-modal target tracking method mentioned by an embodiment of the present application;

[0020] Figure 3 A structural diagram of a tracking device provided by an embodiment of the present application;

[0021] Figure 4 A structural diagram of another tracking device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be described in detail.

[0023] The terms "first", "second", "third", and "fourth" and the like in the description and in the claims of the present application and the accompanying drawings are used for distinguishing between similar objects, not necessarily for describing a particular sequential or chronological order. The terms "include", and "has" and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a list of steps or units are not necessarily limited to the listed steps or units, but can optionally include additional steps or units not expressly listed or inherent to such process, method, product, or device.

[0024] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive or alternative embodiments. It is expressly understood that the embodiments described herein are merely examples from a whole class of comparable embodiments which those skilled in the art will readily appreciate. It is also specifically intended that the various embodiments described herein can be combined in any and all permutations.

[0025] "Multiple" means two or more. "And / or", describing the association relationship of associated objects, means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0026] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0027] Reference is made to Figure 1 , Figure 1 A flowchart of a multi-modal target tracking method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps. Figure 1

[0028] S101, the tracking device acquires a current frame of a target object, the current frame comprising a visible light image and an infrared light image.

[0029] Specifically, the tracking device acquires the visible light image and the infrared light image block from its storage unit; or the tracking device is configured with a visible light camera and an infrared camera, and the tracking device acquires the visible light image through the visible light camera and acquires the infrared light image through the infrared camera; or the tracking device acquires the visible light image and the infrared light image from other devices.

[0030] Optionally, the field of view of the target object in the visible light image and the infrared light image is the same, or can be different.

[0031] S102, the tracking device obtains a first target response map according to the visible light image and a first position filter; and obtains a second target response map according to the infrared light image and a second position filter.​

[0032] wherein the first position filter is based on the visible light image of the history frame and the second position filter is based on the infrared light image of the history frame.

[0033] The tracking device obtains the first position filter, which can be obtained by and denotes that the device performs Fourier transform on the visible light image to obtain a frequency domain signal Z v，t of the visible light image. v，t and the first position filter to obtain a first target response map.

[0034]

[0035] wherein d is the number of dimensions and λ is a constant term.

[0036] The tracking device obtains the second position filter, which can be obtained by and denotes that the device performs Fourier transform on the infrared light image to obtain a frequency domain signal Z i，t of the infrared light image. i，t and the second position filter to obtain a second target response map.

[0037]

[0038] S103, when the maximum energy value of any one of the first target response map and the second target response map is greater than a preset threshold, obtaining a visible light confidence and an infrared light confidence according to the first target response map and the second target response map.

[0039] In one possible embodiment, obtaining the visible light confidence and the infrared light confidence according to the first target response map and the second target response map comprises:

[0040] obtaining an average energy ratio of the first target response map according to the maximum energy value, the minimum energy value and the average energy value of the first target response map, obtaining an average energy ratio of the second target response map according to the maximum energy value, the minimum energy value and the average energy value of the second target response map, and obtaining the visible light confidence and the infrared light confidence according to the maximum energy value of the first target response map, the maximum energy value of the second target response map, the average energy ratio of the first target response map and the average energy ratio of the second target response map.

[0041] Specifically, after obtaining the first target response map and the second target response map, the tracking device obtains a maximum energy value of the first target response map and a maximum energy value of the second target response map, and if any one of the maximum energy value of the first target response map and the maximum energy value of the second target response map is greater than a preset threshold, the tracking device obtains a minimum energy value and an average energy value of the first target response map, and determines an average peak energy ratio β of the first target response map according to the maximum energy value, the minimum energy value and the average energy value of the first target response map v The average peak energy ratio β v may be expressed as:

[0042]

[0043] wherein R max，v , R min，v and R mean，v are the maximum energy value, the minimum energy value and the average energy value of the first target response map respectively.

[0044] The tracking device obtains a minimum energy value and an average energy value of the second target response map, and determines an average peak energy ratio β of the second target response map according to the maximum energy value, the minimum energy value and the average energy value of the second target response map i ; the average peak energy ratio β i may be expressed as:

[0045]

[0046] wherein R max，i , R min，i and R mean，i are the maximum energy value, the minimum energy value and the average energy value of the second target response map respectively.

[0047] Optionally, the preset threshold can be 0.1, 0.2, 0.3, 0.4, 0.5 or other values.

[0048] The tracking device determines the visible light confidence and the infrared light confidence according to the maximum energy value of the first target response map, the average peak energy ratio β of the first target response map v , and the maximum energy value of the second target response map, the average peak energy ratio β of the second target response map i .

[0049] The visible light confidence can be expressed as:

[0050]

[0051] The infrared light confidence can be expressed as: δ i = 1- δ v , wherein ω is a weight. Optionally, the ω can be 0.7.

[0052] S104, determining the position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence and the infrared light confidence.

[0053] In one possible implementation, the determining of the position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence and the infrared light confidence comprises:

[0054] The visible light confidence and the infrared light confidence are respectively taken as the weight of the first target response map and the weight of the second target response map for weighted summation to obtain a fusion response map; the position offset of the target object is determined according to the fusion response map, the position offset of the target object being the position where the maximum energy value of the fusion response map is located; and the position of the target object in the current frame is obtained according to the position of the target object in the previous frame image and the position offset of the target object.

[0055] Specifically, the tracking device takes the visible light confidence and the infrared light confidence as the weight of the first target response map and the weight of the second target response map respectively, and performs weighted summation on the first target response map and the second target response map to obtain a fusion response map. The fusion response map can be expressed as:

[0056] R t = δ v *R v，t + δ i *R i，t

[0057] After obtaining the fusion response map in the above manner, the tracking device determines the position offset of the target object according to the fusion response map; the position offset is the position coordinate where the maximum energy value of the fusion response map is located. Since the position offset of the target object refers to the offset of the position of the target object in the current frame relative to the position of the target object in the previous frame, after obtaining the position offset of the target object, the tracking device determines the position of the target object in the current frame according to the position offset of the target object and the position of the target object in the previous frame.

[0058] It is pointed out here that the coordinates of the target object in the visible light image and the infrared light image of the current frame are the same, and the tracking device can determine the position of the target object in the visible light image of the current frame based on the target offset and the position of the target object in the visible light image of the previous frame, which is also the position of the target object in the infrared light image of the current frame; if the coordinates of the target object in the visible light image and the infrared light image of the current frame are not the same, the tracking device can determine the position of the target object in the visible light image of the current frame based on the target offset and the position of the target object in the visible light image of the previous frame; the tracking device can determine the position of the target object in the infrared light image of the current frame based on the target offset and the position of the target object in the infrared light image of the previous frame.

[0059] After obtaining the position of the target object in the current frame according to the above method, the first position filter and the second position filter are updated according to the visible light image and the infrared light image of the current frame, respectively.

[0060] In a feasible embodiment, the tracking device obtains an infrared light image block and a visible light image block according to the visible light image and the infrared light image, respectively; the visible light image block is a part of the visible light image and contains the target object; the infrared light image block is a part of the infrared light image and contains the target object. The tracking device performs feature extraction on the visible light image block to obtain a first HOG feature matrix and a first CN feature matrix; a second HOG feature matrix is obtained according to the first HOG feature matrix; the dimension of the second HOG feature matrix is smaller than that of the first HOG feature matrix; a second CN feature matrix is obtained according to the first CN feature matrix; the dimension of the second CN feature matrix is smaller than that of the first CN feature matrix; the tracking device performs feature extraction on the infrared light image block to obtain a third HOG feature matrix; a fourth HOG feature matrix is obtained according to the third HOG feature matrix; the dimension of the fourth HOG feature matrix is smaller than that of the third HOG feature matrix; a grayscale feature matrix is obtained according to the infrared light image block; a visible light feature matrix is obtained according to the second HOG feature matrix and the second CN feature matrix; an infrared light feature matrix is obtained according to the fourth HOG feature matrix and the grayscale feature matrix.

[0061] Specifically, the tracking device performs feature extraction on the visible light image block to obtain a first HOG feature matrix and a first CN feature matrix, and the first HOG feature matrix can be a 32-dimensional feature matrix; the first 18 dimensions are selected from the 32-dimensional first HOG feature matrix to obtain a second HOG feature; the first CN feature matrix can be a 10-dimensional feature matrix; the first 2 dimensions are selected from the 10-dimensional feature matrix to obtain a second CN feature matrix.

[0062] The formula for singular value decomposition of the first HOG feature matrix is as follows:

[0063] U∑V=F h (x) T F h (x)

[0064] wherein U,∑ and V are left singular matrix, singular value matrix and right singular matrix obtained by singular value decomposition of the first HOG feature matrix respectively; F h (x) is the first HOG feature matrix, and x is the visible light image block. The second HOG feature matrix is determined according to the following formula:

[0065] F hog2 =C(F h (x)*L d (U))

[0066] L d (·) represents retaining the first d dimensions of the feature, and for the second HOG feature matrix, d=18. C(·) represents performing cosine windowing operation, which aims to reduce the boundary effect caused by Fourier transform.

[0067] It is pointed out here that the second CN feature matrix can be obtained based on the infrared light image block in the above manner; and the fourth HOG feature matrix can be obtained based on the infrared light image in the above manner.

[0068] The tracking device obtains the gray feature matrix according to the following formula:

[0069]

[0070] wherein F gray is the gray feature matrix of the infrared light image block, and F in is the gray pixel value matrix of the infrared light image block.

[0071] The tracking device merges the second HOG feature matrix and the second CN feature matrix to obtain a visible light feature matrix; and merges the fourth HOG feature matrix and the gray feature matrix to obtain an infrared light feature matrix. The visible light feature matrix and the infrared light feature matrix can be respectively represented as:

[0072] F fusion_v =Concat(F hog2 ,F CN )

[0073] F fusion_i =Concat(F hog4 ,F gray )

[0074] wherein F fusion_v is the visible light feature matrix, and F fusion_iis an infrared light feature matrix; f hog2 is a second HOG feature matrix, F hog4 is a fourth HOG feature matrix, F CN is a second CN feature matrix.

[0075] The tracking device updates the first position filter according to the following formula:

[0076]

[0077]

[0078] wherein η is a learning rate, is a kth dimension feature of a frequency domain signal corresponding to a visible light feature matrix, G t is a frequency domain signal corresponding to a two-dimensional Gaussian output label matrix of a target object.

[0079] Similarly, the second position filter can be updated according to the above formula.

[0080] The following specifically describes how to obtain the position filter.

[0081] Features of a training sample are obtained, which can be a visible light image or an infrared light image. A loss function is given as follows:

[0082]

[0083] wherein h is a filter, f is a feature of a training sample, g is a two-dimensional Gaussian output label matrix of the training sample, n identifies a change of a mode, when n is i, it indicates that the training sample is an infrared light image; when n is v, it indicates that the training sample is a visible light image. The right term is a regularization term, and λ mainly plays a control role to avoid overfitting when training the filter. The above formula is solved in the frequency domain to obtain a formula about the filter:

[0084]

[0085] The numerator and the denominator of the above formula respectively represent a cross-correlation of a label and a feature matrix of a training sample, and an autocorrelation of the feature matrix. The formula is split into the upper and lower parts of the numerator and the denominator, and a learning rate is introduced to obtain:

[0086]

[0087]

[0088] After obtaining the position of the target object in the first frame according to the above method, the visible light feature matrix corresponding to the visible light image in the first frame and the infrared light feature matrix corresponding to the infrared light image in the first frame are obtained according to the above method; the position filter is updated according to the visible light feature matrix and the infrared light feature matrix, and an updated position filter is obtained, which can be expressed as and When predicting the position of the target object in the second frame according to the above formula, the position filter is used and After obtaining the position of the target object in the second frame according to the above method, the visible light feature matrix corresponding to the visible light image in the second frame and the infrared light feature matrix corresponding to the infrared light image in the second frame are obtained according to the above method; the position filter is updated according to the visible light feature matrix and the infrared light feature matrix, and an updated position filter is obtained, which can be expressed as and During the whole process of tracking the target object, the position filter can be updated according to the above method.

[0089] In a feasible embodiment, the method of the present application further comprises:

[0090] The tracking device obtains the size of the bounding box in the previous frame, and the area occupied by the bounding box in the previous frame includes the target object; the target image is processed according to the position of the target object in the current frame and the size of the bounding box in the previous frame to obtain a plurality of image blocks, the target image being the visible light image and the infrared light image with the highest confidence; the plurality of image blocks are respectively subjected to feature extraction to obtain a plurality of feature matrices corresponding to the plurality of image blocks; a scale response map is obtained according to the plurality of feature matrices; and the size of the bounding box in the current frame is determined according to the maximum energy value of the scale response map.

[0091] Specifically, after obtaining the position of the target object in the visible light image and the infrared light image of the current frame, the tracking device acquires a target image from the visible light image and the infrared light image of the current frame according to the visible light confidence and the infrared light confidence, the target image being an image corresponding to the highest confidence in the visible light image and the infrared light image of the current frame; if the visible light confidence is higher than the infrared light confidence, the target image is the visible light image of the current frame; if the infrared light confidence is higher than the visible light confidence, the target image is the infrared light image of the current frame; the tracking device acquires the size of the bounding box of the target object in the previous frame; the size of the bounding box of the target object in the previous frame is enlarged 16 times and reduced 16 times respectively with a preset step to obtain 32 sizes of bounding boxes, and the size of the bounding box of the target object in the previous frame is added, so that there are 33 sizes of bounding boxes; the tracking device takes the position of the target object in the target object as the center, and acquires 33 image blocks from the target image according to the 33 sizes of bounding boxes; feature extraction is performed on the 33 image blocks respectively to obtain 33 feature matrices; the 33 feature matrices are respectively unfolded into one dimension to obtain 33 one-dimensional vectors, the length of each one-dimensional vector being n_length, and the 33 one-dimensional vectors are spliced together to obtain a two-dimensional feature matrix with 33 rows and n_length columns; a scale response map with 1*33 is obtained based on the two-dimensional feature matrix with 33 rows and n_length columns in the manner of calculating the first target response map or the second target response map; wherein the output label matrix used for calculation is a one-dimensional Gaussian output label matrix. The size of the bounding box of the target object in the current frame can be obtained from the position coordinates of the maximum energy value of the scale response map:

[0092] S = base_size * stride X-17

[0093] Wherein, S represents the size of the bounding box of the target object in the current frame, base_size represents the size of the bounding box of the target object in the previous frame, stride is a preset step, which can be 0.025, and X represents the position coordinates of the maximum energy value of the scale response map.

[0094] It should be pointed out that the above process is to obtain the size of the bounding box of the target object in the current frame based on the scale filter. The calculation formula of the scale filter is consistent with that of the position filter, and the training process is also consistent, which will not be described here.

[0095] In a feasible embodiment, the method of the present application further comprises:

[0096] When the maximum energy value of the first target response map and the maximum energy value of the second target response map are both less than the preset threshold, a plurality of historical positions of the target object are obtained; the plurality of historical positions are positions of the target object in a plurality of historical frames; and a position of the target object in the current frame is predicted according to the plurality of historical positions, so as to obtain the position of the target object in the current frame.

[0097] Specifically, when the maximum energy value of the first target response map and the maximum energy value of the second target response map are both less than the preset threshold, it indicates that the target object encounters occlusion or heat cross, or a similar interference object overlaps with the target object, and if the position of the target object in the current frame and the bounding box are determined according to the position filter and the scale filter, the result is inaccurate, so other ways are needed to determine the position of the target object in the current frame and the bounding box. At this time, the update of the position filter and the scale filter is stopped.

[0098] In one example, positions of the target object in a plurality of historical frames are obtained, and a position of the target object in a current frame is predicted according to the positions of the target object in the plurality of historical frames; for example, the positions of the target object in the plurality of historical frames are fitted in time to obtain a position function, and then the position of the target object in the current frame is determined by using a current time and the position function. It should be understood that the plurality of historical frames can be consecutive frames or non-consecutive frames, and can be consecutive frames adjacent to the current frame.

[0099] In another example, the tracking device can determine the position of the target object in the current frame by using a Kalman filter. Specifically, the tracking device obtains positions of the target object in the previous two frames of the current frame, determines a velocity component of the target object based on the positions of the target object in the previous two frames, the velocity component refers to a rate of change of the position of the target object in the image, predicts a position of the target object in the current frame based on the velocity component and the position of the target object in the previous frame, the predicted position is a state prediction value of the current frame; and the position of the target object in the current frame is determined according to the state prediction value of the current frame and a state measurement value of the current frame, the position of the target object in the current frame determined by the Kalman filter can be expressed as:

[0100]

[0101] wherein, is the state measurement value of the current frame, K t is the Kalman gain, Z t is the state measurement value of the current frame, and H is a linear sensor observation matrix. The state measurement value of the current frame is the position of the target object in the previous frame determined based on the Kalman filter. The Kalman gain can be expressed as:

[0102]

[0103]

[0104] where S k is the innovation component, is the predicted error covariance, and R is the innovation covariance. The updated covariance matrix is:

[0105]

[0106] Then the updated Kalman filter can be obtained based on the above formula.

[0107] It should be noted that when the position of the target object in the visible light image and the infrared light image of the current frame is inconsistent, the position of the target object in the visible light image and the infrared light image of the current frame can be determined based on the above manner respectively.

[0108] It should be understood that the scheme of the present application is also applicable to other filtering methods, such as minimum output sum of squared error filter (MOSSE), kernelized correlation filter (KCF), discriminative scale space tracing (DSST) algorithm. The feature extraction method is also limited to the above described manner, and also includes but is not limited to the extraction method of Local Binary Pattern (LBP) feature, Lab color feature and the like.

[0109] It can be seen that in the scheme of the present application, by calculating the confidence of different modalities, the negative influence of the modality with large tracking error on the tracking effect is adaptively reduced, thereby improving the success rate of tracking. At the same time, the Kalman filter is added to predict and estimate the position of the target object, and the position of the target object is determined by the Kalman filter in the case that the position filter is unreliable, thereby avoiding the error accumulation of the position filter, and improving the robustness of the algorithm. In addition, the scheme of the present application does not need to repeatedly calculate the past image frames, effectively reduces the calculation complexity, and the method based on the combination of correlation filtering and Kalman filtering has fast calculation speed and strong portability, and can save computing resources in the embedded platform with limited computing power, and has the characteristics of fast speed, good real-time performance, high efficiency and the like.

[0110] In a specific example, as Figure 2As shown, when starting to track the target object, visible light image block samples and infrared light image samples are respectively acquired; HOG features and CN features of the visible light image block samples are extracted to obtain a visible light feature matrix; a position filter and a size filter corresponding to the visible light are trained based on the visible light feature matrix; HOG features and grayscale features of the infrared light image block samples are extracted to obtain an infrared light feature matrix; a position filter and a size filter corresponding to the infrared light are trained based on the infrared light feature matrix; after reading in a frame of image, a first target response map is calculated according to the visible light image of the frame and the trained position filter corresponding to the visible light; a second target response map is calculated according to the infrared light image of the frame and the trained position filter corresponding to the infrared light; if the maximum energy value R max of any one of the first target response map and the second target response map is greater than 0.2, visible light confidence and infrared light confidence are calculated according to the first target response map and the second target response map, the position of the target object in the current frame is determined according to the first target response map, the second target response map, the visible light confidence and the infrared light confidence; the size of the bounding box is determined according to the image corresponding to the highest confidence in the visible light image and the infrared light image and the position of the target object in the current frame; and the Kalman filter is updated; if the maximum energy value R max of any one of the first target response map and the second target response map is greater than 0.2, the Kalman filter is started to predict the position of the target object in the infrared light image and the visible light image of the current frame; it is judged whether the current frame is the last frame, if the current frame is not the last frame, visible light images and infrared light images are respectively acquired from the visible light image and the infrared light image of the current frame; the position filter and the size filter corresponding to the visible light and the position filter and the size filter corresponding to the infrared light are updated in the above manner; after reading in the next frame of image, the position of the target object in the frame is determined in the above manner, and the Kalman filter is updated. The above steps are repeatedly executed until the tracking is ended.

[0111] Referring to Figure 3 , Figure 3 A structure schematic diagram of a tracking device provided by the embodiment of the present application is shown in the figure. Figure 3 As shown, the tracking device 300 comprises:

[0112] An acquisition unit 301 is configured to acquire a current frame of a target object, the current frame comprising a visible light image and an infrared light image;

[0113] The calculation unit 302 is used to obtain a first target response map based on a visible light image and a first position filter; and to obtain a second target response map based on an infrared light image and a second position filter; the first position filter is obtained based on a visible light image of a historical frame, and the second position filter is obtained based on an infrared light image of a historical frame; when the maximum energy value of either the first target response map or the second target response map is greater than a preset threshold, the visible light confidence level and the infrared light confidence level are obtained based on the first target response map and the second target response map.

[0114] The determining unit 302 is used to determine the position of the target object in the current frame based on the first target response map, the second target response map, the visible light confidence score, and the infrared light confidence score.

[0115] In one feasible embodiment, the determining unit 302 is specifically used for:

[0116] The visible light confidence and infrared light confidence are used as weights for the first target response map and the second target response map, respectively, and then weighted and summed to obtain the fused response map. The position offset of the target object is determined based on the fused response map, and the position offset of the target object is the location of the maximum energy value of the fused response map. The position of the target object in the current frame is obtained based on the position of the target object in the previous frame and the position offset of the target object.

[0117] In a feasible embodiment, in obtaining the visible light confidence score and the infrared light confidence score based on the first target response map and the second target response map, the calculation unit 302 is specifically used for:

[0118] The average energy ratio of the first target response map is obtained based on the maximum, minimum, and average energy values ​​of the first target response map; the average energy ratio of the second target response map is obtained based on the maximum, minimum, and average energy values ​​of the second target response map; and the visible light confidence score and infrared light confidence score are obtained based on the maximum energy value of the first target response map, the maximum energy value of the second target response map, the average energy ratio of the first target response map, and the average energy ratio of the second target response map.

[0119] In a feasible embodiment, the acquisition unit 301 is further configured to acquire the size of the bounding box in the previous frame, wherein the area occupied by the bounding box in the previous frame includes the target object;

[0120] The tracking device 300 further includes:

[0121] The processing unit 304 is used to process the target image according to the position of the target object in the current frame and the size of the bounding box in the previous frame to obtain multiple image blocks. The target image is the visible light image and the infrared light image, and the image corresponding to the highest confidence level.

[0122] The feature extraction unit 305 is configured to perform feature extraction on the plurality of image blocks respectively to obtain a plurality of feature matrices corresponding to the plurality of image blocks; and obtain a scale response map according to the plurality of feature matrices.

[0123] The determination unit 303 is further configured to determine the size of the bounding box in the current frame according to the maximum energy value of the scale response map.

[0124] In an embodiment, the acquisition unit 301 is further configured to acquire a plurality of historical positions when the maximum energy value of the first target response map and the maximum energy value of the second target response map are both less than a preset threshold; and the plurality of historical positions are positions of the target object in a plurality of historical frames.

[0125] The tracking device 300 further includes:

[0126] The prediction unit 306 is configured to predict the position of the target object in the current frame according to the plurality of historical positions to obtain the position of the target object in the current frame.

[0127] It should be noted that each unit (the acquisition unit 301, the calculation unit 302, the determination unit 303, the processing unit 304, the feature extraction unit 305 and the prediction unit 306) is configured to perform the related steps of the above method. For example, the acquisition unit 301 is configured to perform the related content of S101, the calculation unit 302 is configured to perform the related content of S102 and S103, and the determination unit 303, the processing unit 304, the feature extraction unit 305 and the prediction unit 306 are configured to perform the related content of S104.

[0128] In this embodiment, the tracking device 300 is presented in the form of units. The "unit" here can refer to an application-specific integrated circuit (ASIC), a processor and a memory executing one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the acquisition unit 301, the calculation unit 302, the determination unit 303, the processing unit 304, the feature extraction unit 305 and the prediction unit 306 can be implemented by the processor 401 of the tracking device shown in the figure. Figure 4

[0129] As shown in the tracking device 400, the tracking device 400 can be implemented in the structure shown in the figure, and the tracking device 400 includes at least one processor 401, at least one memory 402 and at least one communication interface 403. The processor 401, the memory 402 and the communication interface 403 are connected through the communication bus and complete the communication among each other. Figure 4 Figure 4 As shown in the tracking device 400, the tracking device 400 can be implemented in the structure shown in the figure, and the tracking device 400 includes at least one processor 401, at least one memory 402 and at least one communication interface 403. The processor 401, the memory 402 and the communication interface 403 are connected through the communication bus and complete the communication among each other.​​

[0130] The processor 401 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the above solutions.

[0131] The communication interface 403 is configured to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like.

[0132] The memory 402 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magneto-optical disk, a magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory can exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

[0133] The memory 402 is configured to store application program codes for executing the above solutions, and the processor 401 is configured to control the execution of the application program codes stored in the memory 402.

[0134] The codes stored in the memory 402 can execute any one of the multi-modal tracking methods provided above, such as:

[0135] Obtaining a current frame of a target object, the current frame comprising a visible light image and an infrared light image; obtaining a first target response map according to the visible light image and a first position filter; obtaining a second target response map according to the infrared light image and a second position filter; the first position filter is obtained based on a visible light image of a historical frame, and the second position filter is obtained based on an infrared light image of the historical frame; when a maximum energy value of any one of the first target response map and the second target response map is greater than a preset threshold, obtaining a visible light confidence and an infrared light confidence according to the first target response map and the second target response map; and determining a position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence, and the infrared light confidence.

[0136] The embodiments of the present application further provide a computer storage medium, wherein the computer storage medium can store a program, and the program performs, when executed, part or all steps of any one of the multi-modal tracking methods described in the method embodiments.

[0137] The embodiments of the present application further provide a computer program, which is executed to implement part or all steps of any one of the multi-modal tracking methods described in the method embodiments.

[0138] It should be noted that, for the foregoing method embodiments, in order to simply describe, each of the method embodiments is described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0139] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0140] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical or other forms.

[0141] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed on multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0142] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0143] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable memory. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0144] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable memory, which can include: a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0145] The embodiments of the present application are described in detail above, and the principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A multi-modal object tracking method, characterized by, The method comprises: obtaining a current frame of a target object, the current frame comprising a visible light image and an infrared light image; obtaining the size of a bounding box in a previous frame, the area occupied by the bounding box in the previous frame comprising the target object; processing a target image according to the position of the target object in the current frame and the size of the bounding box in the previous frame to obtain a plurality of image blocks, the target image being the visible light image and the infrared light image, the image corresponding to the highest confidence; performing feature extraction on the plurality of image blocks respectively to obtain a plurality of feature matrices corresponding to the plurality of image blocks; obtaining a scale response map according to the plurality of feature matrices; determining the size of the bounding box in the current frame according to the maximum energy value of the scale response map; obtaining a first target response map according to the visible light image and a first position filter; obtaining a second target response map according to the infrared light image and a second position filter; the first position filter is obtained based on the visible light image of a historical frame, and the second position filter is obtained based on the infrared light image of the historical frame; when the maximum energy value of any one of the first target response map and the second target response map is greater than a preset threshold, obtaining a visible light confidence and an infrared light confidence according to the first target response map and the second target response map; determining the position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence and the infrared light confidence.

2. The method of claim 1, wherein, The method comprises: weighting and summing the visible light confidence and the infrared light confidence as the weight of the first target response map and the weight of the second target response map respectively to obtain a fusion response map; determining the position offset of the target object according to the fusion response map, the position offset of the target object being the position where the maximum energy value of the fusion response map is located; obtaining the position of the target object in the current frame according to the position of the target object in the previous frame image and the position offset of the target object.

3. The method according to claim 1 or 2, characterized in that, The method comprises: obtaining the average energy ratio of the first target response map according to the maximum energy value, the minimum energy value and the average energy value of the first target response map; obtaining the average energy ratio of the second target response map according to the maximum energy value, the minimum energy value and the average energy value of the second target response map; obtaining the visible light confidence and the infrared light confidence according to the maximum energy value of the first target response map, the maximum energy value of the second target response map, the average energy ratio of the first target response map and the average energy ratio of the second target response map.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: When the maximum energy value of the first target response map and the maximum energy value of the second target response map are both less than a preset threshold, a plurality of historical positions are obtained; the plurality of historical positions are positions of the target object in a plurality of historical frames; A position of the target object in a current frame is predicted according to the plurality of historical positions, to obtain the position of the target object in the current frame.

5. A tracking device, characterized by Comprise: An acquisition unit is configured to acquire a current frame of a target object, the current frame comprising a visible light image and an infrared light image; The acquisition unit is further configured to acquire a size of a bounding box in a previous frame, the bounding box occupying an area comprising the target object in the previous frame; The tracking device further comprises a processing unit configured to process a target image according to the position of the target object in the current frame and the size of the bounding box in the previous frame, to obtain a plurality of image blocks, the target image being the visible light image and the infrared light image, and an image corresponding to a highest confidence value; A feature extraction unit is configured to perform feature extraction on the plurality of image blocks respectively, to obtain a plurality of feature matrices corresponding to the plurality of image blocks; and obtain a scale response map according to the plurality of feature matrices; A determination unit is further configured to determine the size of the bounding box in the current frame according to a maximum energy value of the scale response map; A calculation unit is configured to obtain a first target response map according to the visible light image and a first position filter, and obtain a second target response map according to the infrared light image and a second position filter; the first position filter is obtained based on a visible light image of a historical frame, and the second position filter is obtained based on an infrared light image of the historical frame; when a maximum energy value of any one of the first target response map and the second target response map is greater than a preset threshold, obtain a visible light confidence value and an infrared light confidence value according to the first target response map and the second target response map; A determination unit is configured to determine the position of the target object in the current frame according to the first target response map, the second target response map, the visible light confidence value, and the infrared light confidence value.

6. The tracking device of claim 5, wherein, The determination unit is specifically configured to: weight and sum the visible light confidence value and the infrared light confidence value as weights of the first target response map and the second target response map respectively, to obtain a fusion response map; determine a position offset of the target object according to the fusion response map, the position offset of the target object being a position where a maximum energy value of the fusion response map is located; obtain the position of the target object in the current frame according to a position of the target object in a previous frame image and the position offset of the target object.

7. The tracking device of claim 5 or 6, wherein, In the aspect of obtaining a visible light confidence value and an infrared light confidence value according to the first target response map and the second target response map, the calculation unit is specifically configured to: obtain an average energy ratio of the first target response map according to a maximum energy value, a minimum energy value, and an average energy value of the first target response map; obtain an average energy ratio of the second target response map according to a maximum energy value, a minimum energy value, and an average energy value of the second target response map; The visible light confidence and the infrared light confidence are obtained according to the maximum energy value of the first target response map, the maximum energy value of the second target response map, the average energy ratio of the first target response map, and the average energy ratio of the second target response map. 8.The tracking apparatus of any one of claims 5-7, characterized in that, The acquisition unit is further configured to acquire a plurality of historical positions when the maximum energy value of the first target response map and the maximum energy value of the second target response map are both less than a preset threshold. The plurality of historical positions are positions of the target object in a plurality of historical frames. The tracking apparatus further comprises: A prediction unit configured to predict a position of the target object in a current frame according to the plurality of historical positions to obtain the position of the target object in the current frame.

Images