Object tracking method and system, electronic equipment, storage medium and program product

Abstract

The invention provides an object tracking method and system, electronic equipment, a storage medium and a program product. A specific implementation mode of the method comprises the following steps: acquiring overall state vector information of a plurality of to-be-tracked objects acquired by an MIMO-UWB radar; the overall state vector information comprises speed information, distance information and angle information of the plurality of to-be-tracked objects; separating the plurality of to-be-tracked objects into a plurality of independent to-be-tracked objects according to the overall state vector information; according to the state vector information corresponding to each to-be-tracked object, respectively determining the position information of each to-be-tracked object at the current moment; sending a tracking instruction to the camera; wherein the tracking instruction is used for indicating the camera to track according to the position information and the image information of each to-be-tracked object at the current moment. According to the method, a plurality of objects can be tracked more accurately and clearly.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar, in particular to an object tracking method and system, an electronic device, a storage medium and a program product. BACKGROUND

[0002] MIMO-UWB (Multiple-Input Multiple-Output Ultra-WideBand Radar, Ultra-Wideband Multiple-Input Multiple-Output, MIMO-UWB for short) radar is a radar that uses nanosecond pulse signals with an extremely wide frequency band for detection. It can obtain state information such as distance, speed, and angle of an object by measuring pulse flight time, Doppler shift, and reflection intensity.

[0003] In some application scenarios, there are often multiple objects, and the state information of these objects can change at any time. Therefore, in order to continuously grasp the information (such as position information) of each object, multiple objects can be tracked.

[0004] However, in the related art, multiple objects cannot be tracked accurately and clearly. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide an object tracking method and system, an electronic device, a storage medium and a program product, which can track multiple objects accurately and clearly.

[0006] In a first aspect, the embodiments of the present application provide an object tracking method, which comprises: obtaining overall state vector information of multiple to-be-tracked objects collected by a MIMO-UWB radar; the overall state vector information comprises speed information, distance information, and angle information of the multiple to-be-tracked objects; separating the multiple to-be-tracked objects into multiple independent to-be-tracked objects according to the overall state vector information; determining position information of each to-be-tracked object at a current time according to the state vector information corresponding to each to-be-tracked object; and sending a tracking instruction to a camera; wherein the tracking instruction is used to instruct the camera to track according to the position information of each to-be-tracked object at the current time and image information. In this way, after the overall state vector information is obtained, the multiple to-be-tracked objects are separated, which facilitates the binding of the to-be-tracked objects and the corresponding image information, improves the interference between the information, and thus accurate tracking of a single to-be-tracked object can be achieved. In addition, tracking in combination with the position information at the same time and the image information is essentially joint tracking that incorporates the spatial features and visual features of the to-be-tracked objects, which can track each to-be-tracked object clearly. Therefore, the present implementation can track multiple objects accurately and clearly.

[0007] Optionally, after separating the plurality of to-be-tracked objects into a plurality of independent to-be-tracked objects according to the overall state vector information, the method further comprises: determining a tracking priority of each to-be-tracked object according to a preset rule; and the tracking instruction is used to instruct the camera to track according to the tracking priority of each to-be-tracked object. In this way, objects with higher importance can be tracked.

[0008] Optionally, the determining of the tracking priority of each to-be-tracked object according to the preset rule comprises: for each to-be-tracked object, determining a distance weight corresponding to the to-be-tracked object according to distance information between the to-be-tracked object and the MIMO-UWB radar; acquiring radar cross-section information collected by the MIMO-UWB radar, and determining a human-like feature weight of the to-be-tracked object according to the radar cross-section information; determining a motion trend weight of the to-be-tracked object according to multi-frame image information collected by the camera; and determining the tracking priority corresponding to the to-be-tracked object according to one or more of the human-like feature weight, the motion trend weight, and the distance weight. In this way, the tracking priority corresponding to the to-be-tracked object can be determined from multiple aspects such as distance, motion, and whether it is a human object, by combining one or more of the human-like feature weight, the motion trend weight, and the distance weight, thereby improving the tracking attention to important objects to some extent, improving the false tracking and wrong tracking of important objects, and improving the tracking accuracy of important objects to some extent.

[0009] Optionally, the determining of the tracking priority of each to-be-tracked object according to the preset rule comprises: for each to-be-tracked object, determining a distance weight corresponding to the to-be-tracked object according to distance information between the to-be-tracked object and the MIMO-UWB radar; and determining the tracking priority of each to-be-tracked object according to the number of tracking requirements and the distance weight corresponding to each to-be-tracked object. In this way, the waste of computing power and the loss of important objects can be improved to some extent, the tracking accuracy of important objects is improved, and flexible tracking with adaptive computing power and tracking accuracy is realized.

[0010] Optionally, after sending the tracking instruction to the camera, the method further comprises: for each to-be-tracked object, updating the position information of the to-be-tracked object in real time based on the state vector information corresponding to the to-be-tracked object according to a Kalman filtering algorithm; and sending an updated tracking instruction to the camera. In this way, the position information of the to-be-tracked object can be updated in real time, thereby assisting the camera in updating each to-be-tracked object in real time, which facilitates obtaining the tracking trajectory of the to-be-tracked object and improves the robustness during tracking.

[0011] Optionally, after separating the plurality of objects to be tracked into a plurality of independent objects to be tracked according to the overall state vector information, the method further comprises: determining a sampling rate of the MIMO-UWB radar and / or an image acquisition frame rate of the camera according to the speed of each object to be tracked respectively; wherein the sampling rate is positively correlated with the speed, and the image acquisition frame rate is positively correlated with the speed. In this way, the sampling rate of the MIMO-UWB radar and the image acquisition frame rate of the camera can be dynamically adjusted based on the speed of the object to be tracked, so as to reduce power consumption on the basis of realizing multi-object tracking.

[0012] Optionally, after determining the position information of each object to be tracked at the current time according to the state vector information of each object to be tracked respectively, the method further comprises: for each object to be tracked, predicting the position information of the object to be tracked at a target time according to the state vector information of the object to be tracked at the current time; the target time is any time after the current time; and sending a pre-tracking instruction to the camera, the pre-tracking instruction being used to instruct the camera to adjust the tracking parameter in advance according to the position information of the object to be tracked at the target time. In this way, the position information of the object to be tracked at the target time can be predicted, so that the camera can be instructed to adjust the tracking parameter in advance, so that the camera can enter the tracking action at the target time immediately after completing the tracking action at the previous time, thereby reducing the tracking delay and improving the accuracy and clarity of tracking to a certain extent.

[0013] Optionally, before sending the tracking instruction to the camera, the method further comprises: acquiring the signal-to-noise ratio corresponding to each object to be tracked collected by the MIMO-UWB radar; for each object to be tracked, determining the confidence degree of the object to be tracked in a motion state according to the signal-to-noise ratio and the speed information corresponding to the object to be tracked; and the sending of the tracking instruction to the camera comprises: if there is a confidence degree not less than a preset confidence threshold, sending the tracking instruction to the camera. Here, considering that the objects to be tracked are mostly moving objects, the confidence degree of the object to be tracked in a motion state can be determined, and whether to send the tracking instruction to the camera can be determined based on the confidence degree, so as to improve the situation that the camera is frequently instructed to frequently start the focusing motor.

[0014] In a second aspect, an object tracking device is provided. The device comprises an obtaining module, a separating module, a determining module, and a tracking module. The obtaining module is configured to obtain overall state vector information of a plurality of objects to be tracked collected by a MIMO-UWB radar. The overall state vector information comprises velocity information, distance information, and angle information of the plurality of objects to be tracked. The separating module is configured to separate the plurality of objects to be tracked into a plurality of independent objects to be tracked according to the overall state vector information. The determining module is configured to determine position information of each object to be tracked at a current time according to state vector information corresponding to each object to be tracked. The tracking module is configured to send a tracking instruction to a camera. The tracking instruction is configured to instruct the camera to track according to the position information of each object to be tracked at the current time and image information.

[0015] In a third aspect, an object tracking system is provided. The system comprises a MIMO-UWB radar, a processor, and a camera. The MIMO-UWB radar is configured to obtain overall state vector information of a plurality of objects to be tracked. The overall state vector information comprises velocity information, distance information, and angle information of the plurality of objects to be tracked. The processor is configured to obtain overall state vector information of a plurality of objects to be tracked collected by the MIMO-UWB radar, separate the plurality of objects to be tracked into a plurality of independent objects to be tracked according to the overall state vector information, determine position information of each object to be tracked at a current time according to state vector information corresponding to each object to be tracked, and send a tracking instruction to the camera. The camera is configured to track according to the position information of each object to be tracked at the current time and image information in the tracking instruction.

[0016] In a fourth aspect, an electronic device is provided. The electronic device comprises a processor and a memory. The memory stores computer readable instructions. When the computer readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.

[0017] In a fifth aspect, a computer readable storage medium is provided. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in the method provided in the first aspect are performed.

[0018] In a sixth aspect, a computer program product is provided. The computer program product comprises a computer program or instructions. When the computer program or instructions are executed by a processor, the method provided in the first aspect is performed.

[0019] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 A flow chart of an object tracking method provided by the embodiments of the present application; Figure 2 A structural block diagram of an object tracking device provided by the embodiments of the present application; Figure 3 A structural block diagram of an object tracking system provided by the embodiments of the present application; Figure 4 A structural schematic diagram of an electronic device for executing an object tracking method provided by the embodiments of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0023] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0024] It should be noted that: in the case of no conflict, the embodiments in the present application or the technical features in the embodiments can be combined.

[0025] In the related art, there is a problem that multiple objects cannot be tracked more accurately and clearly. For example, in a scene including multiple objects, a radar usually collects overall state vector information of multiple objects mixed together, and the information may interfere with each other; then, multiple objects cannot be tracked more accurately and clearly.

[0026] To solve this problem, the present application provides an object tracking method, system, electronic device, storage medium and program product; further, the present application obtains overall state vector information of multiple objects to be tracked collected by a MIMO-UWB radar, then separates each object to be tracked according to the overall state vector information, and then each object to be tracked can be tracked according to the corresponding state vector information. In this way, multiple objects can be tracked accurately and clearly. For example, the interference between signals can be improved to track multiple objects accurately and clearly.

[0027] It should be noted that the defects of the above-mentioned solutions in the related art are obtained by the inventors after long-term practice and careful study, therefore, the discovery process of the above-mentioned problems and the solutions proposed by the embodiments of the present application to solve the above-mentioned problems should be the contribution of the inventors to the present application in the process of the present application.

[0028] In some application scenarios, the above-mentioned object tracking method can be executed by a processor. The processor can be communicatively connected with a radar and a camera, so that information can be transmitted between the three. In addition, the processor can be disposed on the MIMO-UWB radar, or can not have a mechanical connection relationship with the MIMO-UWB radar, which is not limited by the present application.

[0029] Please refer to Figure 1 which shows a flowchart of an object tracking method provided by an embodiment of the present application. As shown in Figure 1 the object tracking method can include the following steps 101 to 104.

[0030] Step 101, obtaining overall state vector information of multiple objects to be tracked collected by a MIMO-UWB radar; the overall state vector information includes speed information, distance information and angle information of the multiple objects to be tracked; The above-mentioned objects to be tracked can include persons, animals and / or other objects.

[0031] The above-mentioned MIMO-UWB radar can continuously output high-resolution and strong-penetration echo information under any lighting conditions due to its own architecture of being able to actively emit ultra-wideband electromagnetic waves and multiple antenna arrays, so that more accurate overall state vector information can be collected under strong light and dark light.

[0032] The overall state vector information represents a mixed observation information set obtained after detecting the echoes of the multiple to-be-tracked objects mixed together collected by the MIMO-UWB radar, and contains observation components of distances, speeds, angles, etc. of the multiple to-be-tracked objects relative to the MIMO-UWB radar.

[0033] In some application scenarios, the MIMO-UWB radar can obtain the overall state vector information of the multiple to-be-tracked objects existing in the current scene, and then transmit the overall state vector information to the processor, so that the processor also obtains the overall state vector information.

[0034] In step 102, the multiple to-be-tracked objects are separated into multiple independent to-be-tracked objects according to the overall state vector information. In some application scenarios, the overall state vector information can be clustered by using a DBSCAN-RF algorithm, a k-means++ algorithm, etc. to separate the multiple to-be-tracked objects into single independent to-be-tracked objects.

[0035] It should be noted that the DBSCAN-RF algorithm, the k-means++ algorithm, etc. are existing algorithms, and a person skilled in the art can independently use these existing algorithms to separate the to-be-tracked objects after learning that the overall state vector information can be processed by using a clustering algorithm to separate the to-be-tracked objects.

[0036] In step 103, the position information of each to-be-tracked object at the current time is determined according to the state vector information corresponding to each to-be-tracked object. In some application scenarios, the position information corresponding to each to-be-tracked object can be obtained according to the distance information and the angle information corresponding to each to-be-tracked object. Specifically, for each to-be-tracked object, the position information of the to-be-tracked object can be obtained according to the distance information and the angle information corresponding to the to-be-tracked object. For example, the distance information and the angle information corresponding to each to-be-tracked object can be used to obtain the position coordinates of the to-be-tracked object in the MIMO-UWB radar coordinate system according to a preset conversion formula. For example, the distance information and the angle information corresponding to each to-be-tracked object can be used to obtain the position coordinates of the to-be-tracked object in the MIMO-UWB radar coordinate system according to a preset conversion formula. The MIMO-UWB radar coordinate system can have the position of the MIMO-UWB radar as the origin of the coordinate system, and the normal direction of the antenna array of the MIMO-UWB radar as the x-axis, and the vertical direction of the x-axis as the y-axis. The preset conversion formula can be, for example, ; wherein, the z-axis can point to the positive direction of the x-axis.

[0037] ​​​In some application scenarios, the position information of each to-be-tracked object at the current moment can also be determined by using a neural network model. Specifically, the state vector information of each to-be-tracked object at the current moment can be taken as the input of the neural network model, so as to obtain the position information of each to-be-tracked object at the current moment. In these application scenarios, the neural network model can be obtained by training a large amount of historical data in advance. Specifically, for example, the state vector information of each historical object at a certain moment can be input into an initial neural network model, and the real position information at the same moment can be taken as the expected output to train the neural network model, and then when the neural network model converges, it can be used to determine the position information of each to-be-tracked object.

[0038] In step 104, a tracking instruction is sent to the camera. The tracking instruction is used to instruct the camera to track according to the position information of each to-be-tracked object at the current moment and the image information.

[0039] It can be understood that the camera can capture the image information of the object. Therefore, the processor can instruct the camera to track each to-be-tracked object in combination with the position information of each to-be-tracked object at the current moment and the image information at the current moment.

[0040] In some application scenarios, the processor can first convert the position information of the to-be-tracked object from the MIMO-UWB radar coordinate system to the camera coordinate system by using a predetermined conversion matrix, and then instruct the camera to track according to the position information of the to-be-tracked object in the camera coordinate system.

[0041] In these application scenarios, for example, the image of a sample object can be captured in advance, and then the feature points of the sample object can be extracted, and the position information of the same sample object captured by the MIMO-UWB radar can be obtained synchronously, and the conversion matrix between the MIMO-UWB radar and the camera can be determined by using the ICP (Iterative Closest Point) algorithm. Thus, when it is necessary to track the to-be-tracked object, the coordinate conversion between the MIMO-UWB radar and the camera can be realized, so as to facilitate the tracking of the camera.

[0042] In the implementation manner, after obtaining the overall state vector information, the plurality of to-be-tracked objects are separated, so that the to-be-tracked objects and their corresponding image information can be bound, the interference between the information is improved, and thus the accurate tracking of the single to-be-tracked object can be realized. In addition, the tracking in combination with the position information and the image information at the same moment is actually a joint tracking by integrating the spatial features and the visual features of the to-be-tracked object, so that each to-be-tracked object can be tracked more clearly. Therefore, the plurality of objects can be tracked more accurately and clearly in the implementation manner.

[0043] In addition, it is considered that the importance of the plurality of to-be-tracked objects is different in different scenes or different requirements. For example, in a scene including a human object and an animal object, the importance of the human object is generally higher than that of the animal object. Or, in a scene including a plurality of human objects, the importance of a human object close to the camera is generally higher than that of a human object far away from the camera. Therefore, the tracking priority of each to-be-tracked object can be determined first, so as to track the object with higher importance.

[0044] Then, in some optional implementation, after separating the plurality of to-be-tracked objects into a plurality of independent to-be-tracked objects according to the overall state vector information in the step 102, the processor can further determine the tracking priority of each to-be-tracked object according to a preset rule. In this way, the tracking instruction in the step 104 can be used to instruct the camera to track according to the tracking priority of each to-be-tracked object.

[0045] In some optional implementation, the processor can perform the following steps to determine the tracking priority of each to-be-tracked object: Step A1, for each to-be-tracked object, determining a distance weight corresponding to the to-be-tracked object according to distance information between the to-be-tracked object and the MIMO-UWB radar.

[0046] For example, the processor can set a relatively high distance weight for the to-be-tracked object with distance information less than a preset distance threshold, and set a relatively low distance weight for the to-be-tracked object with distance information greater than the preset distance threshold. Further, for example, a plurality of preset distance thresholds can be set, each preset distance threshold can correspond to a distance weight, so that the corresponding distance weight can be adaptively determined according to the preset distance threshold corresponding to the distance information of the to-be-tracked object.

[0047] Step A2, obtaining radar cross section information collected by the MIMO-UWB radar, and determining a human-like feature weight of the to-be-tracked object according to the radar cross section information; That is, each to-be-tracked object has a corresponding MIMO-UWB radar cross section, so that the processor can obtain the MIMO-UWB radar cross section information corresponding to each to-be-tracked object respectively, so as to determine the human-like feature weight of the to-be-tracked object. The human-like feature weight can be used to represent the probability that the to-be-tracked object is a human object.

[0048] In some application scenarios, the human-like feature weight can be determined, for example, by means of a pre-trained model (e.g., random forest, support vector machine, etc.). Specifically, for example, the MIMO-UWB radar cross section information can be input into the pre-trained model, and then the human-like feature weight can be determined by the model.

[0049] In some application scenarios, for example, the MIMO-UWB radar cross section information of a plurality of sample objects can be collected, and the true labels of the sample objects are "belonging to a human object" or "not belonging to a human object" when the MIMO-UWB radar cross section information is obtained. Thus, the MIMO-UWB radar cross section information of the plurality of sample objects can be input into the model respectively, and the corresponding true labels can be output as the output of the model, so as to train the model to output the probability that the sample object "belongs to a human object". The probability output by the model is determined as the human-like feature weight.

[0050] Further, in order to improve the prediction accuracy of the model, the speed information, the wideband information of the micro-Doppler spectrum, the length-width information of the camera focus position, and the like information obtained when the MIMO-UWB radar cross section information is obtained can also be acquired, and these information can be input into the model for training the model. In this way, when the model is actually used, in addition to acquiring the MIMO-UWB radar cross section information of the to-be-tracked object, the above information of the to-be-tracked object can also be acquired, so as to improve the accuracy of the human-like feature weight of the to-be-tracked object.

[0051] In step A3, the processor can determine the motion trend weight of the to-be-tracked object according to the plurality of frames of image information collected by the camera. The motion trend weight can be used to represent the probability of the motion of the to-be-tracked object.

[0052] Here, for example, the adjacent multiple frames of images of the same to-be-tracked object can be acquired from the camera, and then the pixel coordinates of the same feature point of the to-be-tracked object in the multiple frames of images can be determined. Then, the change amount of the pixel coordinates can be determined in combination with the interval time length between the frames, and then the change amount can be determined as the data representing the motion trend.

[0053] If the change amount is large, it can be considered that the to-be-tracked object has a high motion trend, and thus a relatively large motion trend weight can be assigned to it. If the change amount is small, it can be considered that the to-be-tracked object has a low motion trend, and thus a relatively small motion trend weight can be assigned to it.

[0054] In some application scenarios, for example, a plurality of preset change amount thresholds can also be set, and each preset change amount threshold can correspond to a motion trend weight. In this way, the corresponding motion trend weight can be adaptively determined according to the preset change amount threshold corresponding to the change amount of the to-be-tracked object.

[0055] Step A4, determining the tracking priority of the to-be-tracked object according to one or more of the human-like feature weight, the motion trend weight, and the distance weight.

[0056] In some application scenarios, the tracking priority can be determined according to the cumulative sum of the three, for example. Specifically, the human-like feature weight, the motion trend weight, and the distance weight corresponding to each to-be-tracked object can be added respectively, and then the cumulative sum is arranged in descending order, and then the tracking priority can be obtained according to the arrangement position corresponding to each to-be-tracked object.

[0057] In other application scenarios, the cumulative weight can also be set according to the importance or actual demand for the human-like feature weight, the motion trend weight, and the distance weight. For example, the cumulative weight corresponding to the human-like feature weight can be set to 0.2, the cumulative weight corresponding to the motion trend weight can be set to 0.3, the cumulative weight corresponding to the distance weight can be set to 0.4, and the remaining cumulative weight 0.1 can be set to the cumulative weight corresponding to other weights. The other weights here can include, for example, the scene weight corresponding to the scene. For example, if the processor determines that the current scene is a motion scene (for example, a multi-person race scene) through image information, the scene weight can be set to a relatively high 0.9, and if it is determined through image information that the current scene is a static scene (for example, a multi-person sitting scene), the scene weight can be set to a relatively low 0.1.

[0058] In the present implementation, the tracking priority of the to-be-tracked object can be determined in combination with one or more of the human-like feature weight, the motion trend weight, and the distance weight of the to-be-tracked object, from the aspects of distance, motion, and whether it is a human object, etc. In a certain extent, the tracking attention to important objects is improved, the false tracking and wrong tracking of important objects are improved, and the tracking accuracy of important objects is improved to a certain extent.

[0059] In other optional implementations, the processor can also perform the following steps to determine the tracking priority of each to-be-tracked object: Step B1, for each to-be-tracked object, determining the distance weight corresponding to the to-be-tracked object according to the distance information between the to-be-tracked object and the MIMO-UWB radar; It can be understood that the process of determining the distance weight of the to-be-tracked object in this step can be the same as or similar to the implementation process of step A1 in the foregoing, which will not be described here.

[0060] Step B2, determining the tracking priority of each to-be-tracked object according to the number of tracking requirements and the distance weight corresponding to each to-be-tracked object respectively.

[0061] The tracking demand quantity can be input by an operator, preset according to experience, or set according to the performance of the MIMO-UWB radar and / or the camera, and is not limited herein.

[0062] In the implementation manner, the tracking priorities of the to-be-tracked objects can be adaptively determined according to the tracking demand quantity and the distance weight of the to-be-tracked objects, so that the distance weight can be dynamically adjusted, for example, a distant object is preferentially tracked when the objects are sparse, and the attention range can be adaptively narrowed when the objects are dense. Therefore, the implementation manner can improve the waste of computing power and the loss of important objects to some extent, improve the tracking accuracy of important objects, and then realize flexible tracking with adaptive computing power and tracking accuracy.

[0063] In some application scenarios, after the processor separates the to-be-tracked objects, the processor can assign a unique identity corresponding to each to-be-tracked object, so as to uniquely identify each to-be-tracked object. In this way, after obtaining the tracking priorities of the to-be-tracked objects, the tracking priorities of the to-be-tracked objects can be bound to the corresponding identities, so as to improve the tracking accuracy to some extent.

[0064] In some optional implementation manners, after the processor sends the tracking instruction to the camera in step 104, the processor can also update the position information of each to-be-tracked object in real time based on the state vector information corresponding to the to-be-tracked object according to a Kalman filtering algorithm. It should be noted that the Kalman filtering algorithm is mainly used to estimate the state of a dynamic system. It uses a set of prediction equations to describe the evolution of the internal state of the system over time to obtain a predicted value, and uses a set of observation equations to represent how to obtain an observation value from the true state. Then, the Kalman gain is updated according to the prediction noise corresponding to the predicted value and the observation noise corresponding to the observation value, so as to obtain a more accurate true value by weighting the observation value and the predicted value through the Kalman gain.

[0065] In the implementation manner, the Kalman filtering algorithm can be used to weight the observation value and the predicted value collected by the MIMO-UWB radar through the Kalman gain, so as to update the position information of the to-be-tracked object in real time. Here, the Kalman filtering algorithm is a prior art, and those skilled in the art can update the position information by using the Kalman filtering algorithm after learning the relevant information disclosed in the present application.

[0066] In this way, the processor can send the updated tracking instruction to the camera, so that the camera can track the to-be-tracked object in real time according to the updated position information and the image information at the corresponding moment.

[0067] In the implementation, the processor can update the position information of the to-be-tracked objects in real time, so as to assist the camera in updating the to-be-tracked objects in real time, and facilitate obtaining the tracking trajectory of the to-be-tracked objects and improving the robustness during tracking.

[0068] In some optional implementations, after separating the plurality of to-be-tracked objects into a plurality of independent to-be-tracked objects according to the overall state vector information in the step 102, the processor can further determine the sampling rate of the MIMO-UWB radar and / or the image acquisition frame rate of the camera according to the speed of each to-be-tracked object corresponding thereto; the sampling rate is positively correlated with the speed, and the image acquisition frame rate is positively correlated with the speed.

[0069] The speed of the to-be-tracked object can be determined according to the speed information thereof.

[0070] The sampling rate can be regarded as the frequency of the MIMO-UWB radar in collecting the overall state vector information.

[0071] The image acquisition frame rate can be regarded as the frequency of the camera in collecting the image of the to-be-tracked object.

[0072] In some application scenarios, if the speed of the to-be-tracked object is relatively fast, the MIMO-UWB radar can be instructed to set a relatively high sampling rate, and the camera can be instructed to set a relatively high image acquisition frame rate. If the speed of the to-be-tracked object is relatively slow, the MIMO-UWB radar can be instructed to set a relatively low sampling rate, and the camera can be instructed to set a relatively low image acquisition frame rate.

[0073] In these application scenarios, generally, one MIMO-UWB radar and one camera are set, so the sampling rate of the MIMO-UWB radar and the image acquisition frame rate of the camera can be determined based on the speed (for example, the average value or the speed median) of all to-be-tracked objects, or the sampling rate and the image acquisition frame rate can be determined based on the highest speed, which is not limited here.

[0074] Further, for example, the corresponding relationship between the sampling rate of the MIMO-UWB radar and the speed can be explored in advance, and then after the speed of the to-be-tracked object is determined, the corresponding sampling rate can be determined based on the corresponding relationship. Here, for example, a plurality of sets of historical data of the MIMO-UWB radar can be acquired, and the corresponding relationship can be detected through the historical data. Specifically, a plurality of historical speeds collected by the MIMO-UWB radar and the sampling rate when the complete speed information of the historical object can be accurately collected under each historical speed can be acquired, and then the corresponding relationship between the speed and the sampling rate can be fitted.

[0075] Similarly, for example, the correspondence between the image acquisition frame rate of the camera and the speed of the object can also be explored in advance, and then after the speed of the to-be-tracked object is determined, the corresponding image acquisition frame rate can be determined based on the correspondence. Here, for example, a plurality of sets of historical data can be obtained, and the correspondence can be detected through the historical data. Specifically, a plurality of historical speeds collected by the MIMO-UWB radar can be obtained, and the image acquisition frame rate at which the complete image information of the historical object can be accurately collected at each historical speed can be obtained, and then the correspondence between the speed and the image acquisition frame rate can be fitted.

[0076] In the implementation mode, the sampling rate of the MIMO-UWB radar and the image acquisition frame rate of the camera can be dynamically adjusted based on the speed of the to-be-tracked object, so as to reduce the power consumption on the basis of realizing multi-object tracking.

[0077] In some application scenarios, for example, the speed of the to-be-tracked object can also be used to determine whether it is in a motion state or a stationary state. If it is in a motion state, the camera can be instructed to perform continuous phase detection auto focus (PDAF) to continuously focus on the moving to-be-tracked object. If it is in a stationary state, the camera can be instructed to perform single contrast detection auto focus (CDAF) to focus on the stationary to-be-tracked object once. In this way, adaptive focusing can be performed according to the state of the to-be-tracked object, thereby further reducing power consumption.

[0078] In some optional implementation modes, after the position information of each to-be-tracked object at the current time is determined according to the state vector information corresponding to each to-be-tracked object in step 103, the processor can further predict the position information of each to-be-tracked object at a target time according to the state vector information of the to-be-tracked object at the current time; In some application scenarios, the processor can predict the position information of the to-be-tracked object at the target time through the speed information, distance information, angle information, and angular velocity information of the to-be-tracked object at the current time. Specifically, for example, the time interval between the current time and the target time can be determined, and then it is assumed that the to-be-tracked object moves at a constant speed. The product of the interval length and the speed of the to-be-tracked object can be determined as the distance increment, so that the distance information at the target time can be obtained by adding the distance increment to the current distance information. Similarly, the product of the interval length and the angular velocity of the to-be-tracked object can be determined as the angle increment, so that the angle information at the target time can be obtained by adding the angle increment to the current angle information.

[0079] In some application scenarios, if the to-be-tracked object is not moving at a constant speed, the acceleration can be determined based on the state vector information at the current moment and the state vector information at the previous moment, and then the distance increment and the angle increment can be determined based on the acceleration and the interval length, respectively, so as to predict the position information of the to-be-tracked object at the target moment.

[0080] Then, the processor can send a pre-tracking instruction to the camera, where the pre-tracking instruction is used to instruct the camera to pre-adjust the tracking parameter according to the position information of the to-be-tracked object at the target moment.

[0081] The tracking parameter can include, for example, a position parameter of an ROI (Region of Interest, ROI for short), a contrast at the time of acquisition, and other parameters that are essentially required when tracking the object.

[0082] In some application scenarios, for example, the position information of the to-be-tracked object at the target moment can be the center of the ROI, and the position of the ROI can be determined according to a preset length and width size, so as to obtain the position parameter of the ROI. After the position parameter of the ROI is sent to the camera, the camera can lock the ROI region in advance, so as to enter the tracking action at the target moment immediately after completing the tracking action at the previous moment.

[0083] In the implementation, the position information of the to-be-tracked object at the target moment can be predicted, so that the camera can be instructed to pre-adjust the tracking parameter, thereby facilitating the camera to enter the tracking action at the target moment immediately after completing the tracking action at the previous moment, reducing the tracking delay, and improving the accuracy and clarity of tracking to a certain extent.

[0084] In some application scenarios, for example, a risk warning can also be issued according to the predicted position information. For example, if the position information is located in the middle of a road, an alarm sound can be issued, so as to prompt the object to stop moving in time.

[0085] In some optional implementations, before the step 104 of sending the tracking instruction to the camera, the processor can further acquire a signal-to-noise ratio corresponding to each to-be-tracked object acquired by the MIMO-UWB radar; and then, for each to-be-tracked object, the confidence degree of the to-be-tracked object in the moving state can be determined according to the signal-to-noise ratio and the speed information corresponding to the to-be-tracked object. The signal-to-noise ratio can be regarded as the ratio of the signal strength received by the MIMO-UWB radar to the background noise strength. In the MIMO-UWB radar, the signal refers to the electromagnetic wave reflected back from the target, and the noise includes various interference and environmental noise. The calculation formula of the signal-to-noise ratio is usually: wherein, a signal-to-noise ratio corresponding to the to-be-tracked object, a signal power corresponding to the to-be-tracked object; a noise power existing when the signal power of the to-be-tracked object is received.

[0086] Therefore, the MIMO-UWB radar can determine the signal-to-noise ratio of the to-be-tracked object by detecting the signal power and the corresponding noise power collected when the to-be-tracked object is detected, and then transmit the signal-to-noise ratio to the processor.

[0087] Then, the processor can determine the confidence degree that each to-be-tracked object is in a motion state. The confidence degree can be regarded as a degree of certainty that the to-be-tracked object is in a motion state.

[0088] In some application scenarios, the confidence degree can be determined by, for example, the following calculation formula: ; wherein, the confidence degree is represented by, a signal-to-noise ratio corresponding to the to-be-tracked object, an influence degree of the signal-to-noise ratio term on the confidence degree, which can be, for example, 0.7; an influence degree of the velocity term on the confidence degree, wherein m+n=1, for example, when the influence degree of the signal-to-noise ratio term on the confidence degree is 0.7, the influence degree of the velocity term on the confidence degree can be 0.3. a maximum velocity of the to-be-tracked object, a change amount of velocity data of the to-be-tracked object in adjacent two radar frames; is an activation function, which is used to convert the signal-to-noise ratio into a value between 0 and 1 to quantify the contribution of the signal quality to the confidence degree; 10 represents a scaling factor of the signal-to-noise ratio.

[0089] In this way, if there is a confidence degree that is not less than a preset confidence threshold, the processor can send a tracking instruction to the camera. The preset confidence can include, for example, 0.7, 0.8, and other relatively high values.

[0090] It should be noted that if the confidence degree is less than the preset confidence threshold, it can be determined that the to-be-tracked object is in a stationary state. If the confidence degree is not less than the preset confidence threshold, it can be determined that the to-be-tracked object is in a motion state. Then, the tracking instruction can be sent to the camera only after it is determined that there is a to-be-tracked object in a motion state.

[0091] In the present implementation, considering that the objects to be tracked are mostly moving objects, the confidence degree that the to-be-tracked object is in a motion state can be determined, and whether to send a tracking instruction to the camera can be determined based on the confidence degree, so that the situation that the camera is frequently instructed to frequently open the focusing motor can be improved.

[0092] Those skilled in the art can understand that the sequence of writing each step in the above method of the specific embodiment does not mean a strict execution sequence and does not constitute any limitation on the implementation process. The specific execution sequence of each step should be determined by its function and possible internal logic.

[0093] Please refer to Figure 2 which shows a structural block diagram of an object tracking device provided by an embodiment of the application. The object tracking device can be a module, a program segment or code on an electronic device. It should be understood that the device corresponds to the above-mentioned Figure 1 method embodiment and can perform Figure 1 each step involved in the method embodiment.

[0094] Optionally, the above-mentioned object tracking device comprises an acquisition module 201, a separation module 202, a determination module 203 and a tracking module 204. The acquisition module 201 is configured to acquire overall state vector information of a plurality of to-be-tracked objects collected by a MIMO-UWB radar. The overall state vector information comprises speed information, distance information and angle information of the plurality of to-be-tracked objects. The separation module 202 is configured to separate the plurality of to-be-tracked objects into a plurality of independent to-be-tracked objects according to the overall state vector information. The determination module 203 is configured to determine position information of each to-be-tracked object at a current time according to state vector information corresponding to each to-be-tracked object, respectively. The tracking module 204 is configured to send a tracking instruction to a camera. The tracking instruction is configured to instruct the camera to track according to the position information of each to-be-tracked object at the current time and image information.

[0095] Optionally, the device further comprises a priority determination module configured to determine a tracking priority of each to-be-tracked object according to a preset rule after separating the plurality of to-be-tracked objects into a plurality of independent to-be-tracked objects according to the overall state vector information. The tracking instruction is configured to instruct the camera to track according to the tracking priority of each to-be-tracked object.

[0096] Optionally, the priority determination module is further configured to determine a distance weight corresponding to each to-be-tracked object according to distance information between the to-be-tracked object and the MIMO-UWB radar, acquire radar cross section information collected by the MIMO-UWB radar, determine a human-like feature weight of the to-be-tracked object according to the radar cross section information, determine a motion trend weight of the to-be-tracked object according to a plurality of image information collected by the camera, and determine the tracking priority corresponding to the to-be-tracked object according to one or more of the human-like feature weight, the motion trend weight and the distance weight.

[0097] Optionally, the priority determining module is further configured to: for each of the to-be-tracked objects, determine a distance weight corresponding to the to-be-tracked object according to distance information between the to-be-tracked object and the MIMO-UWB radar; and determine a tracking priority of each of the to-be-tracked objects according to the number of tracking requirements and the distance weight corresponding to each of the to-be-tracked objects.

[0098] Optionally, the apparatus further comprises an updating module configured to: after the tracking instruction is sent to the camera, for each of the to-be-tracked objects, update position information of the to-be-tracked object in real time according to state vector information corresponding to the to-be-tracked object based on a Kalman filtering algorithm; and send an updated tracking instruction to the camera.

[0099] Optionally, the apparatus further comprises an adjusting module configured to: after the plurality of to-be-tracked objects are separated into a plurality of independent to-be-tracked objects according to the overall state vector information, determine a sampling rate of the MIMO-UWB radar and / or an image acquisition frame rate of the camera according to a speed corresponding to each of the to-be-tracked objects; wherein the sampling rate is positively correlated with the speed, and the image acquisition frame rate is positively correlated with the speed.

[0100] Optionally, the apparatus further comprises a predicting module configured to: after the position information of each of the to-be-tracked objects at the current time is determined according to the state vector information corresponding to each of the to-be-tracked objects, for each of the to-be-tracked objects, predict position information of the to-be-tracked object at a target time according to the state vector information of the to-be-tracked object at the current time; the target time is any time after the current time; and send a pre-tracking instruction to the camera, the pre-tracking instruction being used to instruct the camera to adjust a tracking parameter in advance according to the position information of the to-be-tracked object at the target time.

[0101] Optionally, the apparatus further comprises a confidence determining module configured to: before the tracking instruction is sent to the camera, acquire a signal-to-noise ratio corresponding to each of the to-be-tracked objects collected by the MIMO-UWB radar; for each of the to-be-tracked objects, determine a confidence that the to-be-tracked object is in a motion state according to the signal-to-noise ratio corresponding to the to-be-tracked object and speed information; and the tracking module 204 is further configured to: if there is a confidence that is not less than a preset confidence threshold, send the tracking instruction to the camera.

[0102] It should be noted that, for the convenience and brevity of description, the specific working process of the apparatus described above can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0103] Based on the same inventive concept, the present application also provides an object tracking system, please refer to Figure 3 , which shows a structural diagram of an object tracking system provided by an embodiment of the present application, as shown in Figure 3 , the object tracking system can include a MIMO-UWB radar 301, a processor 302 and a camera 303. Wherein the MIMO-UWB radar 301 is configured to obtain overall state vector information of a plurality of objects to be tracked; the overall state vector information includes speed information, distance information and angle information of the plurality of objects to be tracked; the processor 302 is configured to obtain the overall state vector information of the plurality of objects to be tracked collected by the MIMO-UWB radar 301; and separate the plurality of objects to be tracked into a plurality of independent objects to be tracked according to the overall state vector information; determine the position information of each object to be tracked at the current time according to the state vector information corresponding to each object to be tracked respectively; and send a tracking instruction to the camera 303; the camera 303 is configured to track according to the position information of each object to be tracked at the current time and image information in the tracking instruction.

[0104] It should be noted that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0105] Please refer to Figure 4 , Figure 4 , which shows a structural diagram of an electronic device for executing the object tracking method provided by an embodiment of the present application. The electronic device can include at least one processor 401, such as a CPU, at least one communication interface 402, at least one memory 403 and at least one communication bus 404. Wherein the communication bus 404 is configured to realize the direct connection communication of these components. Wherein the communication interface 402 of the device in the present application is configured to communicate with other node devices. The memory 403 can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory. The memory 403 can also be at least one storage device located away from the foregoing processor. The memory 403 stores computer readable instructions, when the computer readable instructions are executed by the processor 401, the electronic device can execute the method provided by each method embodiment described above.

[0106] It can be understood that Figure 4 , the structure shown is only schematic, the electronic device can further include more or less components than those shown in Figure 4 , or have a different configuration from Figure 4 . Figure 4The components shown in the figures can be implemented in hardware, software or a combination thereof.

[0107] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor, and the computer program can execute the method provided by each method embodiment.

[0108] The embodiment of the present application provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer can execute the method provided by each method embodiment.

[0109] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0110] In addition, the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0111] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0112] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An object tracking method, characterized in that, include: Acquire overall state vector information of multiple objects to be tracked by MIMO-UWB radar; the overall state vector information includes velocity information, distance information, and angle information of the multiple objects to be tracked; Based on the overall state vector information, the multiple objects to be tracked are separated into multiple independent objects to be tracked; Based on the state vector information corresponding to each object to be tracked, determine the position information of each object at the current moment; Send a tracking command to the camera; wherein the tracking command is used to instruct the camera to track each object based on its current position information and image information.

2. The method according to claim 1, characterized in that, After separating the plurality of trackable objects into a plurality of independent trackable objects based on the overall state vector information, the method further includes: Based on preset rules, the tracking priority of each object to be tracked is determined; and The tracking command is used to instruct the camera to track each object according to its tracking priority.

3. The method according to claim 2, characterized in that, The step of determining the tracking priority of each object to be tracked according to preset rules includes: For each of the objects to be tracked, a distance weight corresponding to the object to be tracked is determined based on the distance information between the object to be tracked and the MIMO-UWB radar. The radar cross section information collected by the MIMO-UWB radar is obtained, and the human-like feature weights of the object to be tracked are determined based on the radar cross section information. Based on the multi-frame image information captured by the camera, the motion trend weight of the object to be tracked is determined; The tracking priority of the object to be tracked is determined based on one or more of the human-like feature weights, the motion trend weights, and the distance weights.

4. The method according to claim 2, characterized in that, The step of determining the tracking priority of each object to be tracked according to preset rules includes: For each of the objects to be tracked, the distance weight corresponding to the object to be tracked is determined based on the distance information between the object to be tracked and the MIMO-UWB radar. The tracking priority of each object is determined based on the number of tracking requests and the distance weight of each object to be tracked.

5. The method according to claim 1, characterized in that, After sending the tracking command to the camera, the method further includes: For each object to be tracked, based on the state vector information corresponding to that object, the position information of the object is updated in real time according to the Kalman filter algorithm; and Send updated tracking commands to the camera.

6. The method according to claim 1, characterized in that, After separating the plurality of trackable objects into a plurality of independent trackable objects based on the overall state vector information, the method further includes: The sampling rate of the MIMO-UWB radar and / or the image acquisition frame rate of the camera are determined based on the speeds of each object to be tracked; wherein the sampling rate is positively correlated with the speed, and the image acquisition frame rate is positively correlated with the speed.

7. The method according to any one of claims 1-6, characterized in that, After determining the position information of each trackable object at the current moment based on the state vector information corresponding to each trackable object, the method further includes: For each object to be tracked, the position information of the object at the target time is predicted based on the state vector information of the object at the current time; the target time is any time after the current time. A pre-tracking command is sent to the camera, which instructs the camera to pre-adjust tracking parameters based on the position information of the object to be tracked at the target time.

8. The method according to claim 1, characterized in that, Before sending the tracking command to the camera, the method further includes: Obtain the signal-to-noise ratio of each tracked object acquired by the MIMO-UWB radar; For each object to be tracked, the confidence level that the object is in motion is determined based on its signal-to-noise ratio and velocity information; and Sending tracking commands to the camera includes: If a confidence level of not less than a preset confidence threshold exists, a tracking command is sent to the camera.

9. An object tracking system, characterized in that, include: A MIMO-UWB radar is used to acquire overall state vector information of multiple objects to be tracked; the overall state vector information includes the velocity information, distance information, and angle information of the multiple objects to be tracked. The processor is configured to acquire overall state vector information of multiple objects to be tracked from MIMO-UWB radar; and, based on the overall state vector information, separate the multiple objects to be tracked into multiple independent objects to be tracked; and, based on the state vector information corresponding to each object to be tracked, determine the position information of each object to be tracked at the current moment. And, send tracking commands to the camera; A camera is used to track each object to be tracked based on its current position information and image information in the tracking instructions.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they perform the method as described in any one of claims 1-8.

Images