A target tracking method combining classic recognition model with multimodal large model

Abstract

The present invention discloses a target tracking method integrating a classic recognition model with a multimodal large model, and relates to the technical field of target tracking and detection, including: obtaining multiple groups of multimodal information, training a deep learning model, and outputting an initial recognition model; calculating a recognition frame and a verification data set based on position information, and outputting a recognition similarity value; calculating a recognition frame after division processing, and outputting a model accuracy rate; obtaining multimodal information, recognizing the multimodal information using a multimodal recognition model, and outputting a recognition frame; and updating the recognition frame in real time using a tracking algorithm. The present invention is used to solve the problem that the current target detection algorithm has limited generalization performance and large consumption of computing resources due to the lack of a more efficient, more flexible, and more adaptable target tracking solution in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking and detection, and in particular to a target tracking method that combines a classical recognition model with a multi-modal large model. Background Art

[0002] With the widespread application of unmanned technology, especially in low-altitude monitoring, search and rescue, agricultural plant protection, intelligent logistics and other fields, high-precision target tracking has become one of the key technologies. However, the existing unmanned system target tracking methods face many challenges, especially in the case of dynamic and changing environments and complex target types, the limitations of traditional methods are becoming more and more obvious.

[0003] Existing improvements for target tracking detection usually optimize the target tracking algorithm and improve the performance of the target tracking model in different scenarios. For example, in the Chinese patent with invention publication number CN111489377A, a target tracking self-evaluation method and device are disclosed. The scheme is to obtain multiple trajectories by tracking the target in the video through the target tracking model, as well as the number of multiple trajectories, the number of frames, and the target features in the trajectory. Parameters such as the evaluation result can be obtained, so that the target tracking model can be evaluated in any scene, the performance of the target tracking model in different scenes can be judged, and the parameters can be optimized; although these models perform well on specific data sets, the performance of these models drops sharply when encountering unseen targets or complex backgrounds, especially when the unmanned system is operating at low altitude or on the road, ground objects, lighting changes, occlusions and other factors will seriously affect the tracking accuracy. In order to achieve high accuracy, many target tracking algorithms need to consume a lot of computing resources, which is a heavy burden for resource-constrained unmanned platforms, limiting real-time performance and energy efficiency; in view of this, it is necessary to improve the existing target tracking detection. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the prior art to a certain extent. By improving target tracking detection, it is used to solve the problem that the current target detection algorithm has limited generalization performance and high computing resource consumption due to the lack of a more efficient, flexible and adaptable target tracking solution in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present application provides a target tracking method that integrates a classic recognition model with a multimodal large model, comprising the following steps:

[0006] Acquire multiple sets of multimodal information, divide the multimodal information, and output training data sets and verification data sets; train the deep learning model based on the training data sets and output the initial recognition model;

[0007] Use the initial recognition model to recognize the verification data set and output the recognition box; establish a coordinate system in the multimodal information, and obtain the position information of the recognition box based on the coordinate system; calculate the recognition box and the verification data set based on the position information, and output the recognition similarity value;

[0008] Analyze the recognition similarity values, and divide the recognition frames based on the analysis results; calculate the recognition frames after division and output the model accuracy; analyze the model accuracy, and retrain or output a multimodal recognition model based on the analysis results;

[0009] Acquire multimodal information, use the multimodal recognition model to recognize the multimodal information, and output the recognition box; use the tracking algorithm to update the recognition box in real time; monitor the update results, and output the re-recognition information based on the monitoring results or do no processing.

[0010] Furthermore, multiple groups of multimodal information are obtained, the multimodal information is divided, and a training data set and a verification data set are output; a deep learning model is trained based on the training data set, and the output multimodal recognition model includes:

[0011] Acquire multiple sets of multimodal information, wherein the multimodal information includes multiple sets of modal data acquired by unmanned equipment, including images, videos, and radar signals; the images or videos also include a search box containing an identified target;

[0012] All multimodal information is randomly divided, the multimodal information with the first proportion is divided into the training data set, and the remaining multimodal information is divided into the verification data set;

[0013] The training data set is input into the deep learning model for training, and the resulting deep learning model is output and marked as the initial recognition model.

[0014] Furthermore, the verification data set is recognized using the initial recognition model, and a recognition frame is output; a coordinate system is established in the multimodal information, and the position information of the recognition frame is obtained based on the coordinate system, including:

[0015] The verification data set is input into the initial recognition model, and the initial recognition model recognizes each image or video data in the verification data set and outputs a recognition frame, wherein the recognition frame includes a recognition target;

[0016] Establishing a rectangular coordinate system in the picture or video frame with the horizontal direction as the x-axis, the vertical direction as the y-axis, and the first length as the unit length;

[0017] Get the coordinates of the four vertices of the recognition frame, calculate the sum of the horizontal coordinate and the vertical coordinate of each vertex coordinate, and mark them as the position judgment value;

[0018] Mark the vertex coordinates corresponding to the minimum position judgment value as identification reference coordinates;

[0019] Get the coordinates of the four vertices of the search box and calculate the position judgment value of each vertex coordinate respectively;

[0020] The vertex coordinates corresponding to the minimum position judgment value are marked as search reference coordinates.

[0021] Furthermore, the recognition box and the verification data set are calculated based on the position information, and the output recognition similarity value includes:

[0022] Calculate the distance from the identification reference coordinate to the search reference coordinate, marked as error distance;

[0023] Calculate the area of ​​the recognition box and mark it as the recognition area; calculate the area of ​​the search box and mark it as the search area;

[0024] Use similarity calculation formula to calculate the recognition area, search area and error distance, and output the recognition similarity value;

[0025] The similarity calculation formula is configured as: ; Where SX is the recognition similarity value, BS is the recognition area, MS is the search area, and WJ is the error distance.

[0026] Furthermore, the recognition similarity values ​​are analyzed, and the recognition frames are divided based on the analysis results; the recognition frames after the division are calculated, and the output model accuracy includes:

[0027] Compare the size of the recognition similarity value and the similarity threshold. If the recognition similarity value is greater than the similarity threshold, no division is performed; if the recognition similarity value is less than or equal to the similarity threshold, the recognition frame corresponding to the recognition similarity value is divided into the first group;

[0028] Calculate the ratio of the number of recognition boxes in the first group to the total number of recognition boxes, and mark it as the model accuracy.

[0029] Furthermore, the model accuracy is analyzed, and retraining or outputting the multimodal recognition model based on the analysis results includes:

[0030] Compare the model accuracy with the accuracy threshold. If the model accuracy is greater than or equal to the accuracy threshold, mark the initial recognition model as a multimodal recognition model and output it.

[0031] If the model accuracy is less than the accuracy threshold, the training data set is input into the initial recognition model and the model parameters are fine-tuned. The validation set data is recognized using the initial recognition model after parameter fine-tuning, and the model accuracy of the initial recognition model after parameter fine-tuning is calculated. The relationship between the model accuracy and the accuracy threshold is compared again.

[0032] Furthermore, the multimodal information is obtained, the multimodal information is recognized using a multimodal recognition model, and a recognition frame is output; and the recognition frame is updated in real time using a tracking algorithm, including:

[0033] Use unmanned equipment to obtain multimodal information, use the multimodal recognition model to recognize the multimodal information, output the recognition frame, and suspend the operation of the multimodal recognition model;

[0034] The tracking algorithm is used to track and identify the recognition target in the recognition frame, and the recognition frame is updated in real time based on the tracking and identification.

[0035] Further, monitoring the update result, and outputting the re-identification information or not processing it based on the monitoring result includes:

[0036] At any time, if the tracking algorithm loses the identification target in the multimodal information, the re-identification information is output;

[0037] When the re-identification information is received, the multimodal recognition model is restarted, and the multimodal information is recognized and the recognition box is output.

[0038] Beneficial effects of the present invention: The present invention obtains multiple groups of multimodal information, divides the multimodal information, uses the training data set to train the deep learning model, and outputs the initial recognition model; then calculates the model accuracy of the initial recognition model, and performs retraining processing or outputs the multimodal recognition model based on the analysis result; the advantage is that by verifying the trained model, the model quality can be judged, and retraining is performed when the model quality does not meet the requirements; the quality of the multimodal recognition model is ensured, and the accuracy of subsequent recognition of multimodal information is improved;

[0039] The present invention also obtains multimodal information, uses a multimodal recognition model to recognize the multimodal information, and outputs a recognition frame; uses a tracking algorithm to update the recognition frame in real time; monitors the update results, and outputs re-recognition information based on the monitoring results or does not process it; the advantage of this is that first, the multimodal perception large model technology is used to preliminarily recognize the target, the generalization of the multimodal model is brought into play, and the open field recognition performance is improved, and then combined with classic tracking algorithms such as KCF to ensure the efficiency of tracking, thereby obtaining a high-performance, high-precision new unmanned equipment target recognition and tracking method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the steps of the method of the present invention;

[0041] Figure 2 This is a schematic diagram of the effect obtained by using the initial recognition model for recognition in the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example 1, please refer to Figure 1 As shown, a target tracking method that integrates a classic recognition model with a multimodal large model includes the following steps:

[0044] Step S1, obtaining multiple groups of multimodal information, dividing the multimodal information, and outputting a training data set and a verification data set; training a deep learning model based on the training data set, and outputting an initial recognition model; Step S1 also includes the following sub-steps:

[0045] Step S101, obtaining multiple sets of multimodal information, where the multimodal information includes multiple sets of modal data obtained by using unmanned equipment, including images, videos, and radar signals; the images or videos also include a search box containing an identified target;

[0046] Step S102, dividing all the multimodal information in a random manner, dividing the multimodal information of the first proportion into a training data set, and dividing the remaining multimodal information into a verification data set;

[0047] In specific implementation, the first proportion is set to 80%, and when obtaining multiple sets of multimodal information, the number obtained should not be less than 200; when training the model, the more training set data is used, the better the model effect is, and it is not easy to limit the generalization ability of the model, so that the trained model can better cope with diverse target categories; taking 200 multimodal information as an example, 80%, that is, 160 multimodal information, is divided into the training data set, which can ensure the quality of the model to a minimum; at the same time, under this premise, 40 multimodal information is divided into the verification data set, which ensures the data volume of the verification set and avoids the situation where the model quality is misjudged due to too little data;

[0048] Step S103: input the training data set into the deep learning model for training, and output the obtained deep learning model, which is marked as the initial recognition model.

[0049] Step S2, using the initial recognition model to recognize the verification data set, and outputting the recognition frame; establishing a coordinate system in the multimodal information, and obtaining the position information of the recognition frame based on the coordinate system; calculating the recognition frame and the verification data set based on the position information, and outputting the recognition similarity value; Step S2 also includes the following sub-steps:

[0050] Step S201, inputting the verification data set into the initial recognition model, the initial recognition model recognizes each image or video data in the verification data set, and outputs a recognition box, wherein the recognition box includes the recognition target;

[0051] See also Figure 2 As shown, Figure 2 T1 is the search box in the acquired multimodal information, T2 is the recognition target, T3 is the recognition box obtained by using the initial recognition model to recognize the recognition target, T4 is the search reference coordinate, T5 is the recognition reference coordinate, and the numbers next to the search box and the recognition box represent the side length of the box;

[0052] Taking the unit length of 2cm as an example, among the four vertex coordinates of the search box and the recognition box, the coordinates with the smallest position judgment value must be the coordinates of the lower left corner close to the origin, namely T4 (2, 2) and T5 (3, 3); the error distance is calculated by using the distance formula of two points in the coordinate system to be √2; then the area of ​​the recognition box is calculated based on the coordinates to be 30, and the area of ​​the search box is 12;

[0053] Step S202, establishing a rectangular coordinate system in the picture or video frame with the horizontal direction as the x-axis, the vertical direction as the y-axis, and the first length as the unit length;

[0054] Step S203, obtaining the coordinates of the four vertices of the identification frame, respectively calculating the sum of the horizontal coordinate and the vertical coordinate of each vertex coordinate, marking them as position judgment values; marking the vertex coordinate corresponding to the minimum position judgment value as the identification reference coordinate;

[0055] Step S204, obtaining the coordinates of the four vertices of the search box, and calculating the position judgment value of each vertex coordinate respectively; marking the vertex coordinate corresponding to the minimum position judgment value as the search reference coordinate;

[0056] Step S205, calculating the distance from the identification reference coordinate to the search reference coordinate, marked as the error distance;

[0057] Step S206, calculating the area of ​​the recognition box and marking it as the recognition area; calculating the area of ​​the search box and marking it as the search area;

[0058] Step S207, using a similarity calculation formula to calculate the recognition area, search area and error distance, and outputting a recognition similarity value;

[0059] The similarity calculation formula is configured as: ; Where SX is the recognition similarity value, BS is the recognition area, MS is the search area, and WJ is the error distance;

[0060] In the specific implementation, if BS is 30, MS is 12, and WJ is √2, the calculated recognition similarity value is 1.5+√2; the smaller the recognition similarity value, the closer the recognition box is to the search box; the purpose of calculating the recognition similarity value is to make the recognition box recognized by the trained initial recognition model closer to the search box in size and position, so as to improve the quality of model training.

[0061] Step S3, analyzing the recognition similarity value, dividing the recognition frame based on the analysis result; calculating the recognition frame after the division process, and outputting the model accuracy; analyzing the model accuracy, retraining or outputting the multimodal recognition model based on the analysis result; Step S3 also includes the following sub-steps:

[0062] Step S301, comparing the recognition similarity value with the similarity threshold, if the recognition similarity value is greater than the similarity threshold, no division processing is performed; if the recognition similarity value is less than or equal to the similarity threshold, the recognition box corresponding to the recognition similarity value is divided into the first group;

[0063] In specific implementation, the similarity threshold can be set to 2. The smaller the similarity threshold is set, the higher the quality requirement for the trained model is. It can be adjusted by yourself in actual application.

[0064] Step S302, calculating the ratio of the number of recognition boxes in the first group to the total number of recognition boxes, and marking it as the model accuracy;

[0065] Step S303, comparing the model accuracy with the accuracy threshold, if the model accuracy is greater than or equal to the accuracy threshold, marking the initial recognition model as a multimodal recognition model and outputting it;

[0066] Step S304: if the model accuracy is less than the accuracy threshold, the training data set is input into the initial recognition model, and the model parameters are fine-tuned; the validation set data is recognized using the initial recognition model after parameter fine-tuning, the model accuracy of the initial recognition model after parameter fine-tuning is calculated, and the relationship between the model accuracy and the accuracy threshold is compared again;

[0067] In the specific implementation, the accuracy threshold is set to 95%. Normally, after multiple trainings using the deep learning model, the obtained model accuracy can reach 98% or even higher. Setting it to 95% without affecting the actual application can reduce the number of subsequent trainings and improve the efficiency of model training.

[0068] Step S4, obtaining multimodal information, using a multimodal recognition model to recognize the multimodal information, and outputting a recognition frame; using a tracking algorithm to update the recognition frame in real time; monitoring the update result, and outputting re-recognition information based on the monitoring result or not processing; Step S4 also includes the following sub-steps:

[0069] Step S401, using unmanned equipment to obtain multimodal information, using a multimodal recognition model to recognize the multimodal information, outputting a recognition frame, and pausing the operation of the multimodal recognition model;

[0070] Step S402, using a tracking algorithm to track and identify the identification target in the identification frame, and updating the identification frame in real time based on the tracking and identification;

[0071] In specific implementation, after the multimodal recognition model identifies the target, it passes the target information to a lightweight tracking algorithm, such as KCF (Kernelized Correlation Filters); at the same time, the multimodal recognition model is paused to reduce the amount of calculation in the system; the KCF algorithm is known for its efficient tracking performance, and can continuously track the target while maintaining a high frame rate and consuming less computing resources; by combining the recognition results of the large model, the KCF algorithm can lock the target more accurately and reduce the probability of mistracking and target loss;

[0072] Step S403, at any time, if the tracking algorithm loses the identification target in the multimodal information, outputting the re-identification information;

[0073] In specific implementation, during the tracking process, if the KCF algorithm detects that the target is lost, that is, the target exceeds the preset tracking range or the tracking fails due to occlusion, the system will automatically trigger the feedback mechanism; at this time, the system will restart the multimodal recognition model and perform a new round of scanning and recognition of the current environment; the identified target location information will be passed to the KCF algorithm again to resume the tracking process to ensure the continuity and integrity of the target;

[0074] Step S404, when re-recognition information is received, restart the multimodal recognition model, recognize the multimodal information, and output a recognition box.

[0075] Through the description of the above implementation methods, the embodiments of the present invention can be provided as methods, systems or computer program products. Based on such an understanding, the above technical solutions can be essentially or partly contributed to the prior art in the form of software products, which can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and include several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] In the embodiments provided in the present application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of systems, modules and units can be electrical, mechanical or other forms.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target tracking method that integrates a classic recognition model with a multimodal large model, characterized in that: The steps include: Acquire multiple sets of multimodal information, divide the multimodal information, and output training data sets and verification data sets; train the deep learning model based on the training data sets and output the initial recognition model; Use the initial recognition model to identify the verification data set and output the recognition box; Establish a coordinate system in the multimodal information, and obtain the position information of the recognition frame based on the coordinate system; Calculate the recognition box and the verification data set based on the position information and output the recognition similarity value; Analyze the recognition similarity values ​​and divide the recognition box based on the analysis results; Calculate the recognition frame after segmentation and output the model accuracy; Analyze the model accuracy and perform retraining or output a multimodal recognition model based on the analysis results; Acquire multimodal information, use the multimodal recognition model to recognize the multimodal information, and output the recognition box; Use the tracking algorithm to update the recognition frame in real time; monitor the update results, and output the re-recognition information or do no processing based on the monitoring results; Obtain multiple sets of multimodal information, divide the multimodal information, and output training data sets and verification data sets; The deep learning model is trained based on the training data set, and the output multimodal recognition model includes: Acquire multiple sets of multimodal information, wherein the multimodal information includes multiple sets of modal data acquired by unmanned equipment, including images, videos, and radar signals; the images or videos also include a search box containing an identified target; All multimodal information is randomly divided, the multimodal information with the first proportion is divided into the training data set, and the remaining multimodal information is divided into the verification data set; Input the training data set into the deep learning model for training, and output the obtained deep learning model, which is marked as the initial recognition model; The validation data set is recognized using the initial recognition model, and the recognition frame is output; a coordinate system is established in the multimodal information, and the position information of the recognition frame is obtained based on the coordinate system, including: The verification data set is input into the initial recognition model, and the initial recognition model recognizes each image or video data in the verification data set and outputs a recognition frame, wherein the recognition frame includes a recognition target; Establishing a rectangular coordinate system in the picture or video frame with the horizontal direction as the x-axis, the vertical direction as the y-axis, and the first length as the unit length; Get the coordinates of the four vertices of the recognition frame, calculate the sum of the horizontal coordinate and the vertical coordinate of each vertex coordinate, and mark them as the position judgment value; Mark the vertex coordinates corresponding to the minimum position judgment value as identification reference coordinates; Get the coordinates of the four vertices of the search box and calculate the position judgment value of each vertex coordinate respectively; Mark the vertex coordinates corresponding to the minimum position judgment value as the search reference coordinates; Based on the location information, the recognition box and the verification data set are calculated, and the output recognition similarity values ​​include: Calculate the distance from the identification reference coordinate to the search reference coordinate, marked as error distance; Calculate the area of ​​the recognition box and mark it as the recognition area; calculate the area of ​​the search box and mark it as the search area; Use similarity calculation formula to calculate the recognition area, search area and error distance, and output the recognition similarity value; The similarity calculation formula is configured as: ; Where SX is the recognition similarity value, BS is the recognition area, MS is the search area, and WJ is the error distance.

2. According to claim 1, a target tracking method combining a classic recognition model with a multimodal large model is characterized in that: Analyze the recognition similarity values ​​and divide the recognition box based on the analysis results; Calculate the recognition box after segmentation, and the output model accuracy includes: Compare the size of the recognition similarity value and the similarity threshold. If the recognition similarity value is greater than the similarity threshold, no division is performed; if the recognition similarity value is less than or equal to the similarity threshold, the recognition frame corresponding to the recognition similarity value is divided into the first group; Calculate the ratio of the number of recognition boxes in the first group to the total number of recognition boxes, and mark it as the model accuracy.

3. The target tracking method according to claim 2, characterized in that: Analyze the model accuracy and retrain or output the multimodal recognition model based on the analysis results, including: Compare the model accuracy with the accuracy threshold. If the model accuracy is greater than or equal to the accuracy threshold, mark the initial recognition model as a multimodal recognition model and output it. If the model accuracy is less than the accuracy threshold, the training data set is input into the initial recognition model and the model parameters are fine-tuned. The validation set data is recognized using the initial recognition model after parameter fine-tuning, and the model accuracy of the initial recognition model after parameter fine-tuning is calculated. The relationship between the model accuracy and the accuracy threshold is compared again.

4. The target tracking method according to claim 3, characterized in that: Acquire multimodal information, use the multimodal recognition model to recognize the multimodal information, and output the recognition box; Using the tracking algorithm to update the recognition box in real time includes: Use unmanned equipment to obtain multimodal information, use the multimodal recognition model to recognize the multimodal information, output the recognition frame, and suspend the operation of the multimodal recognition model; The tracking algorithm is used to track and identify the recognition target in the recognition frame, and the recognition frame is updated in real time based on the tracking and identification.

5. The target tracking method according to claim 4, characterized in that: Monitoring the update results and outputting re-identification information or not processing it based on the monitoring results include: At any time, if the tracking algorithm loses the identification target in the multimodal information, the re-identification information is output; When the re-identification information is received, the multimodal recognition model is restarted, and the multimodal information is recognized and the recognition box is output.

Images