Model optimization methods, devices, video processing systems, storage media, and terminals

Abstract

The application discloses a model optimization method and device, a video processing system, a storage medium and a terminal. The method comprises the following steps: acquiring at least one tracking target and a target detection model to be optimized; performing target tracking on the at least one tracking target, and acquiring matching state transition data corresponding to each tracking target; generating tracking state transition data corresponding to each tracking target according to the matching state transition data; determining an abnormal type and an abnormal frame image set according to the matching state transition data and the tracking state transition data; and optimizing the target detection model according to the abnormal frame image set and the abnormal type. By judging whether the tracking abnormality is caused by model detection error, the target detection model is optimized based on the transition condition only when it is determined that the model error exists, the model is optimized based on the abnormal type, and the optimization effect of the model is better.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a model optimization method, apparatus, video processing system, storage medium, and terminal. Background Technology

[0002] Object detection is a fundamental technique in computer vision, enabling software systems to detect and locate objects from images or video streams. Deep learning-based object detection models have made significant progress, achieving high detection performance. However, there are still instances where individual video frames or time periods perform poorly, resulting in inaccurate detection results.

[0003] In existing technologies, the optimization of object detection models typically involves extracting and labeling video frames that perform poorly, and then retraining the model based on these labeled frames. However, this optimization method lacks specificity, resulting in unsatisfactory optimization performance of the object detection model. Summary of the Invention

[0004] In view of this, this application provides a model optimization method, apparatus, video processing system, storage medium and terminal, the main purpose of which is to solve the technical problem that the existing method of extracting video frames with unsatisfactory processing effects to optimize the target detection model lacks specificity and leads to poor model optimization results.

[0005] According to a first aspect of this application, a model optimization method is provided, the method comprising:

[0006] Obtain at least one tracking target and a target detection model to be optimized;

[0007] Track at least one target and obtain matching state transition data for each target.

[0008] Based on the matching state transition data, generate tracking state transition data corresponding to each tracking target;

[0009] Based on the matching state transition data and the tracking state transition data, the anomaly type and the set of anomaly frame images are determined;

[0010] The target detection model is optimized based on the abnormal frame image set and the abnormality type.

[0011] Optionally, the step of obtaining at least one tracking target and a target detection model to be optimized specifically includes:

[0012] In response to a model optimization request, obtain multiple video frame images and object detection models included in the model optimization request;

[0013] Multiple video frame images are input into the object detection model to identify multiple object boxes in the multiple video frame images, and each object box is marked in the multiple video frame images;

[0014] At least one tracking target is determined based on at least one target bounding box marked in the first frame image.

[0015] Optionally, the step of tracking at least one target and obtaining matching state transition data corresponding to each target specifically includes:

[0016] Using a pre-defined target tracking algorithm, target tracking is performed on each target based on multiple target bounding boxes marked in multiple video frame images, generating a matching status for each target in each video frame image, where the matching status includes successful matching or failed matching;

[0017] Based on multiple matching states, generate matching state transition data for each tracked target.

[0018] Optionally, the step of generating tracking state transition data for each tracking target based on the matching state transition data specifically includes:

[0019] Set the initial tracking status of each tracked target in the first frame image to an unconfirmed state;

[0020] Based on the matching status transition data, determine at least one tracking status for each tracking target during the tracking process, wherein the tracking status includes confirmed status, unconfirmed status, and deleted status;

[0021] Based on the initial tracking state and at least one tracking state, generate tracking state transition data for each tracking target.

[0022] Optionally, the step of determining at least one tracking state for each tracking target during the tracking process based on the matching state transition data specifically includes:

[0023] For any tracked target, the first frame number and the second frame number are determined based on the matching state transition data, where the first frame number is the number of consecutively successfully matched video frame images, and the second frame number is the number of consecutively unmatched video frame images.

[0024] If the first frame number is greater than or equal to the first preset threshold, the tracking status of the target corresponding to the first frame number is determined to be a confirmed status.

[0025] If the tracking status of the target is confirmed, and the second frame number is greater than or equal to the second preset threshold, the tracking status of the target is determined to change from confirmed to unconfirmed.

[0026] If the tracking status of the target is unconfirmed, and the matching result of the target in the second video frame is a failure, the tracking status of the target is changed from unconfirmed to deleted.

[0027] Optionally, the step of determining the anomaly type and anomaly frame image set corresponding to the target detection model based on the matching state transition data and the tracking state transition data specifically includes:

[0028] For any tracked target, if the tracking status transition data is from unconfirmed to confirmed, determine whether the matching status in the matching status transition data includes a matching failure.

[0029] If the matching status includes a failed match, obtain the overlap status and target bounding box position of the tracked target in the video frame image of the failed match;

[0030] Based on the overlap status and the target box position, determine whether target overlap or the target exceeds the screen.

[0031] If no target overlap or target exceeds the frame, the anomaly type is determined to be a missed detection by the model.

[0032] Generate an abnormal frame image set based on the video frame images that failed to match in the matching state transition data.

[0033] Optionally, the step of determining the anomaly type and anomaly frame image set corresponding to the target detection model based on the matching state transition data and the tracking state transition data further includes:

[0034] If the tracking state transition data is from unconfirmed state to confirmed state to deleted state, determine whether the number of the first frame of the tracked target is less than a third preset threshold.

[0035] If the number of frames in the first frame is less than the third preset threshold, it is determined that there is a tracking anomaly caused by the target detection model during the target tracking process;

[0036] If the first frame number is greater than or equal to the third preset threshold, obtain the overlap state and target box position of the tracked target in the video frame image where the match failed;

[0037] Based on the overlap status and the target bounding box position, determine whether target overlap has occurred and / or the target exceeds the screen.

[0038] If no target overlap or target exceeds the frame, the first frame number is compared with the third preset threshold.

[0039] If the number of the first frame is less than the third preset threshold, the anomaly type is determined to be a false detection by the model.

[0040] If the number of the first frame is greater than or equal to the third preset threshold, the anomaly type is determined to be a false detection by the model.

[0041] A set of abnormal frame images is generated based on the successfully matched video frame images in the matching state transition data.

[0042] Optionally, the step of determining the anomaly type and anomaly frame image set corresponding to the target detection model based on the matching state transition data and the tracking state transition data further includes:

[0043] When the tracking status transition data shows a transition from an unconfirmed state to a deleted state, the anomaly type is determined to be a false detection in the model.

[0044] A set of abnormal frame images is generated based on the successfully matched video frame images in the matching state transition data.

[0045] Optionally, the method further includes:

[0046] When there are multiple tracking targets, obtain the position information of the target bounding box corresponding to each tracking target in each video frame image;

[0047] Based on multiple location information, determine the intersection-over-union ratio (IoU) between any two tracked targets in each video frame image;

[0048] Based on the crossover ratio and the fourth preset threshold, the overlap state of each tracked target for each video frame image is generated.

[0049] According to a second aspect of this application, a model optimization apparatus is provided, the apparatus comprising:

[0050] The acquisition module is used to acquire at least one tracking target and a target detection model to be optimized;

[0051] The tracking module is used to track at least one target and acquire matching state transition data for each target.

[0052] The generation module is used to generate tracking status transition data for each tracking target based on the matching status transition data.

[0053] The determination module is used to determine the anomaly type and the set of anomaly frame images based on the matching state transition data and the tracking state transition data;

[0054] The optimization module is used to optimize the target detection model based on the abnormal frame image set and the abnormality type.

[0055] According to a third aspect of the present invention, a video processing system is provided, comprising: a model optimization apparatus as described in the second aspect;

[0056] A video acquisition device is communicatively connected to a model optimization device. The video acquisition device is used to acquire video streams and send the acquired video streams to the model optimization device.

[0057] According to a fourth aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored therein, the executable instruction causing a processor to perform an operation corresponding to the model optimization method of the first aspect.

[0058] According to a fifth aspect of the present invention, a terminal is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0059] The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the model optimization method of the first aspect.

[0060] By employing the above-described technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

[0061] This application provides a model optimization method, apparatus, video processing system, storage medium, and terminal. During continuous tracking, the matching state of each tracked target is monitored in each frame to obtain the matching state transitions, and thus the tracking state transitions over different time periods. Subsequently, the transitions are analyzed to identify anomalies caused by the model, determine the anomaly type of the target detection model and its corresponding anomalous frames, and then optimize the target detection model in a targeted manner based on the anomaly type and anomalous frames. Compared to existing technologies that extract frames where detection and tracking do not match as video frames with unsatisfactory model processing results and retrain the model based on these extracted frames, this approach suffers from the technical problem of not being able to optimize the model specifically for anomaly types, having a large amount of redundant data, and ultimately leading to poor optimization results for the target detection model. This application determines whether tracking anomalies are caused by model detection errors. Only when it is determined that the tracking anomaly is caused by a model detection error will the anomaly type and anomaly frame be determined based on the transformation situation to optimize the target detection model. This maximizes the extraction of effective anomaly frames related to the target detection model as training data, reduces redundant data, and optimizes the model in a targeted manner based on the anomaly type, resulting in better optimization performance. It also makes the target detection model more stable and reliable when facing different anomaly situations, which is of great significance for applying the model to target detection and tracking in different environments and scenarios.

[0062] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0063] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0064] Figure 1 This illustration shows one of the flowcharts of a model optimization method provided in an embodiment of this application;

[0065] Figure 2 This is a second schematic diagram of a model optimization method provided in an embodiment of this application;

[0066] Figure 3 This illustration shows one of the target bounding boxes in a video frame image provided in an embodiment of this application;

[0067] Figure 4 This illustrates a second schematic diagram of a target bounding box in a video frame image provided in an embodiment of this application;

[0068] Figure 5 This illustration shows one of the state transition diagrams provided in an embodiment of this application;

[0069] Figure 6 This illustrates the second schematic diagram of state transitions provided in an embodiment of this application;

[0070] Figure 7 This illustrates the third schematic diagram of state transitions provided in an embodiment of this application;

[0071] Figure 8 This paper shows a schematic diagram of the structure of a model optimization device provided in an embodiment of this application;

[0072] Figure 9 A schematic diagram of the terminal structure provided in an embodiment of this application is shown. Detailed Implementation

[0073] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0074] This application provides a model optimization method, such as... Figure 1 As shown, the method includes:

[0075] S101. Obtain at least one tracking target and a target detection model to be optimized.

[0076] This application provides a model optimization method. Specifically, an object detection model is used to detect and identify the location of an object in a video frame image and output a bounding box containing the object's location. It should be noted that the object detection model is a pre-trained model with high detection performance, and the detection results are generally accurate. However, the object detection model cannot achieve 100% accuracy; occasional missed detections and / or false detections may occur in individual frames or time periods (e.g., when the human pose changes, human targets may occasionally be missed, resulting in a flickering bounding box).

[0077] To improve the accuracy of object detection models and make their detection results more precise, existing technologies typically extract video frames where the model's detection and subsequent tracking do not match, treating these as anomalous frames indicating poor model performance. The model is then retrained based on these extracted anomalous frames to optimize it. However, during model optimization, the timing of the detection-tracking mismatch could be due to missed or false detections, or it could be caused by the tracked target exceeding the frame's range or overlapping with other tracked targets. In the latter case, the model performs normally, meaning the detection result is correct. Existing technologies simply extract these anomalous video frames for retraining without analyzing the specific anomalies. This makes it impossible to determine whether the anomalous tracking was caused by the model, resulting in a large amount of redundant data in the extracted video frames that is not beneficial for model optimization. The limited amount of effective data leads to poor model optimization performance.

[0078] Based on the above problems, this application proposes to acquire at least one tracking target, and while tracking at least one tracking target, to monitor the transition of the tracking state of the tracking target in real time to analyze whether tracking anomalies occur, and to conduct specific analysis of the anomalies to determine whether the tracking anomalies are caused by model detection anomalies, and to optimize the model in a targeted manner based on the analysis results.

[0079] S102. Perform target tracking on at least one target and obtain matching state transition data corresponding to each target.

[0080] In this step, the matching state transition data consists of the matching results (successful or unsuccessful) for each tracked target in each frame. Target tracking is performed on each target. During the tracking process, the target bounding box of each tracked target may overlap with the bounding boxes of other tracked targets in the video frame, or the target bounding box may move out of the frame. Furthermore, the target detection model may misjudge, such as false detection (incorrectly identifying a non-target bounding box as a tracked target) or missed detection (failing to detect the target bounding box). Both of these situations can lead to abnormalities in target tracking. To determine whether the tracking abnormality is caused by model detection anomalies, the tracking process of each tracked target needs to be monitored to obtain the matching state of each target in each video frame, and all matching states are aggregated to generate matching state transition data. Then, based on the transition of the matching states, it is determined whether any abnormalities have occurred during target tracking.

[0081] Specifically, the matching status of each tracked target is monitored in each video frame. In any video frame, if the tracked target is successfully identified, the matching status for that video frame is determined to be a successful match; if the tracked target is not successfully identified, the matching status for that video frame is determined to be a failed match. Then, all matching statuses are summarized to form the matching status transition data of the tracked target.

[0082] S103. Generate tracking state transition data for each tracking target based on the matching state transition data.

[0083] In this step, the tracking state transition data consists of the tracking state of each tracking target over multiple time periods. Specifically, the matching state transition data of each tracking target is analyzed, and based on the matching states of consecutive successful matches and / or consecutive failed matches, the tracking state of the tracking target in different time periods is generated, thereby generating the tracking state transition data of each tracking target.

[0084] By using the above method, the tracking state transition of each target in different time periods is determined. By combining the matching state transition information, tracking anomalies are analyzed to determine whether there are any detection anomalies in the target detection model.

[0085] S104. Based on the matching state transition data and the tracking state transition data, determine the anomaly type and the set of anomaly frame images.

[0086] S105. Optimize the target detection model based on the abnormal frame image set and the abnormality type.

[0087] In steps S104 and S105, the anomaly type refers to the detection anomaly type of the target detection model, including missed detections or false detections. The anomaly frame image set is the set of anomaly frames corresponding to each anomaly type. For any tracked target, anomalies during the tracking process may be caused by various factors, including non-target detection model issues, such as the target being partially or completely occluded by other targets, making it undetectable, or the target exceeding the frame; and target detection model issues, such as false detections or missed detections. Therefore, it is necessary to specifically analyze the anomalies in the tracking process based on the matching state transitions and tracking state transitions of the tracked target to determine whether the anomalies are caused by missed or false detections or by non-target detection model issues (i.e., the model is performing normally), and then optimize the target detection model accordingly based on the discrimination results.

[0088] Specifically, based on the matching state transition data and the tracking state transition data, after determining that there are tracking anomalies caused by model detection anomalies, the anomaly type corresponding to the model detection anomaly is determined based on the matching state transition and tracking state transition of the anomaly tracked target, as well as the set of anomaly frame images corresponding to each anomaly type. The set of anomaly frame images is used as training data for model optimization, and the target detection model is optimized in a targeted manner based on the anomaly type and anomaly frames.

[0089] By using the above methods, the types of model anomalies and anomaly frames can be clearly identified. Then, the model can be optimized for different anomaly types to improve the accuracy and robustness of the model. This makes the target detection model more stable and reliable when facing different anomaly situations, which is of great significance for the application of the model in target detection and tracking in different environments and scenarios.

[0090] The model optimization method provided in this application monitors the matching state of each tracked target in each frame during continuous tracking, obtaining the matching state transition status, and thus the tracking state transition status at different time periods. Subsequently, the transition status is analyzed to identify anomalies caused by the model, determining the anomaly type of the target detection model and its corresponding anomaly frames. Then, the target detection model is optimized in a targeted manner based on the anomaly type and anomaly frames. Compared to the prior art, which extracts frames where detection and tracking do not match as video frames with unsatisfactory model processing performance and retrains the model based on these extracted video frames for optimization, this method suffers from the technical problem of not being able to optimize the model specifically for anomaly types, having a large amount of redundant data, and ultimately leading to poor optimization performance of the target detection model. This application determines whether tracking anomalies are caused by model detection errors. Only when it is determined that the tracking anomaly is caused by a model detection error will the anomaly type and anomaly frame be determined based on the transformation situation to optimize the target detection model. This maximizes the extraction of effective anomaly frames related to the target detection model as training data, reduces redundant data, and optimizes the model in a targeted manner based on the anomaly type, resulting in better optimization performance. It also makes the target detection model more stable and reliable when facing different anomaly situations, which is of great significance for applying the model to target detection and tracking in different environments and scenarios.

[0091] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, this application provides another model optimization method, such as... Figure 2 As shown, the method includes:

[0092] S201. In response to the model optimization request, obtain multiple video frame images and the target detection model to be optimized included in the model optimization request.

[0093] In this step, the object detection model is used to detect and identify the location of objects in video frame images and output a bounding box containing the object's location. It should be noted that the object detection model is a pre-trained model with high detection performance, and the detection results are generally quite accurate. However, the object detection model cannot achieve 100% accuracy; occasional missed detections and / or false detections may occur in individual frames or time periods (e.g., human targets may occasionally be missed when the human pose changes, resulting in a flickering bounding box).

[0094] To improve the accuracy of object detection models and make their detection results more precise, existing technologies typically extract video frames where the model's detection and subsequent tracking do not match, treating these as anomalous frames indicating poor model performance. The model is then retrained based on these extracted anomalous frames to optimize it. However, during model optimization, the timing of the detection-tracking mismatch could be due to missed or false detections, or it could be caused by the tracked target exceeding the frame's range or overlapping with other tracked targets. In the latter case, the model performs normally, meaning the detection result is correct. Existing technologies simply extract these anomalous video frames for retraining without analyzing the specific anomalies. This makes it impossible to determine whether the anomalous tracking was caused by the model, resulting in a large amount of redundant data in the extracted video frames that is not beneficial for model optimization. The limited amount of effective data leads to poor model optimization performance.

[0095] Based on the above problems, this application proposes to acquire at least one tracking target, and while tracking at least one tracking target, to monitor the transition of the tracking state of the tracking target in real time to analyze whether tracking anomalies occur, and to conduct specific analysis of the anomalies to determine whether the tracking anomalies are caused by model detection anomalies, and to optimize the model in a targeted manner based on the analysis results.

[0096] In practical applications, object detection models can be used for human pose estimation or image segmentation. Specifically, if the object detection model's inference result itself contains a bounding box, the bounding box can be obtained directly. If the object detection model's inference result only contains human keypoints, the bounding box can be obtained by calculating these keypoints. Furthermore, if the object detection model's inference result is a binary image's target mask, the bounding box can be obtained by calculating the target mask. Specifically, such as... Figure 3 As shown, this is the target mask detected by the target detection model, such as... Figure 4 As shown, the target bounding box is obtained by calculating the AABB bounding box of the target mask.

[0097] S202. Input multiple video frame images into the target detection model, identify multiple target boxes in the multiple video frame images, and mark each target box in the multiple video frame images.

[0098] S203. Determine at least one tracking target based on at least one target bounding box marked in the first frame image.

[0099] In steps S202 and S203, after acquiring multiple video frame images, the multiple video frame images are input into the target detection model for inference. The target detection model analyzes each frame image and identifies and marks the target region in each video frame image to generate a target bounding box.

[0100] Furthermore, after marking the target bounding box in each video frame, one or more target bounding boxes contained in the first frame are selected as the starting point for tracking, which are one or more tracking targets.

[0101] Optionally, after the target detection model automatically selects one or more target boxes corresponding to the tracking targets located in the image, professionals can manually adjust the tracking targets based on the scenario or business needs to make the tracking targets more suitable for the application scenario and improve the practicality of the target detection model.

[0102] S204. Using a preset target tracking algorithm, based on multiple target boxes marked in multiple video frame images, target tracking is performed on each tracked target, and a matching status of each tracked target in each video frame image is generated, wherein the matching status includes successful matching or failed matching.

[0103] S205. Generate matching state transition data for each tracked target based on multiple matching states.

[0104] In steps S204 and S205, a preset target tracking algorithm is used to track each target based on the target bounding box in each subsequent video frame image and update the matching status. It can be understood that the matching status refers to the monitoring result of whether the tracked target matches / is associated with the target bounding box in each frame image. Specifically, the tracking matching status of each target is monitored in real time during the tracking process. If the target bounding box of the tracked target in a certain frame image can be located and identified, then the matching status of the tracked target in that image frame is confirmed as a successful match. If the target bounding box of the tracked target cannot be identified in a certain frame image, that is, it is determined that the target bounding box may be occluded in that image frame, or the target detection model fails to successfully detect the target bounding box in that image frame, then the matching status of the tracked target changes, and the matching status of the tracked target in that image frame is a failed match.

[0105] Furthermore, after monitoring and obtaining the matching status of each tracked target in each frame of the image, the multiple matching statuses are summarized and organized to form the matching status transformation data of each tracked target, and the matching transformation process of the tracked target is recorded. For example, the tracked target undergoes a transformation process of "matching successfully - matching failed - matching successfully again" in multiple video frame images.

[0106] Optionally, the target tracking algorithm can be a mainstream detection-based multi-target tracking algorithm such as Sort, DeepSort, or ByteTrack. This application does not make specific limitations here, but takes the DeepSort algorithm as an example.

[0107] S206. Set the initial tracking status of each tracked target in the first frame image to an unconfirmed state.

[0108] S207. Based on the matching state transition data, determine at least one tracking state for each tracking target during the tracking process, wherein the tracking state includes a confirmed state, an unconfirmed state, and a deleted state.

[0109] In steps S206 and S207, during the tracking process, based on the transition of matching status in different time periods, the same matching status in this time period can be summarized into the same tracking status, which is used to determine whether the target tracking has an abnormality in this time period. The tracking status includes confirmed status, unconfirmed status, and deleted status.

[0110] It is understandable that the initial tracking state of the target is the state before the target has started tracking. Therefore, the initial tracking state can also be set before the first frame image.

[0111] In one embodiment of this application, optionally, step S207, namely determining at least one tracking state for each tracking target during the tracking process based on the matching state transition data, specifically includes: for any tracking target, determining a first frame number and a second frame number based on the matching state transition data, wherein the first frame number is the number of consecutively successfully matched video frame images, and the second frame number is the number of consecutively unmatched video frame images; if the first frame number is greater than or equal to a first preset threshold, determining that the tracking state of the tracking target corresponding to the first frame number is a confirmed state; if the second frame number is greater than or equal to a second preset threshold when the tracking state of the tracking target is a confirmed state, determining that the tracking state of the tracking target is changed from a confirmed state to an unconfirmed state; if the matching result of the tracking target in the second video frame image is a match failure when the tracking state of the tracking target is an unconfirmed state, determining that the tracking state of the tracking target is changed from an unconfirmed state to a deleted state.

[0112] In this embodiment, during target tracking, the number of consecutive successful matches of the tracked target in the first frame and the number of consecutive unsuccessful matches in the second frame are obtained. The tracking status during the tracking process is determined based on the first frame number, the second frame number, and a set first preset threshold and a second preset threshold.

[0113] Specifically, for any tracked target, the matching transition can be divided into two scenarios: the first is "match successful - match failed - match successful again"; the second is "match successful - match failed - match failed (deleted)". If we combine this with the tracking status, the matching transition status of a tracked target within its lifecycle or over a specific period can be categorized into the following four scenarios: Figure 5 The diagram illustrates the tracking state transition in scenario A. In the first frame, the initial state is confirmed as unconfirmed. During subsequent tracking, the target continues to match successfully (i.e., the matching result never changes). The tracking state transition is: unconfirmed state (initial tracking state) - confirmed state. In this scenario, it indicates that the target bounding box can be continuously and correctly detected and tracked, meaning the tracking is performing well without any abnormalities. Figure 6 The diagram illustrates the tracking state transition in scenario B. The tracked target achieves successful matching in multiple consecutive video frames I, fails to match in multiple consecutive video frames II, and achieves successful matching in multiple consecutive video frames III. Therefore, the tracking state transition process is "successful matching - successful matching - ... - failed matching - failed matching - ... - successful matching". If the number of failed matching frames is less than a second preset threshold, it indicates that multiple failed matching video frames represent an unconfirmed state, meaning the tracking target remains in a confirmed state during the tracking process. Therefore, the tracking state transition is "unconfirmed state (initial state) - confirmed state". Figure 7 The diagram illustrates the tracking state transition in scenario C. The tracking target achieves successful matches in multiple consecutive video frames (I) but fails to match in multiple consecutive video frames (II). It's important to note that when the number of failed matches in the second frame reaches a second preset threshold, the tracking target is deleted. The tracking target matching transition process is "successful match - successful match - ... - failed match - failed match - ... - deleted (failed match)". Therefore, the tracking state transition of the tracking target is "unconfirmed state (initial state) - confirmed state - deleted state".

[0114] Furthermore, after the target is matched in the first frame, its initial state is set to unconfirmed. However, if the target fails to match in the next frame, it is deleted. The matching process of the target is "match successful - match failed - deleted (match failed)", and the tracking state of the target is "unconfirmed state (initial state) - deleted state".

[0115] Optionally, the first preset threshold and the second preset threshold can be adaptively adjusted according to the dynamic changes of the target and the background, which is not specifically limited in this application.

[0116] S208. Based on the initial tracking state and at least one tracking state, generate tracking state transition data for each tracking target.

[0117] In this step, tracking status transition data for each target is generated by summarizing the initial tracking status and at least one tracking status during the tracking process. Then, based on the tracking status transitions, it is analyzed whether any anomalies occurred during the tracking process.

[0118] S209. Based on the matching state transition data and the tracking state transition data, determine the anomaly type and the set of anomaly frame images.

[0119] In this step, the anomaly type refers to the type of anomaly detected by the target detection model, including missed detections or false detections. The anomaly frame image set is the collection of anomaly frames corresponding to each anomaly type. For any tracked target, anomalies during the tracking process may be caused by various factors, including non-target detection model issues, such as the target being partially or completely occluded by other targets, making it undetectable, or the target exceeding the frame; and target detection model issues, such as false detections or missed detections. Therefore, it is necessary to analyze the anomalies in the tracking process based on the matching state transitions and tracking state transitions of the tracked target to determine whether the anomalies are caused by missed or false detections, or by non-target detection model issues (i.e., the model is performing normally). Based on the judgment results, the target detection model can then be optimized in a targeted manner.

[0120] Specifically, based on the matching state transition data and the tracking state transition data, after determining that there are tracking anomalies caused by model detection anomalies, the anomaly type corresponding to the model detection anomaly is determined based on the matching state transition and tracking state transition of the anomaly tracked target, as well as the set of anomaly frame images corresponding to each anomaly type. The set of anomaly frame images is used as training data for model optimization, and the target detection model is optimized in a targeted manner based on the anomaly type and anomaly frames.

[0121] By using the above methods, the types of model anomalies and anomaly frames can be clearly identified. Then, the model can be optimized for different anomaly types to improve the accuracy and robustness of the model. This makes the target detection model more stable and reliable when facing different anomaly situations, which is of great significance for the application of the model in target detection and tracking in different environments and scenarios.

[0122] In one embodiment of this application, optionally, step S209, namely determining at least one tracking state for each tracking target during the tracking process based on the matching state transition data, specifically includes: for any tracking target, when the tracking state transition data changes from an unconfirmed state to a confirmed state, determining whether the matching state in the matching state transition data includes a matching failure; if the matching state includes a matching failure, obtaining the overlap state and target box position of the tracking target in the video frame image of the matching failure; determining whether target overlap or target exceeding the frame occurs based on the overlap state and target box position; if no target overlap or target exceeding the frame occurs, determining the anomaly type as model missed detection; and generating an abnormal frame image set based on the video frame image of the matching failure in the matching state transition data.

[0123] In this embodiment, when the tracking state transition data is from unconfirmed state (initial tracking state) to confirmed state, the confirmed state has two scenarios. One is where the tracking target continuously matches successfully and is continuously tracked until tracking is complete; this is a relatively ideal state. The other is where the tracking target's matching state fails for a certain time period, but matches successfully for the rest of the time period. However, the number of failed matching frames is small and does not meet the deletion criteria, so the tracking state remains unchanged. In this case, a tracking anomaly has occurred. Therefore, in this tracking transition scenario, it is determined whether the matching state transition data contains failed matching results. If it does, it indicates a tracking anomaly has occurred. At this point, it is necessary to consider situations such as target occlusion and targets moving out of the frame. That is, to obtain the overlap state of the tracking target at the point of failed matching and the target bounding box position. It should be noted that the video frame image of the failed matching is the first failed matching video frame in the matching state. The overlap state is used to determine whether the tracking target overlaps with other targets in that video frame, and the target bounding box position is used to determine whether the tracking target moves out of the frame in that video frame. If the tracked object does not overlap with other targets and the tracked object does not exceed the edge of the screen, the tracking anomaly is considered to be caused by a model detection anomaly.

[0124] Furthermore, if the tracked object overlaps with other targets or moves out of the frame in a video frame where a match fails, it indicates that the model is performing normally. In other words, the tracking anomaly is not caused by the target detection model, meaning that the abnormal video frame is redundant data for the target detection model. Even if the model is optimized using the abnormal frame, there will be no performance improvement, so there is no need to optimize the target detection model based on the tracking anomaly.

[0125] Furthermore, once it is determined that the tracking anomaly is caused by an anomaly detected by the target detection model, the model needs to be optimized based on the anomaly frames in which the anomaly occurred. The cause of the anomaly may be a missed detection or a false detection by the model. In order to improve the optimization effect, it is necessary to classify the anomaly type based on the anomaly situation, and then optimize the model in a targeted manner according to different anomaly types.

[0126] Specifically, if the model detects an anomaly and the tracking state transition data is from unconfirmed to confirmed, it indicates that the model missed detections in the confirmed state, meaning the anomaly type is model missed detection. In this case, it can be confirmed that the confirmed state contains a small number of consecutively failing video frames; these are the anomalous frames missed by the model. Therefore, after confirming the anomaly type as model missed detection, based on the matching state transition data, multiple video frame images that failed to match (such as...) are identified. Figure 6 In the middle, the video frame images corresponding to time period II are marked and extracted, and summarized to generate a set of abnormal frame images corresponding to the missed detection of the model.

[0127] In one embodiment of this application, optionally, step S209, namely determining at least one tracking state of each tracking target during the tracking process based on the matching state transition data, further includes: when the tracking state transition data changes from an unconfirmed state to a confirmed state to a deleted state, determining whether the number of the first frame of the tracking target is less than a third preset threshold; if the number of the first frame is less than the third preset threshold, determining that there is a tracking anomaly caused by the target detection model during the target tracking process; if the number of the first frame is greater than or equal to the third preset threshold, obtaining the overlap state and target box position of the tracking target in the video frame image of the failed match; determining whether target overlap and / or target exceeding the frame occurs based on the overlap state and target box position; if no target overlap or target exceeding the frame occurs, comparing the number of the first frame with the third preset threshold; if the number of the first frame is less than the third preset threshold, determining the anomaly type as a model false detection; if the number of the first frame is greater than or equal to the third preset threshold, determining the anomaly type as a model false detection; and generating an abnormal frame image set based on the successfully matched video frame images in the matching state transition data.

[0128] In this embodiment, the tracking state transition data is: unconfirmed state (initial tracking state) - confirmed state - deleted state. In this case, the number of successful matches (first frame number) affects the judgment result. Specifically, if a tracked target has few successful matches in the confirmed state, and then the number of consecutive failed matches reaches a threshold leading to the target's deletion, it is inferred that the tracked target likely does not exist; that is, the target detection model has made a false detection during the detection process. Conversely, if a tracked target has many successful matches, and then the number of consecutive failed matches reaches a threshold leading to its deletion, it is inferred that the tracked target exists, and the model has missed detection during the detection process. Therefore, in this tracking state transition scenario, based on the matching state transition data, the number of consecutively successful first frames within the time period of the confirmed state is obtained. If the number of first frames is less than a third preset threshold, it indicates that the number of successful matches within that time period is low, and the tracked target is considered to have been falsely detected during this first time period. The model is considered to be performing normally during the time period from consecutive failed matches to deletion.

[0129] Furthermore, if the number of frames in the first frame is greater than or equal to the third preset threshold, it indicates that there are a relatively high number of successful matches within that time period, and the model is considered to have detected the target normally during that time period. However, a missed detection may occur in the next time period. Specifically, to determine whether the tracking anomaly during this time period is due to a model-induced missed detection or a non-model-related issue, it is necessary to obtain the overlap state of the tracked target at the point of matching failure and the target bounding box position. If the tracked object does not overlap with other targets and does not exceed the edge of the frame, the tracking anomaly during this time period is considered to be caused by a model detection anomaly, i.e., a model-induced missed detection.

[0130] Furthermore, if the tracked object overlaps with other targets or moves out of the frame in a video frame where a match fails, it indicates that the model is performing normally. In other words, the tracking anomaly is not caused by the target detection model, meaning that the abnormal video frame is redundant data for the target detection model. Even if the model is optimized using the abnormal frame, there will be no performance improvement, so there is no need to optimize the target detection model based on the tracking anomaly.

[0131] Furthermore, if the model is confirmed to detect anomalies, and the tracking state transition data shows an unconfirmed state - confirmed state - deleted state, the model may have missed detections or false detections. To clarify the anomaly type, obtain multiple consecutively matched video frame images from the matching state transition data (e.g., Figure 7 (Multiple video frame images corresponding to time period I), count the number of first frames that are successfully matched. If the number of first frames is less than a third preset threshold, it indicates that there are few successfully matched video frames. Therefore, the model is considered to have performed poorly on multiple consecutively matched video frame images (such as...). Figure 7False detections occurred in multiple video frame images corresponding to time period I, such as... Figure 7 During time period II, the absence of detection indicates normal model performance, continuing until the tracked target is deleted. Therefore, it will be as follows: Figure 7 Multiple video frame images in time period I are labeled and extracted, and then summarized to generate a set of abnormal frame images corresponding to the model's false detections.

[0132] Furthermore, if the number of the first frame is greater than or equal to the third preset threshold, it indicates that there are many successfully matched video frames. Therefore, it is assumed that the model detected the target normally in time period I, but subsequently missed detections, causing continuous matching failures until the target is deleted (e.g.). Figure 7 (Multiple video frame images corresponding to time period II). Therefore, multiple video frame images that failed to match consecutively in time period II are marked and extracted, and summarized to generate a set of abnormal frame images corresponding to the missed detections of the model.

[0133] In one embodiment of this application, optionally, step S209, namely determining at least one tracking state of each tracking target during the tracking process based on the matching state transition data, further includes: determining the anomaly type as a model false detection when the tracking state transition data changes from an unconfirmed state to a deleted state; and generating an abnormal frame image set based on the successfully matched video frame images in the matching state transition data.

[0134] In this embodiment, when the tracking state transition data is: unconfirmed state (initial tracking state) - deleted state, starting from the second frame image, multiple consecutive video frames fail to match until the number of failures reaches a threshold and the tracking target is deleted. Therefore, it can be determined that the model has a false detection during the period of consecutive matching failures, that is, there is a tracking anomaly caused by the model in the tracking process.

[0135] Furthermore, if the model detects anomalies and the tracking state transitions from an unconfirmed state to a deleted state, it is confirmed that the model has been missing detections since the second frame. This continues until the number of missed detections reaches a third preset threshold, at which point the target is deleted. Therefore, multiple video frames that fail to match consecutively are marked and extracted, and a set of anomalous frame images resulting from model misdetection is generated.

[0136] Optionally, the third preset threshold can be adaptively adjusted according to the dynamic changes of the target and the background, which is not specifically limited in this application.

[0137] Optionally, it can be determined whether the tracked target exceeds the edge of the screen by detecting the target bounding box in the preset target tracking algorithm. As long as one edge of the target bounding box touches or exceeds the edge of the screen, the tracked target is marked as exceeding the edge of the screen.

[0138] Understandably, during model optimization, the type of anomaly to be optimized in this update (such as model missed detection or model false detection) can be pre-defined, and then the corresponding anomaly frame images in target tracking can be targeted based on the specific anomaly type to be optimized.

[0139] S210. Optimize the target detection model based on the abnormal frame image set and the abnormality type.

[0140] In this step, after determining the anomaly type corresponding to the model detection anomaly and the set of anomaly frame images corresponding to each anomaly type, the set of anomaly frame images is used as training data for model optimization. Based on the anomaly type, the target detection model is optimized in a targeted manner.

[0141] Optionally, since the marked abnormal frames may be from the preceding frames, abnormal frames in offline videos can be extracted using a "global backtracking" method (returning to the beginning of the video and extracting the corresponding abnormal frames based on the marking results), or abnormal frames in online videos can be extracted using a "local backtracking" method (extracting the corresponding abnormal frames from the adjacent video frame buffer queue).

[0142] By optimizing the model for different anomaly types, the accuracy and robustness of the model can be improved, enabling the target detection model to have better stability and reliability when facing different anomalies. This is of great significance for applying the model to target detection and tracking in different environments and scenarios.

[0143] In one embodiment of this application, optionally, in order to improve the accuracy of tracking anomaly analysis, when there are multiple tracking targets, it is possible to count in real time whether each tracking target overlaps with other targets in each video frame image. Specifically, when there are at least multiple tracking targets, the number of target boxes contained in each video frame image is obtained; if the number of target boxes in any video frame image is greater than or equal to two, the position information of each target box in the video frame image is obtained; based on the position information, the cross-union ratio between any two tracking targets in the video frame image is determined; based on the cross-union ratio and a fourth preset threshold, the overlap state of each tracking target is generated.

[0144] In this embodiment, the number of tracked targets is determined. If there are two or more tracked targets, it indicates that multiple target bounding boxes of multiple tracked targets may occlude each other in each frame. Therefore, the position information of each tracked target bounding box in each video frame is obtained, where the position information can be coordinate information. Then, based on multiple position information, the intersection-union ratio (IUGR) between each pair of tracked targets is calculated. Based on the calculated IUGR, if the IUGR is greater than or equal to a fourth preset threshold, the overlap state of the two tracked targets in the video frame is confirmed as yes; if the IUGR is less than the fourth preset threshold, the overlap state of the two tracked targets in the video frame is confirmed as no.

[0145] Optionally, the fourth preset threshold is used to evaluate whether the overlap state is too large. It can be specifically set based on the number of tracked targets and the scene or domain of the video stream. This application does not make specific limitations here.

[0146] In practical applications, let all tracked targets be T(T1, T2, ..., T...). n In each update of the matching state of each frame of image, pairwise discrimination is performed on all tracked targets, that is, for any pair (T) i T j (where i and j are integers from 1 to n), if the intersection-union ratio (IoU) is greater than the threshold, then the target T will be tracked. i T j The overlap state in this video frame image is marked as "yes". It's understandable that target overlap / occlusion can easily lead to target disappearance or missed detection; therefore, updating the overlap state before the target bounding box disappears (i.e., when a match is successful) is more accurate. Therefore, after all tracked targets have been paired, the matching state of each tracked target in each video frame is sequentially confirmed. If a match is successful, the tracking state of that target is correlated with the matching state. Specifically, the coordinate information of the bounding boxes of any pair of tracked targets is determined, the area of ​​the intersection region of the two bounding boxes is calculated, and the area of ​​the union region of the two bounding boxes is also calculated. Then, the intersection area is divided by the union area to obtain the intersection-union ratio (IU).

[0147] Furthermore, as Figure 1 To specifically implement the method, this application provides a model optimization device 300, such as... Figure 8 As shown, the device includes:

[0148] The acquisition module 301 is used to acquire at least one tracking target and a target detection model to be optimized.

[0149] Tracking module 302 is used to track at least one target and acquire matching state transition data corresponding to each target.

[0150] The generation module 303 is used to generate tracking state transition data for each tracking target based on the matching state transition data.

[0151] The determination module 304 is used to determine the anomaly type and the set of anomaly frame images based on the matching state transition data and the tracking state transition data;

[0152] The optimization module 305 is used to optimize the target detection model based on the abnormal frame image set and the abnormality type.

[0153] Optionally, the acquisition module 301 is specifically used to: in response to the model optimization request, acquire multiple video frame images and a target detection model included in the model optimization request; input the multiple video frame images into the target detection model, identify multiple target boxes in the multiple video frame images, and mark each target box in the multiple video frame images; and determine at least one tracking target based on at least one target box marked in the first frame image.

[0154] Optionally, the acquisition module 301 is further configured to: use a preset target tracking algorithm to perform target tracking on each tracked target based on multiple target boxes marked in multiple video frame images, generate a matching state of each tracked target in each video frame image, wherein the matching state includes successful matching or failed matching; and generate matching state transformation data for each tracked target based on multiple matching states.

[0155] Optionally, the generation module 303 is specifically used to: set the initial tracking state of each tracking target in the first frame image to an unconfirmed state; determine at least one tracking state of each tracking target during the tracking process based on the matching state transition data, wherein the tracking state includes a confirmed state, an unconfirmed state, and a deleted state; and generate tracking state transition data for each tracking target based on the initial tracking state and at least one tracking state.

[0156] Optionally, the generation module 303 is further configured to: for any tracking target, determine a first frame number and a second frame number based on the matching state transition data, wherein the first frame number is the number of consecutively successfully matched video frame images, and the second frame number is the number of consecutively unmatched video frame images; if the first frame number is greater than or equal to a first preset threshold, determine that the tracking state of the tracking target corresponding to the first frame number is a confirmed state; if the second frame number is greater than or equal to a second preset threshold when the tracking state of the tracking target is a confirmed state, determine that the tracking state of the tracking target is changed from a confirmed state to an unconfirmed state; if the matching result of the tracking target in the second video frame image is a match failure when the tracking state of the tracking target is an unconfirmed state, determine that the tracking state of the tracking target is changed from an unconfirmed state to a deleted state.

[0157] Optionally, the determining module 304 is specifically used for: for any tracked target, when the tracking state transition data is from an unconfirmed state to a confirmed state, determining whether the matching state in the matching state transition data includes a matching failure; if the matching state includes a matching failure, obtaining the overlap state and target box position of the tracked target in the video frame image of the matching failure; determining whether target overlap or target exceeding the frame occurs based on the overlap state and target box position; if no target overlap or target exceeding the frame occurs, determining the anomaly type as model missed detection; and generating an abnormal frame image set based on the video frame image of the matching failure in the matching state transition data.

[0158] Optionally, the determining module 304 is further configured to, when the tracking state transition data changes from an unconfirmed state to a confirmed state to a deleted state, determine whether the number of the first frame of the tracked target is less than a third preset threshold; if the number of the first frame is less than the third preset threshold, determine that there is a tracking anomaly caused by the target detection model during the target tracking process; if the number of the first frame is greater than or equal to the third preset threshold, obtain the overlap state and target box position of the tracked target in the video frame image of the failed match; determine whether target overlap and / or target exceeding the frame occurs based on the overlap state and target box position; if no target overlap or target exceeding the frame occurs, compare the number of the first frame with the third preset threshold; if the number of the first frame is less than the third preset threshold, determine that the anomaly type is a model false detection; if the number of the first frame is greater than or equal to the third preset threshold, determine that the anomaly type is a model false detection; and generate an abnormal frame image set based on the successfully matched video frame images in the matching state transition data.

[0159] Optionally, the determination module 304 is further configured to: determine the anomaly type as a model false detection when the tracking state transition data is a transition from an unconfirmed state to a deleted state; and generate an anomaly frame image set based on the successfully matched video frame images in the matching state transition data.

[0160] Optionally, the acquisition module 301 is further configured to acquire the position information of the target box corresponding to each tracking target in each video frame image when there are multiple tracking targets;

[0161] Optionally, the determining module 304 is further configured to determine, based on multiple location information, the intersection-over-union ratio between any two tracked targets in each video frame image;

[0162] The generation module 303 is also used to generate the overlap state of each tracked target for each video frame image based on the cross-union ratio and a fourth preset threshold.

[0163] The model optimization device 300 provided in this application monitors the matching state of each tracked target in each frame during continuous tracking, obtains the matching state transition status, and then obtains the tracking state transition status for different time periods. Subsequently, the transition status is analyzed to identify anomalies caused by the model, determine the anomaly type of the target detection model and its corresponding anomaly frames, and then optimize the target detection model in a targeted manner based on the anomaly type and anomaly frames. Compared to the prior art, which extracts frames where detection and tracking do not match as video frames with unsatisfactory model processing performance and retrains the model based on the extracted video frames for optimization, this method has the technical problem of not being able to optimize the model in a targeted manner according to the anomaly type, having a large amount of redundant data, and ultimately leading to poor optimization performance of the target detection model. This application determines whether tracking anomalies are caused by model detection errors. Only when it is determined that the tracking anomaly is caused by a model detection error will the anomaly type and anomaly frame be determined based on the transformation situation to optimize the target detection model. This maximizes the extraction of effective anomaly frames related to the target detection model as training data, reduces redundant data, and optimizes the model in a targeted manner based on the anomaly type, resulting in better optimization performance. It also makes the target detection model more stable and reliable when facing different anomaly situations, which is of great significance for applying the model to target detection and tracking in different environments and scenarios.

[0164] In an exemplary embodiment, this application also provides a video processing system including: a model optimization device as described in the second aspect; and a video acquisition device, which is communicatively connected to the model optimization device, wherein the video acquisition device is used to acquire a video stream and send the acquired video stream to the model optimization device.

[0165] In this embodiment, the video capture device can be a traditional camera, a network camera (IP camera), a motion capture system, or a drone, etc., used to capture video streams and provide high-quality real-time video. It is understood that after capturing the video stream, the video capture device can send the online video (camera preview stream) to the model optimization device in real time for model optimization. Furthermore, the captured video stream can also be stored in a preset database. When model optimization is needed, the stored offline video is sent to the model optimization device.

[0166] In an exemplary embodiment, this application also provides a storage medium storing at least one executable instruction that can execute the model optimization method in any of the above method embodiments.

[0167] In an exemplary embodiment, this application also provides a terminal, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0168] The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the model optimization method of the first aspect.

[0169] Figure 9 The diagram shows a structural schematic of a terminal according to an embodiment of the present invention. The specific implementation of the terminal is not limited by the specific embodiments of the present invention.

[0170] like Figure 9 As shown, the terminal may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0171] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0172] Communication interface 404 is used to communicate with other network elements such as clients or other servers.

[0173] The processor 402 is used to execute program 410, specifically to perform the relevant steps in the above-described module optimization method embodiment.

[0174] Specifically, program 410 may include program code that includes computer operation instructions.

[0175] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The terminal may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0176] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0177] Program 410 can be specifically used to cause processor 402 to execute Figure 2The method for executing the module optimization device, and the implementation of the module optimization device in... Figure 2 The functions of the embodiments shown are not described again in this application.

[0178] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0179] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A model optimization method, characterized by, The method comprises the following steps: acquiring at least one tracking target and a target detection model to be optimized; tracking the at least one tracking target to acquire matching state transition data corresponding to each tracking target; generating tracking state transition data corresponding to each tracking target according to the matching state transition data; determining an abnormal type and an abnormal frame image set corresponding to the target detection model according to the matching state transition data and the tracking state transition data; optimizing the target detection model according to the abnormal frame image set and the abnormal type; the step of tracking the at least one tracking target to acquire matching state transition data corresponding to each tracking target comprises the following steps: tracking each tracking target based on a plurality of target boxes marked in a plurality of video frame images by using a preset target tracking algorithm to generate a matching state of each tracking target in each video frame image, wherein the matching state comprises matching success or matching failure; generating the matching state transition data of each tracking target according to a plurality of matching states; the step of generating tracking state transition data corresponding to each tracking target according to the matching state transition data comprises the following steps: setting an initial tracking state of each tracking target in a first frame image as an unconfirmed state; determining at least one tracking state of each tracking target in a tracking process according to the matching state transition data, wherein the tracking state comprises a confirmed state, an unconfirmed state and a deleted state; generating tracking state transition data of each tracking target according to the initial tracking state and the at least one tracking state.

2. The method of claim 1, wherein, the step of acquiring at least one tracking target and a target detection model to be optimized comprises the following steps: in response to a model optimization request, acquiring a plurality of video frame images and the target detection model included in the model optimization request; inputting the plurality of video frame images into the target detection model to identify a plurality of target boxes in the plurality of video frame images and mark each target box in the plurality of video frame images; determining the at least one tracking target according to at least one target box marked in a first frame image.

3. The method of claim 1, wherein, the step of determining at least one tracking state of each tracking target in a tracking process according to the matching state transition data comprises the following steps: for any tracking target, determining a first frame number and a second frame number according to the matching state transition data, wherein the first frame number is the number of continuously matching successful video frame images, and the second frame number is the number of continuously matching failed video frame images; if the first frame number is greater than or equal to a first preset threshold, determining that the tracking state of the tracking target in the first frame number corresponding state is a confirmed state; if the second frame number is greater than or equal to a second preset threshold in the case that the tracking state of the tracking target is a confirmed state, determining that the tracking state of the tracking target is converted from the confirmed state to an unconfirmed state; In a case where the tracking state of the tracking target is the unconfirmed state, if the matching result of the tracking target in the second video frame image is a matching failure, it is determined that the tracking state of the tracking target is converted from the unconfirmed state to the deletion state.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the abnormal type and the abnormal frame image set corresponding to the target detection model according to the matching state transition data and the tracking state transition data specifically comprises: For any tracking target, in a case where the tracking state transition data is converted from the unconfirmed state to the confirmed state, it is judged whether the matching state in the matching state transition data contains a matching failure; If the matching state contains a matching failure, the overlapping state and the target box position of the tracking target in the video frame image of the matching failure are obtained; According to the overlapping state and the target box position, it is judged whether target overlapping or target exceeding the picture occurs; If target overlapping or target exceeding the picture does not occur, it is determined that the abnormal type is model missed detection; According to the video frame image of the matching failure in the matching state transition data, the abnormal frame image set is generated.

5. The method according to any one of claims 1 to 3, characterized in that, The step of determining the abnormal type and the abnormal frame image set corresponding to the target detection model according to the matching state transition data and the tracking state transition data specifically further comprises: In a case where the tracking state transition data is converted from the unconfirmed state to the confirmed state to the deletion state, it is judged whether the first frame number of the tracking target is less than a third preset threshold value; If the first frame number is less than the third preset threshold value, it is determined that there is a tracking abnormality caused by the target detection model in the target tracking process; If the first frame number is greater than or equal to the third preset threshold value, the overlapping state and the target box position of the tracking target in the video frame image of the matching failure are obtained; According to the overlapping state and the target box position, it is judged whether target overlapping and / or target exceeding the picture occurs; If target overlapping and target exceeding the picture do not occur, the first frame number is compared with the third preset threshold value; If the first frame number is less than the third preset threshold value, it is determined that the abnormal type is model false detection; If the first frame number is greater than or equal to the third preset threshold value, it is determined that the abnormal type is model false detection; According to the video frame image of the matching success in the matching state transition data, the abnormal frame image set is generated.

6. The method according to any one of claims 1 to 3, characterized in that, The step of determining the abnormal type and the abnormal frame image set corresponding to the target detection model according to the matching state transition data and the tracking state transition data specifically further comprises: In a case where the tracking state transition data is converted from the unconfirmed state to the deletion state, it is determined that the abnormal type is model false detection; According to the video frame image of the matching success in the matching state transition data, the abnormal frame image set is generated.

7. The method according to any one of claims 1 to 3, characterized in that, Further comprising: In a case where the at least one tracking target is multiple, the position information of the target box corresponding to each tracking target in each video frame image is obtained; According to the multiple position information, the intersection over union between any two tracking targets in each video frame image is determined; According to the intersection over union and a fourth preset threshold value, the overlapping state of each tracking target for each video frame image is generated.

8. A model optimization apparatus characterized by comprising: Comprise: An acquisition module is configured to acquire at least one tracking target and a target detection model to be optimized. A tracking module is configured to perform target tracking on the at least one tracking target, and acquire matching state transition data corresponding to each tracking target. A generation module is configured to generate tracking state transition data corresponding to each tracking target according to the matching state transition data. A determination module is configured to determine an abnormal type and an abnormal frame image set according to the matching state transition data and the tracking state transition data. An optimization module is configured to optimize the target detection model according to the abnormal frame image set and the abnormal type. The tracking module is specifically configured to perform target tracking on each tracking target based on a plurality of target boxes marked in a plurality of video frame images by using a preset target tracking algorithm, and generate a matching state of each tracking target in each video frame image, wherein the matching state includes matching success or matching failure; and generate matching state transition data of each tracking target according to a plurality of matching states. The generation module is specifically configured to set an initial tracking state of each tracking target in a first frame image as an unconfirmed state; determine at least one tracking state of each tracking target in a tracking process according to the matching state transition data, wherein the tracking state includes a confirmed state, an unconfirmed state and a deletion state; and generate tracking state transition data of each tracking target according to the initial tracking state and the at least one tracking state.

9. A video processing system characterized by Comprise: The model optimization device of claim 8; A video acquisition device, which is in communication connection with the model optimization device, wherein the video acquisition device is configured to acquire a video stream and send the acquired video stream to the model optimization device. 10.A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to perform operations corresponding to the model optimization method of any one of claims 1-7.

11. A terminal comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the model optimization method of any one of claims 1-7.

Images