Video processing methods, apparatus, computer equipment and storage media

Abstract

This application relates to a video processing method, a video processing apparatus, a computer device, and a computer-readable storage medium. The method includes: acquiring a target video frame from a target video stream; processing the target video frame to obtain target information, including target position and confidence level; predicting the predicted position of each historical target based on historical target information from the target video stream; matching the predicted position of each historical target with the target position of each target based on the confidence level of each target, and determining the tracking target in the target video frame based on the matching result; determining the display position of the tracking target in the target video frame, and outputting the target video frame. This method enables the tracking and display of targets in a video stream.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video processing method, a video processing apparatus, a computer device, and a computer-readable storage medium. Background Technology

[0002] During video display, there are often situations where it is necessary to track and display targets within the video stream. For example, in thyroid ultrasound examinations, thyroid nodules are easily confused with other neck structures due to their unclear and blurry ultrasound image boundaries, making them difficult to identify. Currently, the identification and analysis of thyroid nodules rely solely on the experience and subjective judgment of the sonographer, which can easily lead to missed diagnoses or misdiagnoses, thus affecting the screening of thyroid nodules and related structural lesions. Clinical practice urgently requires the automatic detection and tracking display of organs and surrounding tissues within ultrasound video frames to assist sonographers in disease screening. Therefore, a processing method capable of tracking and displaying targets within the video stream is urgently needed. Summary of the Invention

[0003] Therefore, it is necessary to provide a video processing method, video processing device, computer equipment, and computer-readable storage medium that can realize target tracking and display in video frames to address the above-mentioned technical problems.

[0004] Firstly, this application provides a video processing method. The method includes:

[0005] Acquire the target video frame of the target video stream, process the target video frame, and obtain the target information of the target video frame, the target information including the target location and confidence level;

[0006] Based on the historical target information of the target video stream, predict the predicted position of each historical target;

[0007] Based on the confidence level of each target, the predicted position of each historical target is matched with the target position of each target, and the tracking target in the target video frame is determined according to the matching result;

[0008] Determine the display position of the tracking target in the target video frame and output the target video frame.

[0009] In one embodiment, the step of matching the predicted positions of the historical targets with the target positions of the targets based on the confidence levels of the targets, and determining the tracking targets in the target video frames based on the matching results, includes:

[0010] Based on the confidence level of each target, the predicted position of each historical target is matched with the target position of each target to obtain the matching result;

[0011] The target that is successfully matched in the matching results is identified, and the tracking target in the target video frame includes the target that is successfully matched.

[0012] In one embodiment, the tracking target in the target video frame further includes: among the targets with a confidence level greater than a first confidence threshold, the unmatched target.

[0013] In one embodiment, the method further includes:

[0014] Delete targets to be cleaned from the tracked targets. The targets to be cleaned include: newly created tracked targets in the adjacent previous video frame based on targets with a confidence level greater than a second confidence threshold, whose predicted positions do not match the target positions of the targets in the target video frame; and newly created tracked targets in the adjacent previous two video frames based on targets with a confidence level less than a second confidence threshold but greater than a first confidence threshold, whose predicted positions do not match the target positions of the targets in the target video frame.

[0015] In one embodiment, determining the display position of the tracking target in the target video frame and outputting the target video frame includes:

[0016] Based on the tracking results of each video frame of each tracking target within a sliding window of a preset length, determine whether it is necessary to add a display position for the corresponding video frame;

[0017] When it is necessary to add a display position, add a display position for the corresponding video frame;

[0018] The target video frame is output and displayed after a delay of the preset length.

[0019] In one embodiment, determining whether to add a display position for the corresponding video frame based on the tracking results of each video frame within a sliding window of a preset length for each tracking target includes:

[0020] If the tracking target is present in all of the first number of video frames preceding the target video frame, and there are video frames without the tracking target between the video frames with the tracking target in the sliding window of the preset length, it is determined that a display position needs to be added for the video frames without the tracking target.

[0021] In one embodiment, the method further includes:

[0022] In video frames where the tracked target is not present, the predicted position determined for the video frame is added to the corresponding position of the video frame to serve as the display position for the added video frame.

[0023] Secondly, this application also provides a video processing apparatus. The apparatus includes:

[0024] The processing module is used to acquire the target video frame of the target video stream, process the target video frame, and obtain the target information of the target video frame, the target information including the target location and confidence level;

[0025] The prediction module is used to predict the predicted position of each of the historical targets based on the historical target information of the target video stream.

[0026] The matching module is used to match the predicted position of each historical target with the target position of each target based on the confidence level of each target, and to determine the tracking target in the target video frame according to the matching result;

[0027] The display module is used to determine the display position of the tracking target of the target video frame and output the target video frame.

[0028] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0029] Acquire the target video frame of the target video stream, process the target video frame, and obtain the target information of the target video frame, the target information including the target location and confidence level;

[0030] Based on the historical target information of the target video stream, predict the predicted position of each historical target;

[0031] Based on the confidence level of each target, the predicted position of each historical target is matched with the target position of each target, and the tracking target in the target video frame is determined according to the matching result;

[0032] Determine the display position of the tracking target in the target video frame and output the target video frame.

[0033] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0034] Acquire the target video frame of the target video stream, process the target video frame, and obtain the target information of the target video frame, the target information including the target location and confidence level;

[0035] Based on the historical target information of the target video stream, predict the predicted position of each historical target;

[0036] Based on the confidence level of each target, the predicted position of each historical target is matched with the target position of each target, and the tracking target in the target video frame is determined according to the matching result;

[0037] Determine the display position of the tracking target in the target video frame and output the target video frame.

[0038] The aforementioned video processing method, video processing apparatus, computer equipment, and computer-readable storage medium first acquire the target video frame of the target video stream, process the target video frame to obtain target information, including target position and confidence level; predict the predicted position of each historical target based on historical target information of the target video stream; match the predicted position of each historical target with the target position of each target based on the confidence level of each target, and determine the tracking target in the target video frame based on the matching result; determine the display position of the tracking target in the target video frame, and output the target video frame. Thus, by processing the target video frame, the target position and confidence level of each target in the target video frame are obtained, and the predicted position of each historical target is predicted based on historical information obtained from historical frames. Furthermore, based on the matching result of the confidence level of each target, the target position, and the predicted position, a more reliable tracking target is determined. Finally, the display position of the tracking target in the target video frame is determined, and when the target video frame is output, the display position corresponds to the display of the tracking target, thereby realizing the tracking and display of the tracking target in the video stream. Attached Figure Description

[0039] Figure 1 This is an application environment diagram of a video processing method in one embodiment;

[0040] Figure 2 This is a flowchart illustrating a video processing method in one embodiment;

[0041] Figure 3 This is a schematic diagram of a process in one embodiment where the predicted position of each historical target is matched with the target position of each target based on the confidence level of each target, and the tracking target in the target video frame is determined according to the matching result.

[0042] Figure 4This is a flowchart illustrating the process of determining the display position of the tracking target in a target video frame and outputting the target video frame in another embodiment.

[0043] Figure 5 This is an artificially annotated diagram of the thyroid ultrasound structure.

[0044] Figure 6 This is a flowchart illustrating a video processing method in another embodiment;

[0045] Figure 7 Comparison of thyroid ultrasound structures in an ultrasound video stream, with manual annotation and with tracking using this video processing method;

[0046] Figure 8 This is a structural block diagram of a video processing device in one embodiment;

[0047] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] The video processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 acquires target video frames from the target video stream, processes the target video frames to obtain target information, including target location and confidence level; predicts the predicted location of each historical target based on historical target information from the target video stream; matches the predicted location of each historical target with the target location of each target based on the confidence level of each target, and determines the tracking target in the target video frame based on the matching result; determines the display position of the tracking target in the target video frame, and outputs the target video frame. Terminal 102 can be, but is not limited to, various personal computers, laptops, and tablets. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0050] The target video stream can include, but is not limited to, ultrasound detection video streams. The server processes the ultrasound detection video stream acquired by the ultrasound probe and outputs target video frames. The tracking targets in the target video frames include at least one organ, its surrounding tissues, and any related structural lesions (such as nodules, tumors, etc.). The final output target video frame displays the display position of the tracking targets. The display position can exist in the form of a position box, and information such as the type of the tracking target can also be displayed at the appropriate position of the position box. For example, the target video stream is the video stream of the thyroid gland and surrounding tissues acquired by the ultrasound probe during an ultrasound examination of the thyroid gland. Through this video processing method, the detection and tracking of the thyroid gland, thyroid nodules, and surrounding tissues (such as trachea, esophagus, veins, arteries, etc.) in the ultrasound video can be realized. When the target video frame is output, each position box corresponds to a box around the thyroid gland, thyroid nodules, trachea, etc., and the category of the tracking target in each position box (as shown above) is displayed at the appropriate position of that position box, such as "thyroid gland", "thyroid nodule", "trachea", etc.

[0051] The target video stream can also include surveillance video streams captured by surveillance cameras. The server processes the surveillance video streams and outputs target video frames. The tracked targets in the target video frames can include specific or unspecified people, pets, vehicles, etc. The final output target video frame shows the display position of the tracked targets. The display position can also exist in the form of a position box, and information such as the type of tracked target can be displayed at an appropriate position in the position box.

[0052] The target video stream can also include a network video stream. The server processes the network video stream and outputs a target video frame. The tracking target in the target video frame can include specific or non-specific objects from the network video frame. The final output target video frame displays the display position of the tracking target. The display position can also exist in the form of a position box, and information such as the type of tracking target can be displayed at an appropriate position within the position box.

[0053] In one embodiment, such as Figure 2 As shown, a video processing method is provided, which can be applied to... Figure 1 Taking a server as an example, it can be understood that this method can also be applied to a terminal, and also to a system that includes both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 200-500.

[0054] Step 200: Obtain the target video frame of the target video stream, process the target video frame to obtain the target information of the target video frame, including the target location and confidence level.

[0055] The target video stream can include ultrasound detection video streams, surveillance video streams, and network video streams, etc. The target video frame can be the current frame or a historical frame that is a certain number of frames away from the current frame. For ease of explanation, the following explanation will use the example of a thyroid ultrasound examination video stream as the target video stream and the current frame as the target video frame.

[0056] Processing the target video frame yields the target (one or more) and its information, including target location and confidence level. Further information may also include target category and characteristics. Multiple targets can be selected, and each target category can contain one or more targets. Specifically, in an ultrasound examination of the thyroid gland, target categories include the thyroid gland, thyroid nodules, trachea, esophagus, neck muscles, arteries, and veins. A single target of the thyroid nodule category can be selected.

[0057] Step 300: Based on the historical target information of the target video stream, predict the predicted position of each historical target.

[0058] Historical targets refer to targets in all video frames preceding the target video frame, or targets in a certain number of video frames preceding the target video frame. Historical target information can include the target's location, characteristics, category, and confidence level in all video frames preceding the target video frame or in a certain number of video frames preceding the current frame. Predicting the predicted position of each historical target means predicting the position of each historical target in the current frame based on its historical information.

[0059] Furthermore, a tracker can be created for each historical target. Each tracker uses Kalman filtering to predict the predicted position of the corresponding target in the current frame based on the historical position and target features in the historical target information. When there are multiple historical targets, all predicted positions can be combined into a predicted target position set.

[0060] Step 400: Based on the confidence level of each target, match the predicted position of each historical target with the target position of each target, and determine the tracking target in the target video frame according to the matching result.

[0061] Targets can be selected for selection based on their confidence level and a preset selection threshold. For example, if the preset selection threshold is set to 0.1, targets with a confidence level higher than 0.1 are retained, while targets with a confidence level lower than 0.1 are discarded. The positions of the retained targets are then matched with the predicted positions of historical targets. The tracked targets in the target video frame include all successfully matched targets from the matching results.

[0062] Furthermore, the tracked targets in the target video frame can also include unmatched targets with relatively high confidence.

[0063] Step 500: Determine the display position of the tracking target in the target video frame and output the target video frame.

[0064] When the target video frame is the current frame, the display position of the tracked target can be determined in the current frame, or the display position of the tracked target in the current frame can be determined after a certain number of frames based on the matching situation of the tracked target in the subsequent frames, and the current frame can be output and displayed after a preset number of frames.

[0065] When outputting target video frames, a corresponding marker can be set at the display position of the tracked target. The marker can take various forms, such as using a bounding box to enclose the tracked target, or using a different color at the display position than other areas. Depending on the needs, the marker can also display information such as the type and coordinates of the tracked target.

[0066] The aforementioned video processing method processes the target video frame to obtain the target position and confidence level of each target within the frame. Based on historical information obtained from historical frames, it predicts the positions of historical targets. Then, based on the matching results of the confidence level, target position, and predicted position, it determines the most reliable tracking target. Finally, it determines the display position of the tracking target within the target video frame and displays the corresponding tracking target according to its position when outputting the target video frame. This achieves target tracking and display within the video stream. During ultrasound examinations, it enables automatic detection and tracking of organs and lesions in the ultrasound video, aiding ultrasound physicians in disease screening and diagnosis, reducing their workload, and improving the consistency of ultrasound diagnoses.

[0067] In one embodiment, such as Figure 3 As shown, step 400 includes steps 410 and 420.

[0068] Step 410: Based on the confidence level of each target, match the predicted location of each historical target with the target location of each target to obtain the matching result.

[0069] After selecting targets based on their confidence levels and preset selection thresholds, the remaining target locations are matched with the predicted locations of historical targets. In some embodiments, a match can be determined based on the IoU (Intersection over Union) distance between the target location and the predicted locations of historical targets, combined with the category of the target and the historical target. It should be understood that other matching methods may also be used in other embodiments.

[0070] Specifically, firstly, the IoU (Intersection over Union) distance between the predicted locations and each target location in the predicted target location set composed of all predicted locations is calculated. Then, the Hungarian algorithm is used for matching. If the IoU distance between the target location and the predicted location is greater than a preset IoU threshold, and the target corresponding to the target location is of the same category as the historical target corresponding to the predicted location, and the Hungarian algorithm matches successfully, then it is considered a successful match. If the target and the historical target are of the same category, but the IoU distance between the target location and the predicted location of the historical target is not greater than the preset IoU threshold, then it is considered a failed match. If the IoU distance between the target location and the predicted location of the historical target is greater than the preset IoU threshold, but the target and the historical target are of different categories, then it is also considered a failed match.

[0071] Furthermore, since targets with different confidence levels have different levels of reliability, different IoU thresholds can be set during matching based on the target's confidence level. For example, a relatively low IoU threshold can be used when matching targets with relatively high confidence levels, while a relatively high IoU threshold can be used when matching targets with relatively low confidence levels. This makes the matching more targeted and more reliable.

[0072] Step 420: Determine the successfully matched targets in the matching results. The tracked targets in the target video frame include the successfully matched targets.

[0073] The matching results include successful matches and unsuccessful matches. Specifically, it includes successfully matched targets, historically successfully matched targets, unsuccessfully matched targets, and historically unsuccessfully matched targets. Through the matching process, successfully matched targets are identified as the tracking targets in the target video frame, improving the reliability of the tracking targets and resulting in higher tracking accuracy.

[0074] In one embodiment, step 410 includes: matching the predicted positions of each historical target with the target positions of each target whose confidence level is greater than a first matching threshold to obtain a first matching result; matching the unmatched predicted positions of each historical target with each target whose confidence level is less than the first matching threshold and greater than a second matching threshold to obtain a second matching result; the matching result includes the first matching result and the second matching result.

[0075] In this embodiment, during the matching process, two matching operations are performed based on the different confidence levels of the targets. In the first matching operation, the first IoU distance between the target position and the predicted position of each target with a confidence level greater than a first matching threshold is calculated. Then, the Hungarian algorithm is used for matching. If the first IoU distance is greater than a first preset IoU threshold, and the target and the historical target corresponding to the predicted position are of the same category, and the Hungarian algorithm matches successfully, then the matching is considered successful. If the first IoU distance is not greater than the first preset IoU threshold, or the target and the historical target are of different categories, then the matching is considered unsuccessful. In the second matching operation, the second IoU distance between the unmatched predicted position of each historical target and each target with a confidence level less than the first matching threshold but greater than the second matching threshold is calculated. The second IoU distance between these targets and the predicted position is calculated to see if it is greater than a second preset IoU threshold, and whether the categories are the same. If the second IoU distance is greater than the second preset IoU threshold, and the target and the historical target are of the same category, then the matching is considered successful; otherwise, the matching is considered unsuccessful.

[0076] The first matching threshold can be greater than the second matching threshold, thus performing a first matching on targets with higher confidence and a second matching on targets with lower confidence. For example, if the first matching threshold is 0.5 and the second matching threshold is 0.1, then all targets with relatively high confidence (a first confidence level greater than 0.5) are matched first, and then targets with lower confidence (between 0.1 and 0.5) are matched. Targets with confidence levels below 0.1 are not matched, as their reliability is low and they can be directly discarded. Furthermore, the first preset IoU threshold can be lower than the second preset IoU threshold, and the values ​​of the first and second preset IoU thresholds can be set according to actual conditions.

[0077] In one embodiment, the tracking results of the trackers corresponding to each historical target are updated based on the matching results. The tracking results of the trackers include status, position, and hit parameters. Each tracker's tracking result includes the target position, tracking number, and a hit queue of a preset length for each frame. The tracking queue stores the hit parameters for the number of frames corresponding to the preset length. When a historical target corresponding to a tracker is successfully matched, the tracker's state for that frame is set to the active state, and the position in the tracker's tracking result is updated with the position of the successfully matched target. The hit parameters for that frame in the hit queue can be set to the corresponding hit parameter value (e.g., set to 1). When a historical target corresponding to a tracker is not successfully matched, the tracker's state for that frame is set to the missing state, and the hit parameters for that frame in the hit queue can be set to the corresponding miss parameter value (e.g., set to 0).

[0078] In one embodiment, the tracking targets in the target video frame further include: targets that were not successfully matched among those targets with a confidence level greater than a first confidence threshold.

[0079] For targets that fail to match, a new tracker is created based on their confidence level. Specifically, a new tracker is created for targets with a confidence level greater than a first confidence threshold. The state of the newly created tracker is set to the initial state, and its position is updated using the target's location. The hit parameter for that frame is set to the parameter value corresponding to the miss. The first confidence threshold can be set according to actual needs. For example, if it is 0.3, then a new tracker will be created for targets with a confidence level greater than 0.3 among the unmatched targets, while targets with a confidence level lower than 0.3 will be discarded.

[0080] In this embodiment, the tracked targets in the target video frame include both successfully matched targets and targets with high confidence among the unmatched targets. Although these targets are not successfully matched in the target video frame, their reliability is high and they still cannot be directly discarded.

[0081] To better understand, let's assume the first confidence threshold is 0.5, the second confidence threshold is 0.1, the first preset IoU threshold is set to 0.4, and the second preset IoU threshold is set to 0.5.

[0082] Targets with a confidence level greater than 0.5 can first be placed into the high-resolution detection box set D. high In the set of low-score detection boxes, those with confidence levels between 0.1 and 0.5 are placed into the set D. low In the first match: First, calculate D. high The distance between the target location and the predicted location in the data. IOU If the categories are equal and dist IOU If the value is greater than 0.4 and the Hungarian algorithm matches successfully, then the match is considered successful. The tracker's state and position are updated with the target position of the successfully matched target, and then the target is added to the queue. hit The i-th bit (corresponding to the target video frame) is set to 1, indicating that the tracker has hit the target in the i-th frame. After the first match, the target positions that did not match are added to set D. remain In the middle, trackers that did not match are grouped into set T. remain In the middle. For the high-confidence target set D where the first match failed. remain Reinitialize the tracker with the target location.

[0083] Then, the remaining trackers T after the first match remain With low-score detection box set D low Perform a secondary matching; if there is an IoU distance (dist) between the target location and the tracker, then... IOU If the score is greater than 0.5 and the categories are equal, then a successful match is considered. Since the target location here belongs to the low-scoring detection box set D... lowTargets with a confidence level below 0.3 that fail to match are discarded and treated as false bounding boxes. Targets with a confidence level above 0.3 are used to reinitialize the tracker with their target locations.

[0084] Set the status of unmatched trackers to the lost state, and set their hit queue (Queue) to the missing state. hit The i-th bit is set to 0, indicating a miss. When a tracker is in a missing state for n consecutive frames (n frames can be set according to actual needs, such as 15), it is removed from the tracker list.

[0085] In one embodiment, the video processing method further includes: deleting targets to be cleaned from the tracking targets, the targets to be cleaned including: newly created tracking targets whose predicted positions do not match the target positions of each target in the target video frame among newly created tracking targets based on targets with confidence levels greater than a second confidence threshold in an adjacent previous video frame; and newly created tracking targets whose predicted positions do not match the target positions of each target in the target video frame among newly created tracking targets based on targets with confidence levels less than a second confidence threshold and greater than a first confidence threshold in two adjacent previous video frames.

[0086] The second confidence threshold is greater than the first confidence threshold; for example, the first confidence threshold is set to 0.3, and the second confidence threshold is set to 0.5. It can be understood that targets with a confidence level greater than 0.5 that fail to match are inherently highly reliable. Therefore, when the predicted position of a newly created tracker matches successfully in the next frame, it indicates that the target is trustworthy and can be retained. At this point, the tracker's state changes from the initial state to the active state, and the hit parameters of the tracker in both the previous frame (the target video frame) and the current frame are set to the corresponding hit parameter values ​​(i.e., the hit queue). hit The i-th bit (target video frame) and the (i+1)-th bit are set to 1. If no match is found, it means that the target is untrusted. The tracker is set to the delete state, removed from the tracker set, and the target is removed from the tracking targets of the target video frame.

[0087] For targets with a confidence level between 0.3 and 0.5 that fail to match, their reliability is not high. Therefore, it is determined whether the predicted positions of the newly built tracker based on the target match successfully in the next two frames. Only if both matches are successful is the target considered reliable, and the corresponding tracker is set to the active state. The hit parameters of this tracker in the first two frames are set to the corresponding hit parameter values ​​(i.e., the hit queue). hit (The i-th, i+1-th, and i+2-th bits are set to 1). If no consecutive match is found in the next two frames, it indicates that the target is unreliable. The tracker is then set to the delete state, removed from the tracker set, and the target is removed from the tracking targets in the target video frame.

[0088] It is understood that in this embodiment, the threshold for creating a new tracker is essentially divided into 0.5 and 0.3. If the confidence level of the target corresponding to the newly created tracker is greater than 0.5, an aggressive display strategy and frame interpolation strategy will be adopted for the newly created tracker. If the confidence level of the target corresponding to the newly created tracker is between 0.5 and 0.3, a conservative display strategy and frame interpolation strategy will be adopted for the newly created tracker.

[0089] Specifically, to avoid false detections, the tracking target and display position displayed in the i-th frame are usually confirmed only after the (i+K)-th frame has been detected. In the i-th frame, after a new tracker is created, the tracker is in an inactive initial state and is not displayed in the tracker results. Only after K consecutive frames are matched will the tracker be set to an active state and the trajectory be displayed in the video results. For trackers generated by high-confidence targets, the value of K is 1, and for trackers generated by low-confidence targets, the value of K is 2.

[0090] The aggressive display and frame interpolation strategy is as follows: In the i-th frame, after creating a new tracker, the target's position coordinates in the i-th frame are cached in the pcache (page cache). Until the (i+1)-th frame, when the tracker's predicted position matches the target's position again, the tracker is activated and its trajectory is displayed in the video. That is, the detection boxes that were not originally displayed in the i-th frame are restored to the display. Using the previously cached pcache, the frame interpolation function in the sliding window is used to supplement the detection boxes in the i-th frame. This improves the tracking integrity of high-confidence detection trajectories and reduces the impact of the tracking algorithm on the high-confidence detection results.

[0091] The conservative display and frame interpolation strategies are as follows: In frame i, a new tracker is created. The tracker is activated and its trajectory is displayed in the video only if it successfully matches in frames i+1 and i+2 consecutively. Specifically, after a successful match in frame i+1, the current detection box is placed in the pcache cache. The detection boxes that were not displayed in frames i and i+1 are restored to the display only if a successful match is achieved in frame i+2.

[0092] For both confidence levels, consecutive hits are required for display and frame interpolation operations, which improves the tracking integrity of the detection trajectory and enhances the continuity of the display.

[0093] In one embodiment, such as Figure 4 As shown, step 500 includes steps 510, 520 and 530.

[0094] Step 510: Based on the tracking results of each video frame within a sliding window of a preset length for each tracking target, determine whether it is necessary to add a display position for the corresponding video frame.

[0095] The length of the sliding window can be set according to the detection and tracking needs. The tracking result is specifically the matching result (hit parameter value) of each target in each video frame within the sliding window. For example, if the preset length is 8 frames and the target video frame is the current frame, by tracking the matching of the target in these 8 consecutive frames, it can be determined whether there is a display position in these 8 frames where the target needs to be added for tracking.

[0096] In one embodiment, step 510 includes: if a tracking target exists in all of the first number of video frames preceding the target video frame, and there are video frames without tracking targets between the video frames with tracking targets within a sliding window of a preset length, then it is determined that a display position needs to be added for the video frames without tracking targets.

[0097] The first number can also be set according to the needs of detection and tracking. For example, if the first number is set to 2 and the tracking target is a thyroid nodule, the thyroid nodule exists in the i-th frame (current frame) and the i-1-th frame (previous frame) (successful match), does not exist in the i-2-th frame (unsuccessful match), but exists in both the i-3-th and i-4-th frames. This means that it is missing in the i-2-th frame, and the display position of the thyroid nodule in the i-2-th frame should be supplemented.

[0098] Step 520: If a display position needs to be added, add a display position for the corresponding video frame.

[0099] The specific method of adding a display position for the tracked target in the corresponding video frame may include: adding the predicted position determined for the video frame to the corresponding position of the video frame in a video frame where no tracked target exists, so as to use it as the added display position for the video frame.

[0100] The corresponding video frame is the video frame for which the display position needs to be added. It can be understood that the tracking target exists in at least the previous frame of the corresponding video frame. Therefore, the tracker corresponding to the tracking target will predict the predicted position of the tracking target in the corresponding video frame, and then use the predicted position to perform frame interpolation operation on the video frame, that is, use the predicted position as the display position of the tracking target in the corresponding video frame.

[0101] Step 530: Output and display the target video frame after a preset delay.

[0102] The delay length of the target video frame is equal to the length of the sliding window, which ensures the accurate determination of the display position in each video frame within this length. This makes the target tracking effect smoother and more complete during video playback, while also meeting the real-time requirements of video playback.

[0103] In a detailed embodiment, the step of performing frame interpolation within a sliding window of length 8 frames includes: within the current sliding window, if the tracker is active in the last frame of the sliding window and the tracker's consecutive hit count is greater than or equal to 2, then quickly locate the tracker's hit queue (Queue). hit The missing bounding box of frame k is filled in using the previously saved Kalman filter prediction of frame k. When the missing bounding box is filled in by the Kalman filter prediction, the hit symbol Queue of the current frame is... hit It will be reset from 0 to 1. With each passing frame, the oldest frame in the sliding window is popped and the tracking result of the latest frame is introduced to maintain eight historical tracking results in the sliding window. Although there is a delay of nearly eight frames at the beginning of the video, it still meets sufficient real-time requirements in practical applications.

[0104] In one embodiment, the method for processing the target video frame in step 200 to obtain the target information of the target video frame can be selected according to actual needs. For example, the target video frame can be input into a preset target detection network, which can then detect the target and its information in the current frame. The preset target detection network can be a trained deep convolutional neural network. When performing an ultrasound examination on the thyroid gland, the specific training process of the preset target detection network can include: (a) acquiring an ultrasound detection video dataset, organizing and collecting it for ultrasound experts, and obtaining the annotated data as shown in the image. Figure 5(a) The labeled dataset is preprocessed with mosaic, random cropping, and HSV (hue, saturation, value) perturbation to obtain a preprocessed training dataset; (b) The labeled dataset is preprocessed with K-means clustering algorithm to obtain the aspect ratios of the three most representative target boxes in the dataset, which are used as the preset anchor box ratios in the object detection model; (c) The data in the preprocessed training dataset is input into the object detection model to obtain the inference output, and the loss value is obtained by calculating the error between the loss function L and the labeled data in the dataset; (e) The object detection model is optimized by backpropagation using the SGDM algorithm and the loss value; (f) For the remaining batches of data in the training set of the preprocessed dataset, steps (d) and (e) are repeated sequentially until the number of iterations is reached, thereby obtaining a trained deep convolutional neural network.

[0105] The loss value used in the object detection model can be calculated using the following loss function L:

[0106] L=ω cls L cls +ω reg L reg +ω obj L obj , where ω cls ω cls and ω reg L represents the classification weight, position weight, and target confidence weight, respectively. cls L reg and L obj These represent the loss for classification, location, and confidence, respectively.

[0107] The pre-defined target detection model may include a backbone network, a neck network, and a decoupled detection head. The decoupled detection head includes a transition layer, a classification branch, and a regression branch. The backbone network is used to extract image features based on the ultrasound video dataset. The neck network is used to fuse image features from top to bottom and bottom to top to obtain fused features. The transition layer is used to perform convolution operations on the fused features to obtain a feature map. The classification branch is used to obtain the category prediction results of each target in the current frame based on the feature map, and to determine the category of each target based on the category prediction results. The regression branch is used to obtain the location prediction results of the detection box and the object presence probability prediction results of each target based on the feature map, to determine the location of each target based on the location prediction results, and to determine the confidence level of each target based on the category prediction results and the object presence probability prediction results.

[0108] Specifically, Darknet-53 is used as the backbone network, incorporating the CSP (Cross-Path Aggregation) and SPP (Spatial Pyramid Pooling) modules to enhance the robustness of the feature extraction network. CSP's unique cross-stage merging structure segments a large number of gradient flows, allowing them to propagate through different network paths, resulting in richer gradient information combinations. This reduces redundant computation and memory overhead while improving the network's ability to extract image features. The SPP module uses pooling layers of multiple scales to learn and fuse feature maps, effectively increasing the receptive field of the feature extraction network while introducing a small number of parameters. The neck network of the model employs the Path Aggregation Network (PAN) to introduce additional paths for information transmission, achieving top-down and bottom-up feature fusion. This significantly shortens the propagation path of image features from lower-level to higher-level information, effectively improving the network's ability to learn large objects.

[0109] In the detection head part of the model, a lightweight decoupled detection head is adopted, which uses an fc-head (fully connected head) as the classification task branch and a conv-head (convolution head) as the coordinate regression task branch. Its structure includes a transition layer, a classification branch, and a regression branch. The transition layer handles features obtained from the fusion of the backbone network and the neck network. First, use a 1x1 convolution. Dimensionality reduction yields feature maps. Then, the data is fed into the classification and regression branches respectively. Dimensionality reduction significantly reduces the computational cost of subsequent classification and regression branches, and reduces the introduction of parameters. Here, c, h, and w represent the number of channels, height, and width of the input feature map, respectively, and i represents the three feature layers fed into this detection head from the neck network.

[0110] Its number of channels c i The values ​​are 256, 512, and 1024 respectively.

[0111]

[0112] The classification branch consists of three fully connected layers fc1, fc2, and fc cls The structure consists of fc1 and fc2, which form an extended bottleneck structure with (256, 1024, 256) channels. This increases the number of channels in the intermediate layers, retains more feature information, and improves the performance of the classification branch. First, the feature map obtained from the transition layer... The data is fed into fc1 and fc2 in sequence to obtain fc. cls Then send it to FC. cls Obtain the class prediction results P of the network for each anchor box (target). cls .

[0113]

[0114] P cls =fc cls (F cls )

[0115] The regression branch consists of two 3x3 convolutional layers, Conv1 and Conv2, and two 1x1 convolutional layers, Conv... reg Conv obj Composition. We will first obtain the feature map from the transition layer. F is obtained by sequentially feeding the data into Conv1 and Conv2. reg Then they are sent to Conv respectively reg The network's position prediction P for each anchor frame is obtained. reg and Conv obj Obtain the probability prediction P of the object's existence. obj .

[0116]

[0117]

[0118]

[0119] The highest-scoring category prediction result is used as the target category, the highest-scoring location prediction result is used as the target location, and the product of the maximum probability of object existence and the highest score of the category prediction result is used as the target confidence score, thereby realizing the processing of the target video frame.

[0120] To better understand the above embodiments, a detailed explanation is provided below with reference to a specific embodiment. In this embodiment, the application of video processing methods to thyroid ultrasound examination is used as an example for illustration.

[0121] Reference Figure 6First, the ultrasonic detection video stream dataset collected by the ultrasonic detection probe is acquired. After preprocessing the video stream dataset, it is input into the trained target detection model to obtain target information such as target location, confidence level, and category of each target (thyroid gland, thyroid nodules, and surrounding tissues) in the target video frame. Based on the historical information obtained from historical frames, the predicted location of each historical target is predicted. According to the predicted location of the tracker, the target location of high-confidence targets (i.e., high-scoring candidate boxes) is matched once. For high-confidence targets that fail to match after the first match, a new tracker is created for them. The trackers that fail to match are matched a second time with the target locations of low-confidence targets (i.e., low-scoring candidate boxes). Among the targets that fail to match, targets with higher confidence are matched with new trackers, while targets with lower confidence are discarded. For newly created trackers (still in an inactive initial state), frame interpolation is performed using a high-confidence and low-confidence activation threshold module. The module employs an aggressive display and interpolation strategy for newly created trackers targeting targets with high confidence, and a more conservative strategy for those targeting targets with low confidence. Simultaneously, frame interpolation is performed based on the hit status of the trackers corresponding to the targets within the sliding window range to determine the appropriate display position for the target. When the target video frame is output after a delay corresponding to the length of the sliding window, the display position corresponds to the tracked target. This enables automatic detection and tracking of the thyroid gland, thyroid nodules, and surrounding tissues in the ultrasound video during thyroid ultrasound examinations. The tracking display results can be referenced... Figure 7 , Figure 7 The tracking results, including 4 video frames, are identified by manual annotation and by this method. The tracking accuracy is very high, which helps ultrasound physicians complete disease screening and diagnosis, speeds up the screening process, reduces the workload of ultrasound physicians, and improves the consistency of ultrasound diagnosis.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0123] Based on the same inventive concept, this application also provides a video processing apparatus for implementing the video processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more video processing apparatus embodiments provided below can be found in the limitations of the video processing method above, and will not be repeated here.

[0124] In one embodiment, such as Figure 8 As shown, a video processing apparatus is provided, comprising: a processing module 10, a prediction module 20, a matching module 30, and a display module 40, wherein:

[0125] The processing module 10 is used to acquire the target video frame of the target video stream, process the target video frame, and obtain the target information of the target video frame, including the target location and confidence level.

[0126] The prediction module 20 is used to predict the predicted position of each historical target based on the historical target information of the target video stream.

[0127] The matching module 30 is used to match the predicted position of each historical target with the target position of each target based on the confidence level of each target, and to determine the tracking target in the target video frame based on the matching result.

[0128] Display module 40 is used to determine the display position of the tracking target of the target video frame and output the target video frame.

[0129] In one embodiment, the matching module 30 is further configured to match the predicted location of each historical target with the target location of each target based on the confidence level of each target, and obtain a matching result;

[0130] Identify the targets that successfully match in the matching results; the tracked targets in the target video frame include the successfully matched targets.

[0131] In one embodiment, the matching module 30 is also used to identify targets that have not been successfully matched among targets whose confidence level is greater than a first confidence threshold.

[0132] In one embodiment, the matching module 30 is further configured to delete targets to be cleaned from the tracking targets. The targets to be cleaned include: newly created tracking targets whose predicted positions do not match the target positions of each target in the target video frame, which are newly created based on targets with confidence levels greater than a second confidence threshold in the adjacent previous video frame; and newly created tracking targets whose predicted positions do not match the target positions of each target in the target video frame, which are newly created based on targets with confidence levels less than a second confidence threshold and greater than a first confidence threshold in the adjacent previous two video frames.

[0133] In one embodiment, the display module 40 is further configured to determine whether it is necessary to add a display position to the corresponding video frame based on the tracking results of each video frame within a sliding window of a preset length for each tracking target; if it is necessary to add a display position, add a display position to the corresponding video frame; and output and display the target video frame after a preset delay.

[0134] In one embodiment, the display module 40 is further configured to determine that a display position needs to be added for the video frames without tracking targets, given that there are tracking targets in all the first number of video frames adjacent to the target video frame, and there are video frames without tracking targets between the video frames with tracking targets in a sliding window of a preset length.

[0135] In one embodiment, the display module 40 is further configured to determine, in a video frame in which no tracking target exists, the predicted position determined for the video frame, and add a display position for the video frame in which no tracking target exists.

[0136] Each module in the aforementioned video processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0137] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to video processing methods. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a video processing method.

[0138] Those skilled in the art will understand that Figure 9The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0141] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: Acquire the target video frame of the target video stream, process the target video frame, and obtain the target information of the target video frame, the target information including the target location and confidence level; Based on the historical target information of the target video stream, predict the predicted position of each historical target; Based on the confidence level of each target, the predicted position of each historical target is matched with the target position of each target, and the tracking target in the target video frame is determined according to the matching result; Determine the display position of the tracking target in the target video frame, and output the target video frame; The step of matching the predicted positions of each historical target with the target positions of each target based on the confidence level of each target, and determining the tracking targets in the target video frame based on the matching results, includes: matching the predicted positions of each historical target with the target positions of each target based on the confidence level of each target to obtain matching results; determining the successfully matched targets in the matching results, wherein the tracking targets in the target video frame include the successfully matched targets; The tracking targets in the target video frame also include: targets that were not successfully matched among the targets with a confidence level greater than the first confidence threshold; The method further includes: deleting targets to be cleaned from the tracking targets, wherein the targets to be cleaned include: newly created tracking targets in the adjacent previous video frame based on targets with a confidence level greater than a second confidence threshold, whose predicted positions do not match the target positions of the targets in the target video frame; and newly created tracking targets in the adjacent previous two video frames based on targets with a confidence level less than a second confidence threshold but greater than a first confidence threshold, whose predicted positions do not match the target positions of the targets in the target video frame.

2. The method according to claim 1, characterized in that, Determining the display position of the tracking target in the target video frame and outputting the target video frame includes: Based on the tracking results of each video frame of each tracking target within a sliding window of a preset length, determine whether it is necessary to add a display position for the corresponding video frame; When it is necessary to add a display position, add a display position for the corresponding video frame; The target video frame is output and displayed after a delay of the preset length.

3. The method according to claim 2, characterized in that, The step of determining whether to add a display position for the corresponding video frame based on the tracking results of each video frame within a sliding window of a preset length for each tracking target includes: If the tracking target is present in all of the first number of video frames preceding the target video frame, and there are video frames without the tracking target between the video frames with the tracking target in the sliding window of the preset length, it is determined that a display position needs to be added for the video frames without the tracking target.

4. The method according to claim 3, characterized in that, The method further includes: In video frames where the tracked target is not present, the predicted position determined for the video frame is added to the corresponding position of the video frame to serve as the display position for the added video frame.

5. A video processing apparatus, characterized in that, The device includes: The processing module is used to acquire the target video frame of the target video stream, process the target video frame, and obtain the target information of the target video frame, the target information including the target location and confidence level; The prediction module is used to predict the predicted position of each of the historical targets based on the historical target information of the target video stream. The matching module is used to match the predicted position of each historical target with the target position of each target based on the confidence level of each target, and to determine the tracking target in the target video frame according to the matching result; The display module is used to determine the display position of the tracking target of the target video frame and output the target video frame; The step of matching the predicted positions of each historical target with the target positions of each target based on the confidence level of each target, and determining the tracking targets in the target video frame based on the matching results, includes: matching the predicted positions of each historical target with the target positions of each target based on the confidence level of each target to obtain matching results; determining the successfully matched targets in the matching results, wherein the tracking targets in the target video frame include the successfully matched targets; The tracking targets in the target video frame also include: targets that were not successfully matched among the targets with a confidence level greater than the first confidence threshold; The matching module is further configured to delete targets to be cleaned from the tracking targets. The targets to be cleaned include: newly created tracking targets in the adjacent previous video frame based on targets with a confidence level greater than a second confidence threshold, whose predicted positions do not match the target positions of the targets in the target video frame; and newly created tracking targets in the adjacent previous two video frames based on targets with a confidence level less than a second confidence threshold but greater than a first confidence threshold, whose predicted positions do not match the target positions of the targets in the target video frame.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images