Video processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a video processing method, a video processing device, computer equipment and a computer storage medium, and relates to the technical field of video processing. The method comprises the following steps: acquiring a to-be-processed video, and performing target detection on the to-be-processed video based on a pre-configured neural network model to obtain a to-be-tracked object; based on the to-be-tracked object, a Kalman filtering prediction model is constructed in advance, so that the Kalman filtering prediction model predicts a target state vector of a next video frame according to an initial state vector of a historical video frame corresponding to a current video frame, and a motion track of the to-be-tracked object is obtained; and for a target to-be-tracked object in the to-be-tracked objects in the current video frame, according to a pre-trained deep learning appearance model, determining the appearance feature similarity between the target to-be-tracked object and other to-be-tracked objects, so as to perform matching tracking on the target to-be-tracked object based on the appearance feature similarity.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, a video processing apparatus, a computer device, and a computer storage medium. Background Art

[0002] With the rapid development of electronic and internet technologies, a wide variety of terminals have been widely used, and the applications running on these terminals are becoming increasingly diverse. In some video scenarios, such as live streaming, remote video surveillance and analysis, and intelligent video content creation and distribution, there's a need to convert video resources displayed horizontally (i.e., landscape videos) to vertically (i.e., portrait videos). Precise target tracking and intelligent image processing significantly improve the content quality, visual effects, and user immersion of vertical videos.

[0003] Currently, the mainstream approach is to use a single-mode target tracking algorithm combined with cropping to convert the video to a vertical screen. For example, some software simply uses a mean-shift algorithm to track the target and crop the horizontal video to a 16:9 ratio.

[0004] However, the above method fails to fully consider the scene complexity and target dynamic characteristics changes in the video screen, making it susceptible to lighting changes, target occlusion, background interference and target appearance changes in complex scenes, causing tracking drift or loss of target, and unable to dynamically optimize the processing strategy, resulting in loss of vertical screen information, confusion of visual focus or truncation of key elements, weakening the video transmission effect and thus affecting the video quality. Summary of the Invention

[0005] In the embodiments of the present application, a video processing method, a video processing method apparatus, a computer device, and a computer storage medium are provided, thereby overcoming, to at least a certain extent, the technical problems of difficulty in coping with complex scene images, easy tracking of drifting or lost targets, and inability to lose vertical screen image information, which in turn affects video quality, due to the limitations and defects of related technologies.

[0006] A first aspect of an embodiment of the present application provides a video processing method, which includes: obtaining a video to be processed, and performing target detection on the video to be processed based on a preconfigured neural network model to obtain an object to be tracked; based on the object to be tracked, pre-constructing a Kalman filter prediction model, so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame, to obtain a motion trajectory of the object to be tracked; for the target object to be tracked among the objects to be tracked in the current video frame, determining the appearance feature similarity between the target object to be tracked and other objects to be tracked based on a pre-trained deep learning appearance model, so as to match and track the target object to be tracked based on the appearance feature similarity.

[0007] According to a second aspect of an embodiment of the present application, a video processing device is provided, which includes: a target detection module for acquiring a video to be processed, and performing target detection on the video to be processed based on a pre-configured neural network model to obtain an object to be tracked; a motion trajectory prediction module for pre-constructing a Kalman filter prediction model based on the object to be tracked, so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame, to obtain the motion trajectory of the object to be tracked; and a tracking module for determining, for a target object to be tracked among the objects to be tracked in the current video frame, the similarity of appearance features between the target object to be tracked and other objects to be tracked based on a pre-trained deep learning appearance model, so as to match and track the target object to be tracked based on the similarity of appearance features.

[0008] According to a third aspect of an embodiment of the present application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above video processing methods are implemented.

[0009] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above video processing methods are implemented.

[0010] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps of any one of the above video processing methods.

[0011] The technical solution of this application has the following beneficial effects:

[0012] Through the video processing method, a video to be processed is obtained, and target detection is performed on the video to be processed based on a pre-configured neural network model to obtain an object to be tracked; based on the object to be tracked, a Kalman filter prediction model is pre-constructed so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame to obtain the motion trajectory of the object to be tracked; for the target object to be tracked among the objects to be tracked in the current video frame, the appearance feature similarity between the target object to be tracked and other objects to be tracked is determined based on a pre-trained deep learning appearance model, so that the target object to be tracked is matched and tracked based on the appearance feature similarity.

[0013] In this method, the Kalman filter prediction model and the deep learning appearance model are deeply integrated in the specified position recognition and tracking to predict the motion trajectory of the target object to be tracked, and the similarity of the appearance features of the target object to be tracked is determined. This method can accurately identify the object to be tracked that is occluded or has changed in appearance, thereby

[0014] This method effectively addresses short-term target occlusion or rapid motion, reducing the risk of tracking loss. This method avoids the technical issues of traditional tracking algorithms in complex scenes, which are often susceptible to tracking drift or target loss due to changes in lighting, target occlusion, background interference, and changes in target appearance. This method improves tracking stability and continuity in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 An architectural diagram of a video processing system provided in one embodiment of the present application;

[0017] Figure 2 A flowchart of a video processing method provided in one embodiment of the present application;

[0018] Figure 3 A schematic diagram of tracking a single object to be tracked provided in one embodiment of the present application;

[0019] Figure 4 A schematic diagram of tracking multiple objects to be tracked provided in one embodiment of the present application;

[0020] Figure 5 A schematic diagram of the structure of a video processing device provided in one embodiment of the present application;

[0021] Figure 6 A schematic diagram of the computer device structure provided for one embodiment of the present application. DETAILED DESCRIPTION

[0022] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0023] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0024] The flowcharts shown in the accompanying drawings are merely exemplary and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0025] With the rapid development of electronic and internet technologies, a wide variety of terminals have been widely used, and correspondingly, the types and functionality of applications on these terminals have increased. In some video scenarios, such as live streaming, remote video surveillance and analysis, and intelligent video content creation and distribution, there is a need to convert video resources displayed in landscape mode (i.e., landscape videos / horizontal videos) into video resources displayed in portrait mode (i.e., portrait videos / vertical videos). This provides users with a superior visual experience in portrait mode, meeting the diverse demands for vertical video quality, content integrity, and viewing comfort in various scenarios in the mobile internet era. For example, live quiz applications are a very common type of video application. To enhance the interactive experience, an encoding server can convert video frames captured by a camera that are suitable for landscape display (referred to as landscape video frames) into live video streams suitable for portrait display (referred to as portrait video streams). For example, video frames captured by a camera with a 16:9 aspect ratio can be converted into video frames with a 9:16 aspect ratio. Another example: short videos are usually vertical videos. For the massive amount of horizontal video resources, there is a need to convert horizontal video resources into vertical short videos for traffic diversion and promotion.

[0026] Converting horizontal videos to vertical formats often results in a loss of image information. Preserving key information (i.e., the target object / area) is a crucial factor influencing the conversion's effectiveness. Automatically tracking target objects within the video can significantly improve the content quality, visual quality, and user immersion of vertical videos. This effectively addresses the shortcomings of traditional video processing technologies in target tracking and vertical adaptation, meeting the growing demand for vertical video in the market.

[0027] Currently, most target tracking algorithms rely on a single mode combined with a fixed-ratio cropping scheme to convert the video to a vertical orientation. For example, some software simply uses a mean-shift algorithm to track targets and crops horizontal videos to a fixed 16:9 ratio, failing to fully consider scene complexity and the dynamic characteristics of the target. Furthermore, simple target tracking algorithms, such as those based on optical flow and template matching, can track target motion to a certain extent, but their accuracy and robustness are limited.

[0028] That is to say, the above existing technical solutions mainly have the following technical problems:

[0029] 1) Insufficient tracking accuracy: Traditional target tracking algorithms are susceptible to lighting changes, target occlusion, background interference, and changes in target appearance in complex scenes, causing tracking drift or loss of the target. For example, in sports events, where athletes move at high speed and are frequently obscured, these existing technologies often lose track of the athletes, making it impossible to provide a stable target image for vertical videos, severely impacting the viewing experience.

[0030] 2) Poor scene adaptability: Existing technologies use fixed cropping ratios and tracking modes that are difficult to adapt to diverse scene requirements. For example, in surveillance videos with multiple small-scale objects, scenes with rich special effects in movies, and complex interactive live broadcasts, these existing technologies are unable to dynamically optimize processing strategies, resulting in loss of vertical screen information, confusion of visual focus, or truncation of key elements, weakening the video's effectiveness.

[0031] 3) Weak intelligent processing capabilities: These existing technologies lack the ability to understand the semantics of video scenes and analyze target associations, making them incapable of optimizing vertical screen layouts based on the video's theme, character relationships, and plot development. When processing narrative-heavy videos, they struggle to highlight key characters and plot points, reducing vertical videos to simple cropped and spliced images, failing to effectively convey the video's deeper meaning and artistic value.

[0032] In order to solve the above technical problems, the embodiments of the present application provide a video processing method. In this regard, the basic idea of the embodiments disclosed herein is: by integrating multiple advanced AI tracking technologies, such as deep integration of Kalman filtering and deep learning appearance models in specified position recognition and tracking, multiple targets can be accurately tracked at the same time in complex scenarios, solving the problem of confusion and loss of multi-target tracking in existing technologies.

[0033] This application proposes a video processing method and device that can be applied to Figure 1 In the system architecture of the exemplary application environment shown.

[0034] like Figure 1 As shown, the system architecture 100 may include a terminal device 101 and a server 102 .

[0035] The terminal device 101 may be, for example, a smartphone, a personal digital assistant (PDA), a laptop computer, a server, a desktop computer, or any other computing device with networking capabilities, but is not limited thereto. The terminal device 101 may be the same device that supports both horizontal and vertical video display, or may be different terminal devices, for example, one terminal device supports horizontal video and another terminal device supports vertical video. The embodiments of the present application do not impose any special restrictions on this.

[0036] The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The network 103 is a medium for providing a communication link between the terminal device 101 and the server 102. The network 103 can include various connection types, such as wired, wireless communication links, or fiber optic cables. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0037] The video processing method provided in the embodiment of the present application can be executed on the server 102, and accordingly, the video processing device is generally provided in the server 102. The video processing method provided in the embodiment of the present application can also be executed in the terminal device 101, and accordingly, the video processing device can also be provided in the terminal device 101. The video processing method provided in the embodiment of the present application can also be partially executed in the server 102 and partially executed in the terminal device 101, and accordingly, some modules of the video processing device can be provided in the server 102 and some modules can be provided in the terminal device 101.

[0038] For example, in an exemplary embodiment, the server 102 obtains the video to be processed from the terminal device 101, and performs target detection on the video to be processed based on a pre-configured neural network model to obtain the object to be tracked; based on the object to be tracked, a Kalman filter prediction model is pre-constructed so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame, and obtains the motion trajectory of the object to be tracked; for the target object to be tracked among the objects to be tracked in the current video frame, the appearance feature similarity between the target object to be tracked and other objects to be tracked is determined based on a pre-trained deep learning appearance model, so as to match and track the target object to be tracked based on the appearance feature similarity.

[0039] The following uses the server 102 as an execution subject and illustrates the application of the video processing method to the server 102 as an example. Figure 2 A flowchart schematically illustrates a video processing method in this exemplary embodiment, see Figure 2 The video processing method provided in the embodiment of the present application includes the following steps 201 to 203:

[0040] Step 201: Obtain a video to be processed, and perform target detection on the video to be processed based on a pre-configured neural network model to obtain an object to be tracked.

[0041] Step 202: Based on the object to be tracked, a Kalman filter prediction model is pre-built so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame to obtain the motion trajectory of the object to be tracked.

[0042] Step 203: For a target object to be tracked among the objects to be tracked in the current video frame, determine the appearance feature similarity between the target object to be tracked and other objects to be tracked based on a pre-trained deep learning appearance model, so as to match and track the target object to be tracked based on the appearance feature similarity.

[0043] In the above Figure 2 The proposed technical solution uses a Kalman filter prediction model and a deep learning appearance model to predict the trajectory of the target object and determine the similarity of its appearance features during designated location recognition and tracking. This allows for accurate identification of objects that are occluded or have changed appearance, effectively addressing short-term occlusion or rapid motion, and reducing the risk of tracking loss. This approach avoids the technical issues of traditional tracking algorithms being susceptible to tracking drift or target loss in complex scenarios, such as illumination changes, target occlusion, background interference, and changes in target appearance. This approach improves tracking stability and continuity in complex scenarios.

[0044] The following will be combined with specific embodiments to Figure 2 The specific implementation of each step in the embodiment shown is described in detail:

[0045] In step 201, a video to be processed is obtained, and target detection is performed on the video to be processed based on a pre-configured neural network model to obtain an object to be tracked.

[0046] Among them, the video to be processed can be the initial source video captured in real time by the video capture device, or the video file pre-captured and stored by the video capture device and decoded according to the video format, or it can be the video after a series of operations such as decoding, frame rate normalization, and resolution adaptation are performed on the above-mentioned original video or video file according to the video format. The embodiments of the present application do not impose any special restrictions on this.

[0047] For example, a high-speed hardware decoder (e.g., a Field Programmable Gate Array (FPGA) accelerated decoder) can be used to decode the raw video captured by the video capture device according to common video coding standards (e.g., MPEG-2, H.264, H.265) to obtain the processed video. It should be noted that these video coding standards are used to achieve video data compression and efficient transmission.

[0048] For example, the original video captured by the video capture device and the collected video files can be connected to the system using protocols such as the Real-Time Streaming Protocol (RTSP) and the Real-Time Messaging Protocol (RTMP), and then decoded in real time based on the high-speed hardware decoder in the system to obtain the video to be processed.

[0049] The object to be tracked may be an object area on a video frame or a detection frame containing the object.

[0050] In an optional embodiment of the present disclosure, target recognition may be performed before target detection is performed on the video to be processed.

[0051] In order to overcome the lack of semantic understanding of video scenes and target association analysis capabilities in related technical solutions, it is impossible to optimize the vertical screen layout according to the video theme, character relationships and plot development. When processing videos with strong narrative, it is difficult to highlight the core characters and key plots. The vertical screen video is reduced to a simple picture cut and splicing, which cannot effectively convey the deep connotation and artistic value of the video. The technical problem is that the video scene semantics can be evaluated in real time based on the characteristics of the video scene. For example, in monitoring scenarios, there are scenes with a small number of targets and slow movements, scenes with a sharp increase in the number of targets or scenes with a movement intensity exceeding the threshold (measured by target speed variance and target density change rate), etc., and the vertical screen layout can be optimized according to the video theme, character relationships and plot development.

[0052] In an optional embodiment of the present disclosure, target detection is performed on the video to be processed based on a preconfigured neural network model to obtain an object to be tracked, including: for each target video frame in the video to be processed, extracting a multi-scale feature map of the target video frame according to a feature extraction network configured in the neural network model; performing feature fusion on the multi-scale feature map of the target video frame according to a feature fusion module, and performing target detection of the object to be tracked on the fused multi-scale feature map via an output layer to obtain the object to be tracked.

[0053] The pre-configured neural network model can be a modified YOLOv5 neural network for object detection in video frames. It includes a feature extraction network (CSPDarknet53 structure) and a feature fusion module, PANet. The feature extraction network is used to extract multi-scale feature maps for each target video frame, while the feature fusion module, PANet, aggregates information across layers of the features extracted from the backbone network to further optimize target detection accuracy. After feature fusion, the multi-scale feature maps of the image can be used in the output layer (Head) for final detection prediction of the object to be tracked.

[0054] For example, the improved multi-target detection network based on YOLOv5 mentioned above incorporates an attention mechanism (CBAM module) to improve target detection accuracy. The network divides the input image into multi-scale grid cells, each of which predicts the target bounding box, category probability, and target center confidence. Multi-scale feature fusion is achieved through the Feature Pyramid Network (FPN) and Path Aggregation Network (PAN), enhancing small target detection capabilities.

[0055] In order to further ensure the accuracy and effectiveness of the final target detection results, the non-maximum suppression algorithm can be used for post-processing to remove candidate boxes with too high overlap and only retain the detection box with the highest confidence.

[0056] In step 202, based on the object to be tracked, a Kalman filter prediction model is pre-constructed so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame to obtain the motion trajectory of the object to be tracked.

[0057] Among them, the Kalman filter prediction model predicts the state vector of the future video frame of the object to be tracked based on the initial state vector of the historical video frame, thereby obtaining the motion trajectory of the object to be tracked.

[0058] It should be explained that the object to be tracked may be one or more objects. When there are multiple objects to be tracked, it is necessary to pre-build Kalman filter prediction models for each of the multiple objects to be tracked.

[0059] In an optional embodiment of the present disclosure, the initial state vector includes one or more of the following: position, velocity, acceleration components of the object to be tracked in a two-dimensional plane, and target size change parameters.

[0060] Exemplarily, the state vector includes the components of the target position, velocity, and acceleration in a two-dimensional plane and the target size change parameter, and the possible state of the target in the next frame is predicted through the system dynamic model.

[0061] In an optional embodiment of the present disclosure, the Kalman filter prediction model includes at least a process noise covariance matrix, historical motion data of the object to be tracked is obtained, and the historical motion pattern of the object to be tracked is determined based on the historical motion data. According to the mapping relationship between the historical motion pattern and the process noise covariance matrix, the process noise covariance matrix included in the Kalman filter prediction model is adaptively adjusted.

[0062] The historical motion mode is one of a uniform motion mode, an accelerated motion mode, and a variable speed motion mode.

[0063] For example, based on historical motion patterns, such as uniform speed, acceleration, and variable speed motion patterns, the process noise covariance matrix can be adaptively adjusted so that the motion trajectory of the object to be tracked can be accurately estimated based on the process noise covariance matrix contained in the Kalman filter prediction model, thereby effectively dealing with scenes where the object to be tracked is occluded or moves rapidly in a short period of time, thereby reducing the risk of tracking loss.

[0064] In step 203, for the target object to be tracked among the objects to be tracked in the current video frame, the appearance feature similarity between the target object to be tracked and other objects to be tracked is determined based on the pre-trained deep learning appearance model, so as to match and track the target object to be tracked based on the appearance feature similarity.

[0065] The target object to be tracked is any one of multiple objects to be tracked. The deep learning appearance model learns the target appearance feature vector by introducing the ResNet network structure.

[0066] For example, in the process of training a deep learning appearance model, the network weight value can be optimized through a large number of positive and negative sample pairs (objects of the same type to be tracked are positive samples, and conversely, objects of different types to be tracked are negative samples), so that the generated feature vectors of the same type are gathered together and the different types are separated in the feature space, so as to accurately identify targets that are occluded or have changed appearance, and improve tracking stability and continuity in complex scenes.

[0067] In an optional embodiment, a soft label assignment strategy can be used to optimize the selection of positive and negative samples, improve model training efficiency and generalization ability, accurately detect multiple targets at one time in complex scenarios, and lay a solid data foundation for collaborative tracking.

[0068] The soft label assignment strategy dynamically adjusts the distribution of positive and negative samples in object detection tasks by calculating soft labels and positive and negative weights based on the predicted results and the ground truth boxes. This strategy dynamically adjusts the weights of positive and negative samples based on the model's predictions during training, thereby optimizing the model's training performance. Commonly used soft label assignment strategies can be ATSS (Adaptive Training Sample Selection), which adaptively selects positive and negative samples based on L2 distance and IoU threshold; SimOTA (Simplified Online Target Assignment): dynamically selects positive and negative samples based on cost matrix and IoU, and dynamically adjusts the distribution of positive and negative samples through cost matrix and IoU value; AutoAssign: automatically divides positive and negative samples based on score and center prior, and dynamically adjusts the weights of positive and negative samples through the degree of match between prediction results and true boxes; GFL (Generalized Focal Loss): calculates soft labels and positive and negative weights based on prediction results and true boxes, and distributes positive and negative samples and calculates loss according to the weights of positive and negative samples; or TOOD (Tood Loss): dynamically adjusts soft labels and positive and negative weights during training, and optimizes the model training process according to the degree of match between prediction results and true boxes. The embodiments of the present application do not impose any restrictions on this.

[0069] The appearance feature similarity may be measured by cosine similarity or by other similarity calculation methods.

[0070] Furthermore, in addition to performing matching tracking on the target object to be tracked based on the similarity of appearance features, matching tracking can also be performed based on the following embodiments:

[0071] In an optional embodiment of the present disclosure, the initial state vector includes at least the position of the object to be tracked, and the target object to be tracked is matched and tracked based on the similarity of appearance features, including: determining the position intersection-and-union ratio between the target object to be tracked and other objects to be tracked based on the position of the object to be tracked in the video to be processed; and matching and tracking the target object to be tracked based on the appearance feature similarity and the position intersection-and-union ratio.

[0072] The Intersection over Union (IoU) is a metric used in computer vision to measure the degree of overlap between two bounding boxes. It is widely used in tasks such as object detection and semantic segmentation. It can be obtained by calculating the intersection / union area between multiple bounding boxes.

[0073] For example, when matching and tracking the object to be tracked, the position intersection over union (IoU) between the target object to be tracked and other objects to be tracked is calculated (for example, cosine similarity measurement), and combined with the spatial position correlation (i.e., position intersection over union (IoU) measurement), and then the match of the object to be tracked is determined by weighted fusion decision of the position intersection over union and appearance feature similarity, so as to accurately identify targets that are occluded or have changed appearance, and improve tracking stability and continuity in complex scenes.

[0074] When there are multiple objects to be tracked based on target detection, that is, in a multi-target tracking scenario, there is a data association and matching problem between the multiple objects detected in each frame and the tracking target set. To avoid confusion when switching between the identities of multiple objects to be tracked, the Hungarian algorithm can be used to solve the minimum cost matching problem.

[0075] In an optional embodiment of the present disclosure, there are multiple objects to be tracked, and the method further includes: performing minimum cost matching on the multiple objects to be tracked via a cost matrix based on the Hungarian algorithm; and performing matching tracking on the multiple objects to be tracked.

[0076] Among them, the cost matrix includes: the spatial distance between the predicted position and the detection frame position, the similarity of appearance features, and the motion continuity constraints of multiple objects to be tracked; the predicted position is based on the Kalman filter prediction model to determine the positions of multiple objects to be tracked in the next video frame.

[0077] For example, the cost matrix is constructed by comprehensively considering the spatial distance between the Kalman filter prediction position and the detection frame position, the similarity of appearance features, and the target motion continuity constraints (such as speed and direction consistency). Through efficient matching and association, the identity consistency and trajectory coherence of each target in multi-target tracking scenarios are ensured, avoiding target identity switching confusion and providing accurate data support for multi-target presentation on vertical screens.

[0078] The target object to be tracked may be manually determined by the user.

[0079] The following will be combined Figure 3 、 Figure 4 The process of the above embodiment is described.

[0080] Reference Figure 3 As shown, assuming that the video to be processed is a picture of a person, such as a live broadcast of a large-scale gathering or a group performance in a film or television scene, after starting the AI character tracking process, the user manually determines the target object to be tracked, and then only the target object to be tracked can be matched and tracked to form a corresponding motion trajectory.

[0081] See also Figure 4Assuming that the video to be processed is a picture of a person, after starting the AI person tracking process, by selecting multiple target objects to be tracked, target matching confirmation and tracking movement of multiple target objects to be tracked can be performed to form corresponding multiple motion trajectories.

[0082] In the above embodiment, by integrating multiple advanced AI tracking technologies, such as deep integration of Kalman filtering, deep learning appearance model and Hungarian algorithm in specified location recognition and tracking, multiple targets can be accurately tracked simultaneously in complex scenarios, solving the problem of confusion and loss in multi-target tracking of existing technologies.

[0083] In addition, since the object to be tracked can be one or more, in order to enhance the robustness and accuracy of multi-target tracking and ensure the quality of multi-target scene processing in vertical video, a distributed tracking framework can be used, for example,

[0084] For multiple objects to be tracked, multiple target trackers (based on a combination of a Kalman filter prediction model and a deep learning appearance model) are constructed to run in parallel and exchange information in real time. When there is occlusion or complex interaction between targets, the tracking parameters are collaboratively adjusted based on the target relationship graph model. For example, the motion information of visible targets around the occluded target is shared to assist in the prediction of the occluded target, or the deviation of individual trackers is corrected based on the target group motion consistency constraint. Through collaborative optimization, the limitations of single target tracking are overcome, the robustness and accuracy of multi-target tracking are enhanced, and the processing quality of multi-target scenes in vertical screen videos is guaranteed.

[0085] Accordingly, to increase flexibility, the system can preset multiple tracking modes and switching thresholds, and evaluate switching needs in real time based on the characteristics of the video scene. For example, in a surveillance scenario, when the number of targets is small and the movement is slow, the single-target high-precision tracking mode is activated; when the number of targets increases dramatically or the intensity of the movement exceeds the threshold (measured by the target velocity variance and the target density change rate), it seamlessly switches to the multi-target collaborative tracking mode. During the switching process, transfer learning technology is used to initialize the new tracking mode parameters, accelerate the convergence of the stable tracking state, ensure efficient and accurate tracking throughout the process, and optimize the continuity and integrity of the target presentation in vertical screen videos.

[0086] In the above-described embodiment, tracking modes and parameters are intelligently switched based on the characteristics of the video scene, seamlessly transitioning from low-motion surveillance to high-motion sports, and from single-target close-ups to multi-target panoramic scenes. Tracking priorities are dynamically determined based on semantic understanding to ensure the presentation of core elements. This adaptive capability overcomes the limitations of fixed modes in existing technologies, for example, automatically optimizing tracking strategies for different live broadcasts or surveillance scenarios, ensuring that the vertical screen image meets the needs of the scene, and enhancing the user experience's versatility and immersion.

[0087] In an optional embodiment of the present disclosure, an error evaluation index parameter is determined; the error evaluation index includes one or more of the following: a deviation between the actual position and the predicted position of the object to be tracked, a number of tracking loss frames, and a frequency of mistracking; if the error evaluation index parameter is greater than a preset error evaluation index parameter, a correction process is initiated;

[0088] The correction process includes one or more of the following:

[0089] The particle filter algorithm is used to resample the historical trajectory data of historical video frames to obtain the optimized target state vector;

[0090] Adjust the model parameters of the Kalman filter prediction model and the deep learning appearance model according to the error evaluation index parameters;

[0091] The error evaluation index parameters are fed back to the video acquisition device to adjust the device parameters of the video acquisition device.

[0092] Exemplarily, a tracking error evaluation model is established, with the deviation between the actual position of the target and the predicted position (such as the center coordinate offset, the bounding box IoU deviation), the number of tracking lost frames and the frequency of mistracking as key indicators. When the error exceeds the tolerance interval, the error correction mechanism is activated. On the one hand, multiple frames of historical trajectory data are traced back, and the particle filter algorithm is used to resample and optimize the target state estimation; on the other hand, the tracking model parameters (such as Kalman filter parameters, appearance model update rate) are reversely fine-tuned according to the error characteristics. At the same time, the error data is fed back to the front-end acquisition device or scene control system, such as adjusting the camera parameters (focal length, aperture, pan-tilt steering) to improve the tracking environment, forming a closed-loop optimization loop, and continuously improving the tracking performance and the stability of the vertical screen video processing quality.

[0093] Through the above-mentioned embodiments, the real-time tracking error correction mechanism and closed-loop feedback adjustment significantly enhance tracking stability. In the face of interference such as occlusion, sudden changes in illumination, and significant changes in target appearance, tracking accuracy is maintained through backtracking optimization, parameter fine-tuning, and environmental collaborative adjustment. For example, in scenarios such as sports events where athletes frequently enter and exit shadows or change equipment, continuous and stable tracking is achieved, reducing tracking jitter and loss, laying a solid technical foundation for creating high-quality content for vertical screen videos, far surpassing the weak anti-interference and easy interruption defects of existing technologies.

[0094] Finally, during the object tracking process, motion trajectory and target status information can be generated in real time and transmitted to the screen cropping and optimization module. According to the target position, size, posture and preset vertical screen specifications (such as 9:16, 4:3), the dynamic programming algorithm and image segmentation technology are used to accurately crop the screen. The image quality is then improved by the image post-processing unit (including color correction, noise reduction, and super-resolution reconstruction). Finally, it is encoded and packaged into a vertical screen video stream suitable for the playback device and pushed to the terminal for playback. The entire process is controlled by the performance monitoring and feedback adjustment module to optimize the processing parameters in real time based on bandwidth, delay, and user interaction feedback to ensure a smooth and stable playback experience.

[0095] It should be understood that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0096] To implement the above video processing method, see Figure 5 An embodiment of the present application provides a video processing device, which may include a target detection module 501 , a motion trajectory prediction module 502 , and a tracking module 503 .

[0097] Among them, the target detection module 501 is used to obtain the video to be processed, and perform target detection on the video to be processed based on a pre-configured neural network model to obtain the object to be tracked; the motion trajectory prediction module 502 is used to pre-construct a Kalman filter prediction model based on the object to be tracked, so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame, and obtains the motion trajectory of the object to be tracked; the tracking module 503 is used to determine the appearance feature similarity between the target object to be tracked and other objects to be tracked according to a pre-trained deep learning appearance model for the target object to be tracked in the objects to be tracked in the current video frame, so as to match and track the target object to be tracked based on the appearance feature similarity.

[0098] In an optional embodiment of the present disclosure, the target detection module 501 is used to extract a multi-scale feature map of the target video frame for each target video frame in the video to be processed according to the feature extraction network configured in the neural network model; perform feature fusion on the multi-scale feature map of the target video frame according to the feature fusion module, and perform target detection of the object to be tracked on the fused multi-scale feature map through the output layer to obtain the object to be tracked.

[0099] In an optional embodiment of the present disclosure, the initial state vector includes one or more of the following: position, velocity, acceleration components of the object to be tracked in a two-dimensional plane, and target size change parameters.

[0100] In an optional embodiment of the present disclosure, the initial state vector includes at least the position of the object to be tracked, and the tracking module 503 is used to determine the position intersection-over-union ratio between the target object to be tracked and other objects to be tracked based on the position of the object to be tracked in the video to be processed; and match and track the target object to be tracked based on the appearance feature similarity and the position intersection-over-union ratio.

[0101] In an optional embodiment of the present disclosure, the Kalman filter prediction model includes at least a process noise covariance matrix, and the device further includes a motion mode determination module and an adjustment module. The motion mode determination module is used to obtain historical motion data of the object to be tracked, and determine the historical motion mode of the object to be tracked based on the historical motion data, where the historical motion mode is one of a uniform motion mode, an accelerated motion mode, and a variable speed motion mode; the adjustment module is used to adaptively adjust the process noise covariance matrix included in the Kalman filter prediction model based on the mapping relationship between the historical motion mode and the process noise covariance matrix.

[0102] In an optional embodiment of the present disclosure, there are multiple objects to be tracked, and the device further includes a matching module, which is used to perform minimum cost matching on the multiple objects to be tracked through a cost matrix according to the Hungarian algorithm; so as to perform matching tracking on the multiple objects to be tracked respectively; the cost matrix includes: the spatial distance between the predicted position and the detection frame position, the similarity of the appearance features and the motion continuity constraint of the multiple objects to be tracked; wherein the predicted position is based on the Kalman filter prediction model to determine the positions of the multiple objects to be tracked in the next video frame.

[0103] In an optional embodiment of the present disclosure, the device further includes an indicator parameter determination module and a correction process starting module, the indicator parameter determination module is used to determine an error evaluation indicator parameter, the error evaluation indicator includes one or more of the following: the deviation between the actual position and the predicted position of the object to be tracked, the number of tracking loss frames and the frequency of mistracking;

[0104] A correction process starting module is used to start the correction process if the error evaluation index parameter is greater than the preset error evaluation index parameter;

[0105] The correction process includes one or more of the following:

[0106] The particle filter algorithm is used to resample the historical trajectory data of historical video frames to obtain the optimized target state vector;

[0107] Adjust the model parameters of the Kalman filter prediction model and the deep learning appearance model according to the error evaluation index parameters;

[0108] The error evaluation index parameters are fed back to the video acquisition device to adjust the device parameters of the video acquisition device.

[0109] For the specific definition of the above-mentioned video processing device, please refer to the definition of the video processing method above, and will not be repeated here. Each module in the above-mentioned video processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0110] In one embodiment, a computer device is provided. The internal structure diagram of the computer device can be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a video processing method as described above is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any step in the video processing method as described above is implemented.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above video processing method can be implemented.

[0112] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0116] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0117] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A video processing method, characterized in that: include: Obtaining a video to be processed, and performing target detection on the video to be processed based on a pre-configured neural network model to obtain an object to be tracked; Based on the object to be tracked, a Kalman filter prediction model is pre-built, so that the Kalman filter prediction model predicts the target state vector of the next video frame according to the initial state vector of the historical video frame corresponding to the current video frame, thereby obtaining the motion trajectory of the object to be tracked; For a target object to be tracked among the objects to be tracked in the current video frame, the appearance feature similarity between the target object to be tracked and other objects to be tracked is determined based on a pre-trained deep learning appearance model, so as to match and track the target object to be tracked based on the appearance feature similarity.

2. The method according to claim 1, characterized in that The method of performing target detection on the video to be processed based on a pre-configured neural network model to obtain an object to be tracked includes: For each target video frame in the video to be processed, extracting a multi-scale feature map of the target video frame according to a feature extraction network configured in the neural network model; The feature fusion module performs feature fusion on the multi-scale feature map of the target video frame, and performs target detection of the object to be tracked on the fused multi-scale feature map through the output layer to obtain the object to be tracked.

3. The method according to claim 1, characterized in that The initial state vector includes one or more of the following: position, velocity, acceleration components of the object to be tracked in a two-dimensional plane, and target size change parameters.

4. The method according to claim 3, characterized in that The initial state vector includes at least the position of the target object to be tracked, and matching tracking is performed on the target object to be tracked based on the appearance feature similarity, including: Determining, based on the position of the object to be tracked in the video to be processed, a position intersection-over-union ratio between the target object to be tracked and the other objects to be tracked; Matching tracking is performed on the target object to be tracked according to the appearance feature similarity and the position intersection-over-union ratio.

5. The method according to claim 1, wherein The Kalman filter prediction model at least includes a process noise covariance matrix, and the method further includes: Acquiring historical motion data of the object to be tracked, and determining a historical motion mode of the object to be tracked based on the historical motion data, wherein the historical motion mode is one of a uniform motion mode, an accelerated motion mode, and a variable speed motion mode; According to the mapping relationship between the historical motion pattern and the process noise covariance matrix, the process noise covariance matrix included in the Kalman filter prediction model is adaptively adjusted.

6. The method according to claim 1, characterized in that There are multiple objects to be tracked, and the method further includes: Based on the Hungarian algorithm, a minimum cost matching is performed on a plurality of objects to be tracked via a cost matrix; and matching tracking is performed on the plurality of objects to be tracked respectively. The cost matrix includes: the spatial distance between the predicted position and the detection frame position, the similarity of the appearance features, and the motion continuity constraints of the multiple objects to be tracked; The predicted position is determined based on the Kalman filter prediction model to determine the positions of the multiple objects to be tracked in the next video frame.

7. The method according to claim 1, characterized in that The method further comprises: Determining error evaluation index parameters, wherein the error evaluation index includes one or more of the following: a deviation between the actual position and the predicted position of the object to be tracked, a number of tracking loss frames, and a mistracking frequency; If the error evaluation index parameter is greater than the preset error evaluation index parameter, the correction process is started; The correction process includes one or more of the following: Resampling the historical trajectory data of the historical video frames using a particle filter algorithm to obtain an optimized target state vector; Adjusting the model parameters of the Kalman filter prediction model and the deep learning appearance model according to the error evaluation index parameter; The error evaluation index parameter is fed back to the video acquisition device to adjust the device parameters of the video acquisition device.

8. A video processing device, characterized in that: include: The target detection module is used to obtain the video to be processed and perform target detection on the video to be processed based on a pre-configured neural network model to obtain the object to be tracked; A motion trajectory prediction module is configured to pre-build a Kalman filter prediction model based on the object to be tracked, so that the Kalman filter prediction model predicts the target state vector of the next video frame based on the initial state vector of the historical video frame corresponding to the current video frame, thereby obtaining the motion trajectory of the object to be tracked; The tracking module is used to determine the appearance feature similarity between a target object to be tracked and other objects to be tracked in the current video frame according to a pre-trained deep learning appearance model, so as to match and track the target object to be tracked based on the appearance feature similarity.

9. A computer device comprising: The method comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

Images