Video target segmentation method and device, electronic equipment, storage medium and program product

Abstract

The application provides a target segmentation method and device of a video, an electronic device, a storage medium and a program product. The method comprises: acquiring frame features of key frames in a video to be processed, and acquiring track features of description information of the video, wherein the track features are used to represent a track of a target object in the video; fusing the frame features of the key frames and the track features to obtain fused features; performing target segmentation on the key frames based on the fused features to obtain a first segmentation result of the key frames; performing target segmentation on non-key frames in the video based on the first segmentation result to obtain a second segmentation result of the non-key frames; and determining a target segmentation result of the video based on the first segmentation result and the second segmentation result, wherein the target segmentation result is used to separate the target object from the video. Through the application, the accuracy of target segmentation of the video can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, storage medium, and program product for target segmentation of video. Background Technology

[0002] Video segmentation is the process of segmenting each frame of a video sequence. Since video is a continuous sequence of frames, video segmentation needs to consider not only the image features within a single frame but also the temporal relationships between frames. In video segmentation, each frame can be segmented independently, and then the segmentation results are stitched together to improve the accuracy and coherence of the segmentation. Video segmentation is widely used in video editing, content analysis, detection, and autonomous driving.

[0003] In related technologies, for target segmentation in video, the target is usually segmented independently for each video frame, and then the segmentation results of each video frame are stitched together to obtain the video segmentation result. Since the objects in different video frames have trajectory continuity, segmenting the video frame independently in this way will lead to low accuracy of target segmentation in video. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for target segmentation in video, which can effectively improve the accuracy of target segmentation in video.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a video target segmentation method, including:

[0007] The frame features of keyframes in the video to be processed are obtained, and the trajectory features of the target object in the video are obtained. The trajectory features are used to characterize the trajectory of the target object in the video.

[0008] The frame features of the keyframes are fused with the trajectory features to obtain fused features;

[0009] Based on the fusion features, target segmentation is performed on the key frame to obtain the first segmentation result of the key frame;

[0010] Based on the first segmentation result, target segmentation is performed on the non-key frames in the video to obtain the second segmentation result of the non-key frames;

[0011] Based on the first segmentation result and the second segmentation result, a target segmentation result is determined for the video, and the target segmentation result is used to separate the target object from the video.

[0012] This application provides a video target segmentation apparatus, including:

[0013] The acquisition module is used to acquire the frame features of key frames in the video to be processed, and to acquire the trajectory features of the target object in the video, wherein the trajectory features are used to characterize the trajectory of the target object in the video;

[0014] The fusion module is used to fuse the frame features of the key frame with the trajectory features to obtain fused features;

[0015] The segmentation module is used to perform target segmentation on the keyframe based on the fusion features to obtain a first segmentation result of the keyframe; and to perform target segmentation on non-keyframes in the video based on the first segmentation result to obtain a second segmentation result of the non-keyframes.

[0016] A determining module is configured to determine a target segmentation result of the video based on the first segmentation result and the second segmentation result, wherein the target segmentation result is used to separate the target object from the video.

[0017] In the above scheme, the acquisition module is further used to acquire initial description information of the video, the initial description information being used to describe the behavior of the target object; and to obtain the trajectory features of the target object based on the video and the initial description information.

[0018] In the above scheme, the acquisition module is further configured to obtain a prompt based on the initial description information, the prompt being used to instruct the trajectory recognition model to output the trajectory features of the target object based on the initial description information; and to identify the trajectory of the target object in the video based on the video and the prompt, thereby obtaining the trajectory features of the target object.

[0019] In the above scheme, the acquisition module is further configured to acquire a first sample prompt from the video sample. The first sample prompt is obtained based on the first sample description information and is used to instruct the initial trajectory recognition model to output the sample trajectory features of the sample target object based on the first sample description information. The first sample description information is used to describe the behavior of the sample target object in the video sample. Based on the video sample and the first sample prompt, the initial trajectory recognition model is used to identify the trajectory of the sample target object in the video sample to obtain the sample trajectory features of the sample target object. Based on the sample trajectory features, the initial trajectory recognition model is trained to obtain the trajectory recognition model.

[0020] In the above scheme, the acquisition module is further configured to acquire a second sample prompt from the video sample. The second sample prompt is obtained based on the location information of the target object in the video sample and is used to instruct the initial trajectory recognition model to output the sample behavior features of the target object based on the location information. The location information is used to indicate the position of the target object in multiple video frames in the video sample. Based on the video sample and the second sample prompt, the initial trajectory recognition model is used to identify the behavior of the target object in the video sample to obtain the sample behavior features of the target object. Based on the sample behavior features, the initial trajectory recognition model is trained to obtain the trajectory recognition model.

[0021] In the above scheme, the segmentation module is further used to encode the frame features of the key frame to obtain a first encoded feature; and to decode the first encoded feature based on the fusion feature to obtain a first segmentation result of the key frame.

[0022] In the above scheme, the segmentation module is further used to encode the frame features of the non-key frame to obtain a second encoded feature; and to decode the second encoded feature based on the first segmentation result to obtain a second segmentation result of the non-key frame.

[0023] In the above scheme, the frame features of the key frame include multiple sub-frame features. The above fusion module is also used to determine the similarity between each of the sub-frame features of the key frame and the trajectory features respectively; and to perform weighted fusion of each sub-frame feature and the trajectory features with each similarity as a weight to obtain the fused features corresponding to the key frame.

[0024] In the above scheme, the key frames and non-key frames in the video are ordered by frame, and the target segmentation of the key frames and the target segmentation of the non-key frames are performed sequentially according to the frame order. The segmentation module is further configured to perform the following processing on each of the non-key frames in the video: determine the position of the non-key frame in the frame order, and determine the video frames preceding the non-key frames based on the position, wherein the video frames preceding the non-key frames include key frames preceding the non-key frames and other non-key frames preceding the non-key frames; and perform target segmentation on the non-key frames based on the first segmentation result of the key frames preceding the non-key frames and the second segmentation result of the other non-key frames preceding the non-key frames to obtain the second segmentation result of the non-key frames.

[0025] In the above scheme, the first segmentation result of the key frame preceding the non-key frame and the second segmentation results of other non-key frames preceding the non-key frame are stored in the result library; the segmentation module is further configured to replace the second segmentation result of at least one other non-key frame with the first frame in the result library with the second segmentation result of the non-key frame when the storage space of the result library reaches a storage threshold; or to compress the second segmentation result of at least one other non-key frame with the first frame in the result library and store the second segmentation result of the non-key frame in the result library when the storage space of the result library reaches a storage threshold.

[0026] This application provides an electronic device, including:

[0027] Memory is used to store executable instructions or computer programs.

[0028] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video target segmentation method provided in the embodiments of this application.

[0029] This application provides a computer-readable storage medium storing computer-executable instructions for inducing a processor to execute and implement the video target segmentation method provided in this application.

[0030] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video target segmentation method described in this application.

[0031] The embodiments of this application have the following beneficial effects:

[0032] By acquiring the frame features of keyframes in the video to be processed and obtaining the trajectory features of the target object in the video, the frame features of the keyframes and the trajectory features are fused to obtain fused features. Based on the fused features, target segmentation is performed on the keyframes to obtain the first segmentation result of the keyframes. Based on the first segmentation result, target segmentation is performed on the non-keyframes in the video to obtain the second segmentation result of the non-keyframes. Based on the first and second segmentation results, the target segmentation result of the video is determined. In this way, since the trajectory features represent the trajectory of the target object in the video, reflecting the segmentation requirements of the video, while the frame features of the keyframes reflect the spatial information in the keyframes, by fusing the frame features of the keyframes and the trajectory features, the obtained fused features can comprehensively reflect the spatial information and segmentation requirements of the keyframes. By performing target segmentation on the keyframes based on the fused features to obtain the first segmentation result of the keyframes, the segmentation accuracy of the keyframes is effectively improved. Because objects in keyframes and non-keyframes in a video exhibit dynamic trajectory relationships, the non-keyframes in the video are segmented based on the first segmentation result of the keyframes, resulting in a second segmentation result for the non-keyframes. This process fully considers the dynamic trajectory relationships between non-keyframes and keyframes when segmenting non-keyframes, making the second segmentation result for non-keyframes more accurate. Consequently, the target segmentation result of the video determined based on the first and second segmentation results is more accurate, effectively improving the accuracy of target segmentation in the video. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the architecture of the video target segmentation system provided in the embodiments of this application;

[0034] Figure 2 This is a schematic diagram of the structure of an electronic device for target segmentation of video provided in an embodiment of this application;

[0035] Figure 3 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 1 ;

[0036] Figure 4 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 2 ;

[0037] Figure 5 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 3 ;

[0038] Figure 6 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 4 ;

[0039] Figure 7 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 5 ;

[0040] Figure 8 This is a schematic diagram illustrating the principle of the video target segmentation method provided in the embodiments of this application. Figure 1 ;

[0041] Figure 9 This is a schematic diagram illustrating the principle of the video target segmentation method provided in the embodiments of this application. Figure 2 ;

[0042] Figure 10 This is a schematic diagram illustrating the principle of the bidirectional alignment mechanism provided in the embodiments of this application;

[0043] Figure 11 This is a schematic diagram comparing the video target segmentation method provided in this application embodiment with the effects of related technologies. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0048] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0049] 1) Video Frame: A video frame is a single image in a video; it is the basic unit of a video. During video playback, video frames are displayed continuously at a certain rate, creating the visual effect of continuous motion. Each frame is a still image in the video; when they are played continuously at a sufficiently high speed, the human eye cannot distinguish the transitions between each frame, thus perceiving smooth motion. A keyframe is a special frame in a video sequence that contains crucial information about the video sequence and is typically used for video compression and editing. A keyframe can be the beginning or end of a scene, or a frame with specific action or content.

[0050] 2) Image Segmentation: Image segmentation is a fundamental task in computer vision. It refers to the process of subdividing a digital image into multiple parts or objects. These parts typically share similar characteristics, such as color, brightness, and texture. The goal of image segmentation is to simplify image analysis, making each part of the image easier to identify, understand, and process.

[0051] 3) Video Segmentation: Video segmentation is the process of dividing a video sequence into individual frames. Since video is a continuous sequence of frames, video segmentation needs to consider not only the image features within a single frame but also the temporal relationships between frames. In video segmentation, each frame can be segmented independently, and then the segmentation results are stitched together to improve the accuracy and coherence of the segmentation. Video segmentation is widely used in video editing, content analysis, detection, and autonomous driving.

[0052] 4) Trajectory: This refers to the spatial-temporal path an object follows from one frame to another in a video. This path reflects the object's dynamic behavior and positional changes within the video sequence. A trajectory typically consists of a series of points, each representing the object's position within a specific time frame. In computer vision and video analytics, trajectory detection and analysis are crucial for understanding object motion patterns, behavioral characteristics, and interaction relationships.

[0053] 5) Multimodal Large Language Models (MLLMs): MLLMs are a class of pre-trained language models capable of processing and understanding multiple modal inputs (such as text, images, and audio). By combining information from different modalities, these models can more comprehensively understand and generate complex, cross-modal content. Compared to traditional unimodal language models, MLLMs have the following characteristics: Multimodal Input Processing: MLLMs can accept and process different types of input data; for example, they can simultaneously receive text descriptions and images as input. Cross-Modal Information Fusion: The model has internal mechanisms to integrate information from different modalities, thereby utilizing comprehensive contextual information when understanding and generating content. Complex Task Processing: MLLMs can typically handle more complex tasks, such as image description generation, visual question answering, and video understanding, which require understanding both visual and textual information simultaneously. Enhanced Representation Capabilities: By fusing information from different modalities, MLLMs can generate richer and more accurate representations, which is very helpful in improving the model's performance in various NLP tasks. Multimodal large language models (MLLMs) are an important research direction in the field of natural language processing. Their applications are wide-ranging, including but not limited to intelligent assistants, content generation, information retrieval, automatic summarization, and sentiment analysis. With continuous technological advancements, MLLMs are expected to play an increasingly important role in understanding human communication and interaction.

[0054] 6) Target Segmentation: A term in computer vision and image processing, referring to the process of separating a specific target object from the background in an image or video frame. This process involves identifying and extracting regions of interest, i.e., targets, in an image for further analysis or processing.

[0055] During the implementation of the embodiments of this application, the applicant discovered the following problems with the related technology:

[0056] In related technologies, for target segmentation in video, the target is usually segmented independently for each video frame, and then the segmentation results of each video frame are stitched together to obtain the video segmentation result. Since the objects in different video frames have trajectory continuity, segmenting the video frame independently in this way will lead to low accuracy of target segmentation in video.

[0057] This application provides a video target segmentation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of video target segmentation. The following describes an exemplary application of the video target segmentation system provided in this application.

[0058] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video target segmentation system 100 provided in the embodiments of this application. The terminal (terminal 400 is shown as an example) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0059] Terminal 400 is used by a user to access client 410 and display the target segmentation results on a graphical interface 410-1 (graphical interface 410-1 is shown as an example). Terminal 400 and server 200 are interconnected via wired or wireless network.

[0060] In some embodiments, server 200 can be a standalone physical server, a server cluster or business system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smart TV, smartwatch, in-vehicle terminal, etc., but is not limited to these. The electronic device provided in this application embodiment can be implemented as a terminal or a server. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment.

[0061] In some embodiments, server 200 acquires the frame features of keyframes and the trajectory features of target objects, fuses the frame features and trajectory features to obtain fused features, performs target segmentation on keyframes based on fused features to obtain a first segmentation result of keyframes, performs target segmentation on non-keyframes based on the first segmentation result to obtain a second segmentation result, determines the target segmentation result based on the first segmentation result and the second segmentation result, and sends the target segmentation result to terminal 400.

[0062] In other embodiments, terminal 400 acquires the frame features of key frames and the trajectory features of target objects, fuses the frame features and trajectory features to obtain fused features, performs target segmentation on key frames based on the fused features to obtain a first segmentation result of key frames, performs target segmentation on non-key frames based on the first segmentation result to obtain a second segmentation result, determines the target segmentation result based on the first segmentation result and the second segmentation result, and sends the target segmentation result to server 200.

[0063] See Figure 2 , Figure 2This is a schematic diagram of the structure of an electronic device 500 for target segmentation of video provided in an embodiment of this application, wherein, Figure 2 The electronic device 500 shown can be Figure 1 Server 200 or terminal 400 in the middle, Figure 2 The illustrated electronic device 500 includes at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0064] Processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 430.

[0066] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0067] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0068] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0069] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, such as Bluetooth, WiFi, and Universal Serial Bus.

[0070] In some embodiments, the video target segmentation apparatus provided in this application can be implemented in software. Figure 2 A target segmentation apparatus 455 for video stored in memory 450 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: acquisition module 4551, fusion module 4552, segmentation module 4553, and determination module 4554. These modules are logically linked and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0071] In other embodiments, the video target segmentation device provided in this application can be implemented in hardware. As an example, the video target segmentation device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video target segmentation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] In some embodiments, the terminal or server can implement the video target segmentation method provided in this application by running a computer program or computer-executable instructions. For example, the computer program can be a native program in the operating system (e.g., a dedicated target segmentation program) or a software module, such as a target segmentation module that can be embedded in any program (e.g., an instant messaging client, a photo album program, an electronic map client, a navigation client); for example, it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run. In summary, the above-mentioned computer program can be any form of application, module, or plugin.

[0073] The video target segmentation method provided in this application will be described in conjunction with exemplary applications and implementations of the server or terminal provided in the embodiments of this application.

[0074] See Figure 3 , Figure 3 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 1 , will combine Figure 3 Steps 101 to 105 are described below. The video target segmentation method provided in this application embodiment can be implemented by the server or the terminal alone, or by the server and the terminal working together. The following description will take the implementation by the server alone as an example.

[0075] In step 101, the frame features of key frames in the video to be processed are obtained, and the trajectory features of the target object in the video are obtained.

[0076] In some embodiments, a keyframe, also known as an I-frame (Intra-coded frame), is a frame in a video sequence that contains complete image data of the video scene. In video compression, keyframes are typically used as reference frames for comparison and compression with other frames. Keyframes have the following characteristics: Self-containment: A keyframe contains enough information to be decoded independently without relying on other frames. Completeness: A keyframe typically represents a clear, complete frame in the video. Compression ratio: Because keyframes contain complete image data, their compression ratio is relatively low.

[0077] In some embodiments, non-keyframes include P-frames (Predicted frames) and B-frames (Bidirectionally-predicted frames). These are transitional frames in a video sequence used to improve compression efficiency. Non-keyframes lack the completeness and self-containment of keyframes. Here are the two types of non-keyframes: P-frames: P-frames are forward-predicted frames that compress data by comparing the current frame with previous keyframes or P-frames. P-frames record these differences, so previous frames are referenced during decoding. B-frames: B-frames are bidirectionally-predicted frames that compress data by referencing not only preceding frames but also following frames. B-frames typically offer higher compression ratios but require more computational resources for decoding. Non-keyframes reduce redundant information through prediction and interpolation techniques, thus achieving efficient compression of video data. Here are their characteristics: Dependency: Non-keyframes depend on other frames (usually keyframes) for correct decoding. Compression Ratio: Non-keyframes typically have higher compression ratios than keyframes because they only record the changing parts. Transition: Non-keyframes are transitional frames in a video sequence, used to connect keyframes and make video playback smooth.

[0078] In some embodiments, the above-mentioned acquisition of frame features of keyframes in the video to be processed can be achieved in the following manner: acquiring object parameters of the target object in each video frame of the video, wherein the target object parameters include at least one of the motion parameters of the target object and the screen proportion of the target object in the video frame; when the value of the object parameter of the target object in the video frame is greater than the parameter threshold, the video frame is determined as a keyframe; and feature extraction is performed on the keyframe to obtain the frame features of the keyframe.

[0079] In some embodiments, a video frame is the basic unit that constitutes a video, and each frame is a still image within the video. An object refers to a specific object within a video frame, such as a person, a vehicle, or a particular object. Object parameters are various parameters used to describe the characteristics of an object in the video. Object parameters mentioned in the description include: Motion parameters: describing the object's motion in the video, such as speed, acceleration, and direction of motion. Frame occupancy: this refers to the proportion of area occupied by the object within the video frame, which can be used to represent the object's size or importance. Parameter threshold is a preset numerical standard used to determine whether the object parameters are large enough to identify a video frame as a keyframe. Keyframes are important frames in a video sequence, typically containing key visual information about the scene, and can be used as reference frames when compressing the video.

[0080] In some embodiments, object parameters of specific objects in a video frame are analyzed and extracted. These parameters may include the object's motion parameters and its proportion within the frame. These object parameters are then checked to see if they exceed a preset parameter threshold. The parameter threshold is a pre-defined standard used to determine when an object is significant enough to warrant designating its frame as a keyframe. If the object parameter values ​​in a video frame exceed the parameter threshold, that video frame is designated as a keyframe. This means that the frame contains important visual information, potentially representing significant motion or change of an object, allowing for the automatic identification and treatment of frames containing important object activity as keyframes.

[0081] In some embodiments, when the value of the object parameter of the target object in the video frame is not greater than the parameter threshold, the video frame is determined to be the non-key frame.

[0082] In some embodiments, the server analyzes each video frame and extracts object parameters for specific objects within the frame. The server compares the extracted object parameter values ​​with preset parameter thresholds. Non-keyframes are identified: if the value of the object parameter in a video frame is not greater than (i.e., less than or equal to) the parameter threshold, the server determines that the video frame is a non-keyframe. This means that the object in the frame does not have significant motion or does not occupy a sufficiently large proportion of the screen, so it does not contain enough important information to be considered a keyframe. Through this determination, the server can effectively classify video frames into keyframes or non-keyframes, which is crucial for video compression and video content management. Non-keyframes typically contain less information variation, so more efficient compression algorithms can be used to reduce data volume, while keyframes, because they contain important information, require more precise encoding to maintain quality.

[0083] This allows for the precise identification of important frames, or keyframes, in the video, which is crucial for video content understanding, retrieval, and efficient storage. Identifying objects with parameter values ​​exceeding a set threshold as keyframes ensures that the extracted information represents significantly changed or important parts of the video content, thereby improving the targeting and efficiency of video analysis. Feature extraction from keyframes yields more specific and useful video feature information, providing effective data support for subsequent video processing tasks such as object detection, behavior recognition, and content classification. Feature extraction not only improves the accuracy of recognition and classification but also reduces the computational load of subsequent processing. Identifying frames that do not meet the parameter thresholds as non-keyframes helps handle redundant information and optimizes resource utilization. This selective processing strategy allows the system to focus on key information while ignoring frames with minimal impact on the overall analysis, thus improving processing speed and system efficiency while ensuring analysis quality.

[0084] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 2 , Figure 3 The step 101 shown illustrates obtaining the trajectory features of the target object in the video, which can be achieved through... Figure 4 Steps 1011 to 1012 shown are implemented.

[0085] In step 1011, the initial description information of the video is obtained, which is used to describe the behavior of the target object.

[0086] In some embodiments, the behavior of a target object refers to the specific actions and activity patterns exhibited by a particular object (such as a person, vehicle, animal, etc.) in a video or observed scene. This includes the target object's movement, stillness, interaction, and any other observable actions. In video analytics, the behavior of a target object typically involves the recognition and understanding of features such as the object's motion, posture, gestures, and facial expressions. For example, in a video, a person may exhibit behaviors such as walking, running, or waving.

[0087] In some embodiments, the trajectory of a target object in a video refers to the object's movement path within the video over a period of time. This path can consist of a series of location points, which can be used to depict the object's movement route. In the field of video detection, motion trajectories help analyze the dynamic behavior of objects, such as direction of travel, speed, and movement patterns. The description of motion trajectories is crucial for understanding and predicting the future behavior of objects; it can help determine the object's intentions and patterns, such as patrolling, chasing, or escaping.

[0088] As an example, see Figure 9 , Figure 9 This is a schematic diagram illustrating the principle of the video target segmentation method provided in the embodiments of this application. Figure 2 The initial description information for the video can be: (text to trajectory) Can you segment out the kitten that is bullying the other kitten in this video? The initial description information is used to describe the behavior of the object.

[0089] In step 1012, the trajectory features of the target object are obtained based on the video and initial description information.

[0090] In some embodiments, initial descriptive information typically refers to a preliminary summary of the behavior of objects in the video content. This description is often brief, containing only the most basic information. To describe the behavior and movement trajectory of objects more comprehensively and in detail, the initial descriptive information needs to be expanded. Initial descriptive information is the foundation of video analysis; it typically includes the object's type, behavior category, or the basic direction of its movement trajectory. Initial descriptive information is often insufficient to provide enough information to gain a deeper understanding of the object's behavior or movement trajectory, thus requiring expansion. Expanded descriptive information can make the analysis more precise and helps in understanding the specific behavior and movement trajectory of the object.

[0091] As an example, the initial description information describes the behavior. This part details the object's behavioral characteristics, including the specific actions, state, and duration of the behavior. For example, expanding "a person running" to "a person running fast, leaning forward, swinging their legs at a high frequency, and with a focused expression." The trajectory information describes the motion trajectory. This part describes the object's movement path in the video, including the start point, end point, direction, speed, and acceleration. For example, expanding "a person running" to "a person running from the left, running along a straight track to the right, gradually increasing their speed."

[0092] In some embodiments, after receiving the initial description information, the server needs to predict trajectory features based on the video and the initial description information. The server first receives the video stream and the corresponding initial description information. The initial description information may be preliminary information input by the user or automatically extracted by the server. The server needs to identify whether the initial description information is used to describe behavior.

[0093] As an example, the initial description information describes the behavior of the target object; the initial description information is: a person is swimming. The server receives a video stream containing a person swimming. Based on the initial description information, the server initiates the behavior analysis module. The server identifies the swimmer's movements using computer vision technology and determines the swimming path through motion trajectory analysis. Trajectory features: The features of the swimmer's predicted trajectory obtained by the server, such as the length of the swimming pool, the direction of swimming (back and forth or one-way), and the swimming speed.

[0094] Thus, by combining initial descriptive information with video content to predict trajectory features, this process significantly improves the richness and accuracy of descriptive information. Introducing detailed information about movement trajectories when describing object behavior not only makes the description more comprehensive but also helps users understand the context in which the behavior occurs, such as the speed, direction, and path of the movement, thereby providing deeper insights.

[0095] In some embodiments, step 1012 above can also be implemented as follows: based on the initial description information, a prompt is obtained, the prompt being used to instruct the trajectory recognition model to output the trajectory features of the target object based on the initial description information; based on the video and the prompt, the trajectory of the target object in the video is recognized by the trajectory recognition model to obtain the trajectory features of the target object.

[0096] In some embodiments, prompts are obtained based on initial descriptive information. These prompts are then converted into a format understandable by the trajectory recognition model to guide the model in outputting trajectory features of the target object. These prompts can be specific codes, labels, or natural language instructions, containing key information about the target object's behavior and movement. Using these prompts and raw video data as input, the trajectory recognition model identifies the target object's trajectory in the video.

[0097] In some embodiments, the trajectory recognition model described above can be a large language model, which refers to a deep learning model trained with a large amount of text data, containing billions or even more parameters. It can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical rules and semantic relationships of language to build a large language knowledge base, thereby simulating human language understanding and generation capabilities.

[0098] As an example, the trajectory recognition model described above includes an encoding layer and a decoding layer. Based on video and prompts, the trajectory recognition model identifies the trajectory of the target object in the video and obtains the trajectory features of the target object. This can be achieved as follows: extract features from the video to obtain video features, extract features from the prompts to obtain prompt features, call the encoding layer to encode the prompt features and video features to obtain target features, and call the decoding layer to decode the target features to obtain trajectory features.

[0099] In some embodiments, prior to step 1012 above, the following processing may also be performed: obtaining a first sample prompt for the video sample, wherein the first sample prompt is obtained based on the first sample description information and is used to instruct the initial trajectory recognition model to output the sample trajectory features of the sample target object based on the first sample description information, wherein the first sample description information is used to describe the behavior of the sample target object in the video sample; based on the video sample and the first sample prompt, identifying the trajectory of the sample target object in the video sample through the initial trajectory recognition model to obtain the sample trajectory features of the sample target object; and training the initial trajectory recognition model based on the sample trajectory features to obtain the trajectory recognition model.

[0100] In some embodiments, a prompt is extracted or generated from the video sample, based on first sample description information. The first sample description information describes the behavior of the target object in the video sample. The purpose of the prompt is to instruct the initial trajectory recognition model how to output the sample trajectory features of the target object based on the first sample description information. Using the initial trajectory recognition model, combined with the video sample and the first sample prompt, the trajectory of the target object in the video is identified. The recognition result is the sample trajectory features of the target object, which may be trajectory-related information such as position, velocity, and acceleration. The obtained sample trajectory features are used to train the initial trajectory recognition model to improve its accuracy in identifying trajectories. The trained model is called the trajectory recognition model, and it should be able to more accurately identify the trajectory of the target object in the video. The entire process is a supervised learning process, where the video sample and the sample prompt serve as training data to train the initial trajectory recognition model, enabling it to learn how to identify the trajectory features of the target object from the video. The trained trajectory recognition model can then be used to perform trajectory recognition on new video data.

[0101] In some embodiments, the first sample cue refers to an instruction or description used to guide the initial trajectory recognition model on how to identify and process target objects in video samples. The cue is derived from the first sample description information; that is, the behavior of the target object in the video sample must first be described, and this description information is used to generate the cue. The cue guides the initial trajectory recognition model to identify the sample trajectory features of the target object. Sample behavior features refer to the behavior patterns of the target object in the video, such as movement, stillness, acceleration, etc. Sample trajectory features refer to the motion path and trajectory of the target object in the video. Based on the video samples and the first sample cue, the initial trajectory recognition model identifies the trajectory of the target object in the video samples. The initial trajectory recognition model is a preliminary model that uses the sample cue as input and outputs sample trajectory features. Based on the identified sample trajectory features, the initial trajectory recognition model is trained. By continuously adjusting the model parameters, the model can more accurately identify and predict the trajectory of the target object in the video samples.

[0102] In some embodiments, since the first sample prompt used in the training phase is obtained based on the first sample description information and is used to instruct the initial trajectory recognition model to output the sample trajectory features of the sample target object based on the first sample description information, after the initial trajectory recognition model is trained and the trajectory recognition model is obtained, only the trajectory features obtained by the trajectory recognition model are used as the input of the subsequent fusion process, and the frame features of the key frame are fused with the trajectory features to obtain the fused features.

[0103] As an example, suppose we have a video sample depicting a dog running, jumping, and playing in a park. The first sample description could be: "A dog is running, jumping, and playing in the park in the video." Based on this first sample description, a first sample prompt can be generated, such as: "Based on the dog's behavior of running, jumping, and playing in the park in the video, output the dog's sample behavioral features and sample trajectory features." Then, an initial trajectory recognition model can be used, based on this first sample prompt and the video sample, to identify the dog's trajectory in the video, obtaining the dog's sample trajectory features. Finally, based on these sample trajectory features, the initial trajectory recognition model can be trained to obtain a more accurate trajectory recognition model.

[0104] Thus, by acquiring the first sample prompts from video samples and using the initial trajectory recognition model to identify the trajectory features of the target object, the accuracy of trajectory recognition can be improved. Furthermore, training the initial trajectory recognition model based on the sample trajectory features can further optimize model performance and enhance its ability to recognize the trajectory of the target object.

[0105] In some embodiments, prior to step 1012 above, the following processing may also be performed: obtaining a second sample prompt for the video sample, wherein the second sample prompt is obtained based on the location information of the sample target object in the video sample, and is used to instruct the initial trajectory recognition model to output the sample behavior features of the sample target object based on the location information, wherein the location information is used to indicate the position of the sample target object in multiple video frames in the video sample; based on the video sample and the second sample prompt, identifying the behavior of the sample target object in the video sample through the initial trajectory recognition model to obtain the sample behavior features of the sample target object; and training the initial trajectory recognition model based on the sample behavior features to obtain the trajectory recognition model.

[0106] In some embodiments, both the first sample prompt and the second sample prompt instruct the initial trajectory recognition model to output sample behavior features and sample trajectory features. After training the initial trajectory recognition model is completed, the obtained trajectory recognition model will also output trajectory features and behavior features. During the inference phase, only the trajectory features are used for subsequent fusion with the frame features of the keyframes, and the behavior features are not used for subsequent fusion processes.

[0107] In some embodiments, location information of the target object is extracted from the video samples, and a second sample prompt is generated based on this location information. This location information may be the target object's coordinates or other forms of location description in the video, indicating the target object's positional changes across multiple video frames. The trajectory of the target object in the video samples is identified using an initial trajectory recognition model and the second sample prompt. This model outputs sample behavioral features of the target object based on the location information. Behavioral features may include the target object's actions, speed, acceleration, etc., while trajectory features may include the target object's movement path, direction, etc. Based on the identified sample behavioral features, the initial trajectory recognition model is trained to optimize its performance. This training process may involve adjusting the model's parameters to more accurately identify the target object's trajectory and behavioral features. By training a trajectory recognition model, it is possible to accurately identify the trajectory and behavioral features of the target object in the video.

[0108] In some embodiments, the second sample prompt is generated based on the location information of the target object in the video sample. The main function of the second sample prompt is to instruct the initial trajectory recognition model how to output the target object's sample behavioral features and sample trajectory features based on the location information. Location information refers to the specific position of the target object within multiple video frames of the video sample. Location information is crucial for recognizing the target object's motion trajectory. The video sample and the second sample prompt are input into the initial trajectory recognition model. The initial trajectory recognition model identifies the target object's trajectory based on the location information in the video sample. During trajectory recognition, the initial trajectory recognition model simultaneously outputs the target object's sample behavioral features. The extracted sample behavioral features are used to train the initial trajectory recognition model. The accuracy of the location information directly affects the quality of trajectory recognition; therefore, the generation of the second sample prompt must accurately reflect the target object's positional changes in the video. Through training based on sample behavioral features, the model can better understand trajectory features under different behavioral patterns, thereby improving recognition accuracy.

[0109] In some embodiments, the construction of the second sample prompt can involve directly incorporating location information into the prompt, extracting the location information of the target object across multiple video frames from the video samples. This location information may include the target object's coordinates, bounding box dimensions, etc. The extracted location information is then directly converted into text or numerical form to form the second sample prompt. For example, the target object's coordinates and bounding box dimensions can be included as part of the prompt. The constructed second sample prompt is then input into the initial trajectory recognition model, instructing the model to output the sample behavior features of the target object based on the location information.

[0110] In some embodiments, the construction process of the second sample prompt may involve extracting local region images from an image and using the features of the local images to construct the prompt; extracting the position information of the target object in multiple video frames from a video sample; and cropping a local region image containing the target object from the video frames based on the extracted position information. Feature extraction is performed on the local region image to obtain its visual features. The extracted visual features are then converted into a format suitable for model input to form the second sample prompt. These features may be in vector form and contain information such as the appearance and posture of the target object. The constructed second sample prompt is input into the initial trajectory recognition model, instructing the model to output the sample behavior features of the target object based on these features.

[0111] As an example, see Figure 10 The construction process of the second sample prompt can be as follows: extract a local region image 32 from image 31 and use the features of the local region image 32 to construct the prompt; extract the position information of the target object in multiple video frames from the video sample; based on the extracted position information, crop the local region image 32 containing the target object from the video frames; perform feature extraction on the local region image 32 to obtain the visual features of the local image; and convert the extracted visual features into a format suitable for model input to form the second sample prompt (e.g., such as...). Figure 10 The second sample prompt 33 and the second sample prompt 34 are shown.

[0112] As an example, suppose there is a video containing multiple pedestrians. The goal is to identify the trajectories of these pedestrians. First, the location information of each pedestrian needs to be extracted from the video. This can be achieved using computer vision techniques, such as object detection algorithms. Location information is typically represented by the bounding box coordinates of the target object in each video frame. Second sample prompts are generated based on the location information of the sample target objects in the video samples. These prompts can be natural language descriptions describing the changes in the target object's position in the video, or structured data encoding the location information. For example, for a pedestrian, the second sample prompt could be: "The pedestrian moves from the top left corner to the bottom right corner, gradually increasing speed." An initial trajectory recognition model is used to identify the trajectories of the sample target objects in the video samples. This model might be a deep learning-based model, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), which can handle time-series data, such as location information. The model's input is the video samples and the second sample prompts, and the output is the sample behavior features and sample trajectory features of the sample target objects. The initial trajectory recognition model is trained based on the sample behavior features to optimize its performance. This typically involves defining a loss function that measures the difference between the trajectory features predicted by the model and the actual trajectory features. Using the backpropagation algorithm, the model's parameters can be adjusted to minimize the loss function, thereby improving the model's accuracy.

[0113] As an example, suppose there is an intelligent detection system used to detect a busy intersection. Our goal is to identify and analyze the trajectories of pedestrians and vehicles to improve the efficiency and safety of traffic management. First, video samples are acquired from detection cameras. Then, object detection algorithms (such as YOLO or SSD) are used to extract the positional information of pedestrians and vehicles in the video. This positional information is represented by the bounding box coordinates of the target objects in each video frame. Based on the extracted positional information, we generate second-sample prompts. These prompts can be natural language descriptions describing the changes in the target object's position in the video, or structured data encoding the positional information. For example, for a pedestrian, the second-sample prompt could be: "The pedestrian moves from the top left corner to the bottom right corner, gradually increasing speed." An initial trajectory recognition model is used to identify the trajectories of sample target objects in the video samples. The initial trajectory recognition model is trained based on sample behavioral features to optimize its performance, measuring the difference between the model's predicted trajectory features and the actual trajectory features. Through backpropagation, we can adjust the model's parameters to minimize the loss function, thereby improving the model's accuracy. A trajectory recognition model capable of accurately identifying the trajectories of pedestrians and vehicles in videos can be trained.

[0114] Thus, by acquiring second-sample prompts based on location information, the initial trajectory recognition model can be more effectively guided to learn the behavioral and trajectory features of the target object. This not only improves the model's accuracy in recognizing the target object's trajectory but also enhances the model's adaptability to complex scenes.

[0115] In some embodiments, a trajectory recognition model is a machine learning model used to analyze and predict the motion trajectory of objects in a video. This model learns object features and time-series information from video frames to understand the dynamic behavior of objects and infer the continuity of trajectories between video frames. Sample description information is a textual description or feature vector representation of the object's behavior or trajectory in the video sample. It is typically manually labeled or automatically extracted to provide semantic information or quantitative descriptions of the object's behavior or trajectory. Video samples refer to short segments extracted from a video dataset for model training, validation, or testing. Video samples typically contain a sufficient number of frames to demonstrate the object's behavior or trajectory patterns. Trajectory recognition information is descriptive information about the object's motion trajectory extracted from the video sample by the trajectory recognition model. This information may include the object's position, velocity, acceleration, direction of motion, etc., used to characterize the object's motion state and trajectory features. Label recognition information is manually labeled or predefined label or classification information about the object's trajectory associated with the video sample. This information is typically used to describe the type of an object's trajectory (such as linear motion, curvilinear motion, etc.) or specific behavior (such as walking, running, jumping, etc.), and is used to guide model training during supervised learning.

[0116] Thus, by generating prompts based on initial descriptive information and using a trajectory recognition model to identify the trajectory features of the target object based on these prompts and video data, the accuracy and efficiency of trajectory recognition can be significantly improved. This not only enhances the model's ability to understand the behavior of the target object but also improves its ability to predict the movement trajectory of the target object in complex scenes.

[0117] In step 102, the frame features of the keyframe are fused with the trajectory features to obtain fused features.

[0118] In some embodiments, the frame features of the keyframe include multiple subframe features, see [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 3 , Figure 3 Step 102 shown can be achieved through Figure 5 Steps 1021 to 1022 shown are implemented.

[0119] In step 1021, the similarity between the features of each subframe of the keyframe and the trajectory features is determined.

[0120] In some embodiments, keyframes are representative frames extracted from a video sequence, typically containing important scene changes or actions within the video. Subframe features refer to features of further subdivided regions or time segments within a keyframe. Feature extraction is performed on each subframe of the keyframe. These features may include low-level features such as color, shape, texture, and motion vectors, or higher-level semantic features such as object category and behavior labels. The similarity between each subframe feature and trajectory features is calculated. Trajectory features may be vectors obtained from the initial descriptive information of the video through feature extraction. Similarity calculation can use various metrics, such as cosine similarity, Euclidean distance, and Jaccard similarity, to measure the degree of similarity between two feature vectors. The similarity results are used as weights to reflect the importance of each subframe feature to the overall keyframe features. High similarity means that the subframe feature is highly correlated with the trajectory feature, and therefore should be given higher weights during the fusion process. The calculated similarity weights are used to perform weighted fusion of the subframe features. This process can be implemented through weighted averaging or other weighting methods. The purpose of weighted fusion is to generate a comprehensive feature vector that better represents the content of the keyframe and highlights the parts most relevant to the trajectory features.

[0121] In some embodiments, the method for determining subframe features can be selected based on different application requirements and the characteristics of the video content. For example, the keyframe can be divided into a fixed number of sub-regions, and features are extracted from each sub-region as subframe features. For instance, the keyframe can be divided into a 4x4 grid, with each small region being a subframe. Another example is using image segmentation techniques, such as threshold-based segmentation, edge detection, and region growing, to automatically determine sub-regions based on the visual characteristics of the keyframe content. If the keyframe is a series of frames in a video, frames can be selected as subframe features at fixed time intervals (e.g., one frame per second). Yet another example is using object detection algorithms to identify objects in the keyframe; each detected object and its surrounding area can serve as a subframe feature. By detecting points of interest (e.g., corners, edges) in the keyframe, sub-regions are defined centered on these points, and features are extracted as subframe features. Yet another example is dividing the keyframe according to different levels, such as first dividing it into large regions and then further subdividing each large region to form hierarchical subframe features. Features of the keyframe can be extracted at different scales, such as using a pyramid method to downsample the keyframe and then extracting features at different scales.

[0122] In some embodiments, subframe features refer to a set of local features extracted from a keyframe. These features typically represent visual or semantic information about a specific region (spatial subframe) or a specific time segment (temporal subframe) within the keyframe. Subframe features can be spatial subframe features, temporal subframe features, object subframe features, scene subframe features, etc. Spatial subframe features refer to features extracted from a specific spatial region in the keyframe. These regions can be predefined grids, regions automatically determined by image segmentation techniques, or regions defined based on content (such as objects or scenes). Temporal subframe features refer to features extracted from consecutive time segments in the keyframe. These features reflect the dynamic changes in video content over a short period of time and are typically used to capture motion or behavioral patterns in the video. Object subframe features refer to features extracted from objects or portions of objects detected in the keyframe. These features help identify and classify specific objects or object behaviors in the video. Scene subframe features refer to features extracted from regions in the keyframe representing different scenes or backgrounds. These features help distinguish different scene types or environments in the video.

[0123] In step 1022, the features of each subframe and the trajectory features are weighted and fused together using the similarity as the weight to obtain the fused features corresponding to the keyframe.

[0124] In some embodiments, keyframes are extracted from a video sequence and contain important visual information or events in the video. Subframe features refer to the features extracted from each unit after the keyframe is further divided into smaller spatial or temporal units. These features can be low-level features such as color, shape, texture, and motion, or high-level semantic features such as objects and behaviors. For each subframe feature, its similarity to a predefined trajectory feature is calculated. Trajectory features are typically extracted from the video's description or tags and represent important content and semantics of the video. Similarity calculation can use various metrics, such as cosine similarity, Euclidean distance, and Jaccard similarity, to measure the degree of similarity between two feature vectors. The calculated similarity is used as a weight, which reflects the degree of association between each subframe feature and the trajectory feature; a high similarity means that the subframe feature is highly correlated with important information in the video. The subframe features are then weighted and fused using the similarity weights. This is typically achieved through weighted averaging or other weighting methods. The purpose of weighted fusion is to generate a comprehensive keyframe feature vector that highlights the subframe features most relevant to the trajectory feature.

[0125] As an example, suppose we have a video of a person running in a park, and we want to analyze and extract keyframe features from the video to identify and describe this scene. By analyzing the video content, a keyframe is identified that captures the exact moment the runner starts running from a standstill. Subframe feature extraction: The keyframe is divided into several subframes, for example, four regions: the runner's head, torso, legs, and background. Features are extracted for each region (subframe), for example: Head subframe features: color, shape (face outline), movement (slight head nod). Torso subframe features: color, shape (body outline), movement (leaning forward). Leg subframe features: color, shape (leg outline), movement (running posture). Background subframe features: color, texture (pattern of grass). Based on the video description, a feature vector is defined, representing the semantic features of a person running in a park, which may include keywords such as person, movement, park, running, etc. The similarity between each subframe feature and the trajectory features is calculated. For example, cosine similarity is used to measure the correlation between feature vectors. Assume the following similarity scores are obtained: head subframe feature similarity is 0.7, torso is 0.8, legs are 0.9, and background is 0.5. Using the calculated weights, a weighted fusion is performed on the features of each subframe. A simple fusion method is a weighted average: Fusion Feature = (Head Feature * 0.7 + Torso Feature * 0.8 + Leg Feature * 0.9 + Background Feature * 0.5) / (0.7 + 0.8 + 0.9 + 0.5). In this example, leg features have the largest weight in the fusion feature because they are most relevant to the trajectory features. The final fusion feature is a new feature vector that integrates the features of each subframe, more accurately representing the content of the keyframe, especially emphasizing the leg features most relevant to running behavior.

[0126] In other embodiments, the fused features correspond one-to-one with the keyframes. When the number of keyframes is N, the keyframes include the i-th keyframe, where N is an integer greater than 1. The above step 102 can also be implemented in the following way: traverse i and perform the following processing: fuse the frame features of the i-th keyframe with the trajectory features to obtain the fused features of the i-th keyframe.

[0127] In some embodiments, each keyframe has a corresponding set of fused features, obtained by combining the visual features (frame features) of the keyframe with descriptive information (trajectory features) of the video. This correspondence ensures that the segmentation and analysis of each keyframe is based on its specific fused features, improving the targeting and accuracy of the processing. The description mentions that there are N keyframes in total, where N is an integer greater than 1, indicating that the video contains at least two keyframes. The i-th keyframe is identified by its index i, which iterates through all keyframes from 1 to N during processing. Processing starts from the first keyframe and proceeds sequentially to the Nth keyframe, ensuring that each keyframe is processed. Visual features, such as color, shape, and texture, are extracted from the keyframes. Descriptive information associated with the keyframes is obtained, which may be extracted from the video's metadata, narration, title, or automatically generated descriptions. The extracted frame features are fused with the trajectory features. Fusion can be achieved in various ways, such as using feature concatenation, feature weighting, multimodal learning models, or deep learning frameworks. Feature fusion results in fused features for each keyframe. These fused features incorporate both visual and non-visual information, providing a more comprehensive data foundation for subsequent video analysis tasks. Fusion features can more accurately reflect objects and scenes within keyframes, thereby improving performance in tasks such as object segmentation, recognition, and detection.

[0128] In this way, by determining the similarity between the features of each subframe of the keyframe and the trajectory features, and then using these similarities as weights for weighted fusion, the accuracy and representativeness of the keyframe feature representation are significantly improved. By calculating similarity, the subframe features most relevant to the trajectory features can be identified, thus highlighting this important information in the keyframe and filtering out noise or irrelevant information. Weighted fusion ensures that the feature vector of the keyframe not only contains the core information of the video content, but also that the contribution of each subframe feature is proportional to its correlation with the trajectory features. This makes the final fused features provide more accurate and robust input when used for tasks such as video recognition, classification, or retrieval, thereby improving the overall performance of the model and the reliability of predictions.

[0129] In step 103, the keyframe is segmented based on the fusion features to obtain the first segmentation result of the keyframe.

[0130] In some embodiments, target segmentation, a term in the fields of computer vision and image processing, refers to the process of separating a specific target object from the background in an image or video frame. This process involves identifying and extracting regions of interest, i.e., targets, from an image for further analysis or processing.

[0131] In some embodiments, see Figure 6 , Figure 6This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 4 , Figure 3 Step 103 shown can be achieved through Figure 6 Steps 1031 to 1032 shown are implemented.

[0132] In step 1031, the frame features of the keyframe are encoded to obtain the first encoded features.

[0133] As an example, see Figure 9 (b) Call the key prompt encoder ( Figure 9 (b) The target marker shown is used to encode the frame features of the keyframe to obtain the first encoded feature.

[0134] In some embodiments, the main function of the key hint encoder is to encode frame features using the fused features of the input keyframes to obtain a new feature representation, namely the first encoded feature. The key hint encoder receives frame features as input. The encoder encodes the original frame features of the keyframes, a process that includes feature selection, feature extraction, feature transformation, or dimensionality reduction, with the aim of generating a new feature representation. After the encoding process, the key hint encoder outputs the first encoded feature.

[0135] In some embodiments, the frame features of keyframes are encoded to obtain first encoded features, a process of significant importance in video processing and encoding. By extracting and encoding the features of keyframes, redundancy in video data can be effectively reduced, lowering storage and transmission costs while preserving the main information of the video content. Keyframes typically contain the most important information in a video; encoding the features of these keyframes enables efficient video compression, reducing storage space and bandwidth requirements. The encoded first encoded features retain the core information of the keyframes, which is crucial for subsequent video analysis, retrieval, and processing. The encoding process can further process the features of keyframes to enhance their robustness to noise and variations, ensuring good recognition performance under different conditions.

[0136] In some embodiments, the above-mentioned fusion features correspond one-to-one with keyframes; there are multiple keyframes, and the keyframes and non-keyframes in the video are ordered.

[0137] In some embodiments, the above-mentioned encoding of the frame features of the key frame to obtain the first encoded feature can be achieved in the following way: for each key frame, the frame features of the key frame are encoded to obtain the first encoded feature.

[0138] In some embodiments, each keyframe has a corresponding fused feature vector. The fused feature is extracted based on the keyframe, combining features from multiple subframes within the keyframe and weighted with some predefined trajectory features. The fused feature for each keyframe is unique, reflecting the specific visual content and semantic information of that frame. In a video sequence, keyframes are frames containing important visual information or events, which are typically selected for further analysis. There can be multiple keyframes, and the frames in the video (whether keyframes or non-keyframes) are ordered chronologically. This ordering helps in understanding and detecting changes in video content. The fused features of the keyframes are used to encode the frame features. The purpose of encoding is to transform the complex features of the keyframes into a more compact and tractable form while preserving key information.

[0139] In some embodiments, key frames and non-key frames in the video are ordered, and the encoding of frame features of key frames and non-key frames is performed sequentially according to the frame order.

[0140] In other embodiments, after encoding the frame features of the keyframe to obtain the first encoded features, the following processing can also be performed: storing the first encoded features of the keyframe in a feature repository, which is used to store the first encoded features of the keyframe that has been encoded and the second encoded features of the non-keyframe that has been encoded.

[0141] In some embodiments, storing the first encoded features of the keyframe in the feature repository can be achieved as follows: obtaining the remaining storage space of the feature repository; when the remaining storage space is sufficient to store the encoded features of the keyframe, storing the first encoded features of the keyframe in the feature repository; when the remaining storage space is insufficient to store the encoded features of the keyframe, compressing at least one encoded feature in the feature repository whose storage duration is greater than a duration threshold to obtain compressed encoded features; updating the feature repository based on the compressed encoded features, and storing the first encoded features of the keyframe in the updated feature repository.

[0142] In some embodiments, storing the first encoded features of the keyframe in the feature repository can be achieved as follows: obtaining the remaining storage space of the feature repository; when the remaining storage space is sufficient to store the encoded features of the keyframe, storing the first encoded features of the keyframe in the feature repository; when the remaining storage space is insufficient to store the encoded features of the keyframe, deleting at least one encoded feature in the feature repository whose storage duration is greater than a duration threshold, obtaining the deleted encoded features; updating the feature repository based on the deleted encoded features, and storing the first encoded features of the keyframe in the updated feature repository.

[0143] In some embodiments, a feature repository is a data structure or system specifically designed to store and manage coded features generated during video processing. In the field of video analytics, the primary function of a feature repository is to store coded features of keyframes and non-keyframes. These features are extracted from video frames and represent the visual and informational attributes of the video content. The feature repository is responsible for storing the first coded features of keyframes and the second coded features of non-keyframes. The first coded features are typically generated based on the fused features of the keyframes, while the second coded features are generated based on the first coded features and the frame features of the non-keyframes. The feature repository needs to manage its storage space to ensure sufficient space for storing new coded features. Before storing new features, the remaining space in the repository is checked. When the remaining space in the feature repository is insufficient to store new coded features, the stored coded features are compressed or deleted according to a certain strategy (such as a storage duration threshold) to free up space. The compression or deletion process reduces the data size of the features, but should maintain the validity of the features as much as possible. After the compression or deletion operation, the feature repository is updated to reflect the new storage state. The updated repository will contain the compressed or deleted coded features and the first coded features of the newly stored keyframes. The feature repository provides a central storage solution for video analytics tasks, enabling coded features to be shared across multiple processing steps and algorithms, thus improving processing efficiency. It allows features to be quickly retrieved and reused when needed, which is particularly important for real-time video analytics and large-scale video data processing.

[0144] This ensures maximized space utilization efficiency in the feature repository. By monitoring and adjusting storage space in real time, the algorithm intelligently handles situations of insufficient space, avoiding feature data loss or processing interruptions due to space limitations. When storage space is sufficient, the first encoded features of keyframes can be directly stored, maintaining processing continuity. When space is insufficient, compressing or deleting long-unused encoded features not only frees up storage space but also optimizes the storage structure, enabling the feature repository to operate continuously and efficiently. This enhances the robustness and flexibility of the video processing system, providing strong support for real-time analysis and large-scale data processing, while ensuring the efficient and accurate execution of video analysis tasks.

[0145] In step 1032, the first encoded features are decoded based on the fused features to obtain the first segmentation result of the keyframe.

[0146] In some embodiments, key frames and non-key frames in the video are ordered, and target segmentation of key frames and non-key frames is performed sequentially according to the frame order.

[0147] In some embodiments, the first encoded feature is generated based on fused features, which integrates key information from keyframes, thus enabling more accurate identification and separation of foreground and background targets during object segmentation. Since the dimensionality of the first encoded feature is reduced, the computational complexity of object segmentation is also reduced, making the segmentation process more efficient, especially suitable for applications requiring real-time processing. The dimensionality-reduced feature vector reduces the resource requirements for data processing, including computing power and storage space, which is particularly beneficial for resource-constrained devices (such as mobile devices). The first encoded feature helps improve the performance of the segmentation model and enhances the distinction between foreground and background by removing redundant information and reducing noise. The optimized feature vector allows the segmentation model to have better generalization ability when encountering new data, reducing the risk of overfitting. The first encoded feature focuses more on key information, which helps interpret the segmentation results and makes the model's decisions more transparent. Based on the first encoded feature, the segmentation algorithm can more clearly identify foreground targets (such as people or objects in a video) and background targets (such as static trees or buildings).

[0148] In some embodiments, the ordering of video frames ensures the orderly segmentation of the target, meaning that the processing can proceed sequentially through each frame in chronological order. This maintains the logical order while potentially improving overall computational efficiency through pipelined processing. Since keyframes typically contain the main events or important information in the video, prioritizing target segmentation on keyframes can quickly capture crucial information. The first encoded feature, while preserving the core information of the keyframes, undergoes dimensionality reduction, enabling more accurate identification and segmentation of foreground and background targets, thus improving the quality and accuracy of segmentation. Segmenting in frame order ensures the temporal continuity and spatial consistency of the video content, helping to maintain consistent segmentation standards across multiple frames and preventing abrupt changes in segmentation results between consecutive frames. For non-keyframes, which may contain less important information, simpler segmentation strategies or lower resolutions can be used, optimizing the allocation of computational resources. Depending on the video content, the segmentation strategies for keyframes and non-keyframes can be flexibly adjusted. For example, in videos with frequently changing scenes, the detection frequency of keyframes can be increased, while in videos with relatively static scenes, the detection frequency of keyframes can be reduced.

[0149] In some embodiments, after step 1032 above, the following process may also be performed: storing the first segmentation result in a result library, which is used to store the first segmentation result of the key frame that has been segmented into the target and the second segmentation result of the non-key frame that has been segmented into the target.

[0150] In some embodiments, the results repository is a storage structure used to store the segmentation results of keyframes and non-keyframes that have undergone target segmentation. This storage mechanism facilitates subsequent querying, analysis, and application. The results repository stores not only the first segmentation results of keyframes but also the second segmentation results of non-keyframes. The second segmentation result refers to the background segmentation result obtained by performing target segmentation on non-keyframes. Storing the segmentation results of non-keyframes helps maintain the integrity and continuity of video content analysis, as important information or changes may be detected even in non-keyframes.

[0151] In some embodiments, storing the first segmentation result in the result library can be achieved as follows: obtaining the remaining storage space of the result library; when the remaining storage space is insufficient to store the first segmentation result, compressing or deleting at least one segmentation result in the result library whose storage duration is greater than the duration threshold, to obtain the compressed or deleted segmentation result; updating the result library based on the compressed or deleted segmentation result, and storing the first segmentation result in the updated result library.

[0152] In some embodiments, before storing a new first segmentation result, the remaining storage space in the result repository is checked. This is done by querying the available space in the storage system to ensure there is sufficient space to store the new data. If insufficient remaining storage space is detected to accommodate the new first segmentation result, action is taken to free up space. When the result repository approaches its storage capacity limit, a mechanism is needed to ensure its continued usability. To free up space, segmentation results stored in the result repository for more than a preset threshold are searched. These results may no longer require frequent access or contribute little to the current analysis task. These segmentation results can be compressed or deleted, which can be achieved by reducing image resolution, applying image compression algorithms (such as JPEG, PNG), or reducing data precision. The purpose of compression or deletion is to reduce the data size of the segmentation results, thereby saving storage space. After compression or deletion, the result repository can be updated using the compressed or deleted segmentation results. Once the result repository has sufficient remaining space, the new first segmentation result is stored, ensuring that all important segmentation results are preserved.

[0153] As an example, the following is a concrete example illustrating how to determine the remaining storage space in the results repository and, when space is insufficient, store new first segmentation results by compressing or deleting older ones. A video detection system continuously segments keyframes and stores the results in a results repository. The following scenarios are possible: The remaining storage space in the results repository is checked periodically. For example, suppose the repository has a total capacity of 1TB, and 800GB is currently used, leaving 200GB remaining. A keyframe is segmented, resulting in a first segmentation result of 100GB. The remaining 200GB is insufficient to store a new 100GB segmentation result, so space needs to be freed up. A duration threshold of 30 days is set, meaning only segmentation results stored for more than 30 days are compressed or deleted. The system searches the results repository and finds two segmentation results stored for 60 days and 45 days respectively, exceeding the duration threshold. The system chooses to compress or delete the 60-day-old segmentation result, which originally had a size of 150GB. Using an image compression or deletion algorithm, the system compresses or deletes this segmentation result to 50GB. The original 150GB partition result is replaced with a compressed or deleted 50GB partition result, thus freeing up 100GB of storage space. Now the result library has enough remaining space (200GB - 100GB = 100GB), so the new 100GB first partition result is stored in the result library.

[0154] Thus, by acquiring the remaining storage space in the result repository and compressing or deleting segmentation results that exceed the time threshold when space is insufficient, this method effectively achieves dynamic management of storage resources. The compressed or deleted segmentation results not only free up necessary storage space to accommodate new first segmentation results but also optimize storage efficiency and extend the lifespan of the result repository.

[0155] In step 104, based on the first segmentation result, target segmentation is performed on the non-key frames in the video to obtain the second segmentation result of the non-key frames.

[0156] In some embodiments, the first segmentation result focuses on foreground objects in keyframes, while the second segmentation result for non-keyframes provides information about the video background or other inactive areas. The combination of these two types of information provides more comprehensive information for video content analysis, contributing to a deeper understanding of the video scene. Although non-keyframes may not contain major events, they serve to connect keyframes on the video's timeline. Segmenting non-keyframes maintains the temporal continuity of video content, which is crucial for detecting and analyzing continuous actions or changes in the video. Storing the second segmentation result in a results library, managed together with the first segmentation result, maintains consistency in the stored results. This structured storage method facilitates subsequent data retrieval, analysis, and application. Segmenting non-keyframes based on the first segmentation result allows for the reuse of keyframe processing results to some extent, reducing redundant computation and optimizing resource utilization. Different strategies or parameters can be used for segmenting non-keyframes to accommodate their typically limited amount of critical information. This helps improve overall processing efficiency, especially in scenarios requiring real-time processing.

[0157] In some embodiments, see Figure 7 , Figure 7 This is a flowchart illustrating the video target segmentation method provided in the embodiments of this application. Figure 5 , Figure 3 Step 104 shown can be achieved through Figure 7 Steps 1041 to 1042 shown are implemented.

[0158] In step 1041, the frame features of non-key frames are encoded to obtain the second encoded features.

[0159] In some embodiments, the feature dimension of the second encoded feature is smaller than the feature dimension of the frame feature of the non-key frame.

[0160] In some embodiments, the frame features of non-critical frames are encoded to obtain second encoded features, a process that is of great significance in video processing and encoding. By extracting and encoding the features of non-critical frames, redundancy in video data can be effectively reduced, storage and transmission costs can be lowered, while the main information of the video content can be preserved. Non-critical frames often contain less important information in the video; by encoding the features of these non-critical frames, efficient video compression can be achieved, reducing storage space and bandwidth requirements. The encoded second encoded features retain the core information of the non-critical frames. The encoding process can process the features of non-critical frames to enhance their robustness to noise and changes, ensuring good recognition performance under different conditions.

[0161] In some embodiments, step 1041 above can also be implemented as follows: determine the position of the non-key frame in the frame sorting, and determine the video frame preceding the non-key frame based on the position, wherein the video frame preceding the non-key frame includes the key frame preceding the non-key frame and the non-key frame preceding the non-key frame; based on the first encoding feature of the key frame preceding the non-key frame and the second encoding feature of the non-key frame preceding the non-key frame, encode the frame feature of the non-key frame to obtain the second encoding feature.

[0162] In some embodiments, by determining the position of a non-keyframe in the frame sequence, information about its preceding video frames can be obtained. This positional information is crucial for understanding the temporal relationships and content changes in the video. Determining the preceding video frames for a non-keyframe may include both keyframes and non-keyframes, ensuring that the encoded features of the non-keyframe not only consider the information from the keyframes but also preserve the continuity between non-keyframes. Encoding using the first encoded features of the keyframe preceding the non-keyframe allows the encoded features of the non-keyframe to maintain a certain correlation with the keyframe, helping to retain important information from the keyframe within the non-keyframe, especially in frames near scene changes or events. The second encoded features generated by the encoding method not only reflect the characteristics of the non-keyframe itself but also incorporate information from the preceding keyframe, making these features more representative in video content analysis. By referencing the features of the preceding keyframe, the encoded features of the non-keyframe can better maintain continuity with the overall video content, thus providing a smoother transition when analyzing and understanding the video. Using the encoded features of the keyframe as a reference allows for more accurate extraction and encoding of relevant information in non-keyframes, improving the accuracy of feature encoding. Since non-key frames typically contain less critical information, encoding them using the coding features of preceding key frames can reduce the need for separate processing of non-key frames, thereby optimizing the allocation of computational resources.

[0163] As an example, here's a concrete example illustrating how to encode the frame features of non-keyframes based on a first encoding feature to obtain a second encoding feature: Suppose there's a video detection system that divides video frames into keyframes and non-keyframes and processes these frames using frame sorting. Here are the steps of the process: Video frames are sorted chronologically. In the video stream, frames 1, 10, and 20 are marked as keyframes, and the remaining frames are non-keyframes. Suppose we are interested in frame 15, which is a non-keyframe. We need to determine its position in the frame sorting, i.e., frame 15. Based on the position of frame 15, all video frames preceding it are identified, including both keyframes and non-keyframes. For example, this might include frames 1, 2, 10, 11, 12, 13, and 14. For the frames marked as keyframes in the preceding frames (frames 1 and 10 in this example), their first encoded features are extracted. For the frames marked as non-keyframes in the preceding frames (frames 2 to 9 and 11 to 14 in this example), their second encoded features are extracted. These features are obtained by encoding and dimensionality reduction of the keyframes. The frame features of frame 15 are encoded using the first encoded features of frames 1 and 10, and the second encoded features of frames 2 to 9 and 11 to 14. For example, the feature vector of frame 15 can be adjusted by utilizing statistical information (such as mean and variance) of the features of the preceding keyframes, or a more sophisticated machine learning model can be used to learn how to transform non-keyframe features based on keyframe features. After the encoding process, the second encoded features of frame 15 are obtained. This feature vector has a lower dimension than the original frame feature vector and contains semantic information inherited from the keyframes. For example, frame 15 might contain a scene of a person walking in a static background. Since frames 1 and 10 are likely key moments in the character's appearance, their first encoded features will contain the character's appearance and movement patterns. By encoding frame 15, the second encoded features not only reflect the content of that frame but also incorporate information about the character's continuity within the video.

[0164] Thus, by determining the position of non-key frames in the frame sequence and using this positional information to trace back to previous key and non-key frames, and then encoding the non-key frames based on the first encoding features of the previous key frames to obtain the second encoding features, the process effectively utilizes the temporal and spatial information of video frames. Through the transmission of encoding features from key frames, it achieves effective encoding of non-key frame content. This not only improves encoding accuracy but also optimizes feature representation, making the encoded features of non-key frames more consistent with the dynamic changes in video content. It helps reduce the waste of computational resources, avoids the need for complete encoding of all video frames, and thus improves the overall efficiency and performance of video processing.

[0165] In other embodiments, the above-mentioned encoding of the frame features of the non-key frame based on the first encoding feature to obtain the second encoding feature can be achieved by concatenating the first encoding feature and the frame features of the non-key frame to obtain the initial encoding feature of the non-key frame; and reducing the dimensionality of the initial encoding feature of the non-key frame to obtain the second encoding feature.

[0166] In some embodiments, the first encoded features of the keyframe are concatenated with the frame features of the non-keyframe. The concatenation operation combines elements of the two feature sets into a longer feature vector. The purpose of concatenation is to combine the encoded information of the keyframe and the original frame information of the non-keyframe to more comprehensively represent the content of the non-keyframe. The concatenated feature vector constitutes the initial encoded features of the non-keyframe. These initial encoded features contain information inherited from the keyframe and visual information from the non-keyframe itself, serving as the starting point for subsequent processing. Since the concatenated feature vector may be high-dimensional, contain a large amount of information, and may be redundant, dimensionality reduction is required. The dimensionality-reduced feature vector forms the second encoded features of the non-keyframe. These features are more compact, reducing computational burden while still retaining information crucial for tasks such as object segmentation. The lower dimensionality of the second encoded features helps improve the efficiency and accuracy of subsequent object segmentation.

[0167] In other embodiments, when the number of non-key frames is M, M is an integer greater than 1, and the non-key frames include the j-th non-key frame, the above-mentioned encoding of the frame features of the non-key frames based on the first encoding feature to obtain the second encoding feature can be implemented as follows: traversing j and performing the following processing: when i equals 1, encoding the frame features of the j-th non-key frame based on the first encoding feature corresponding to the 1st key frame to obtain the second encoding feature of the j-th non-key frame; when i is greater than 1, encoding the frame features of the j-th non-key frame based on the first encoding features corresponding to the 1st key frame to the i-th key frame respectively to obtain the second encoding feature of the non-key frame.

[0168] In some embodiments, the total number of non-keyframes is M, where M is an integer greater than 1, indicating that the video contains at least two non-keyframes. The j-th non-keyframe is identified by its index j, which is used to iterate through all non-keyframes from 1 to M during processing. The first encoded features are generated based on the fusion features of the keyframes in the previous steps. These encoded features reflect the important visual and trajectory features of the keyframes. The purpose of encoding non-keyframes with the first encoded features is to propagate the information from the keyframes throughout the video, so as to maintain consistency and accuracy in target segmentation even in areas without keyframes. For each non-keyframe, the following operations are performed: When there is only one keyframe (i=1), the first encoded features of this keyframe are directly used to encode the frame features of the j-th non-keyframe, resulting in the second encoded features of the j-th non-keyframe. When there are multiple keyframes (i>1), all the first encoded features from the 1st keyframe to the 1st keyframe are combined to encode the frame features of the j-th non-keyframe, resulting in the second encoded features of the j-th non-keyframe. The second encoded features can provide consistency in detecting and segmenting targets in non-keyframes, especially in areas where the video content does not change significantly.

[0169] In some embodiments, the above-mentioned encoding of the frame features of the j-th non-key frame based on the first encoding feature corresponding to the first key frame to obtain the second encoding feature of the j-th non-key frame can be implemented as follows: when j equals 1, the frame features of the j-th non-key frame are encoded based on the first encoding feature corresponding to the first key frame to obtain the second encoding feature of the j-th non-key frame; when j is greater than 2, the frame features of the j-th non-key frame are encoded based on the first encoding feature corresponding to the first key frame and the second encoding features corresponding to the first non-key frame to the (j-1)-th non-key frame respectively to obtain the second encoding feature of the j-th non-key frame; when j equals 2, the frame features of the j-th non-key frame are encoded based on the first encoding feature corresponding to the first key frame and the second encoding feature corresponding to the first non-key frame to obtain the second encoding feature of the j-th non-key frame.

[0170] In some embodiments, the generation of the second encoded feature is based on the first encoded feature (the encoded feature obtained from the keyframe) and the frame features of non-keyframes. By encoding the frame features of each non-keyframe, consistent characteristics and segmentation quality are maintained throughout the video. When j equals 1, this is the first non-keyframe, and its second encoded feature is generated directly based on the encoded feature of the first keyframe. This approach ensures that the encoded features of the first non-keyframe are consistent with those of the first keyframe, which is beneficial for conveying information from the keyframe. When j equals 2, this is the second non-keyframe, and its second encoded feature is not only based on the encoded feature of the first keyframe but also incorporates the second encoded feature of the first non-keyframe. This approach utilizes information from the previous non-keyframe to provide more accurate encoding for the current non-keyframe. When j is greater than 2, for subsequent non-keyframes, their second encoded features are generated based on the encoded feature of the first keyframe and the second encoded features of all previous non-keyframes (from the first non-keyframe to the (j-1)th non-keyframe). This recursive encoding method can progressively accumulate and convey information in the video, ensuring that the encoded features of each non-keyframe fully utilize information from previous frames.

[0171] In this way, the core information in keyframes is effectively utilized, and the continuity and accuracy of target segmentation in video analysis are enhanced by recursively incorporating the encoded features of non-keyframes. For the first non-keyframe, the encoded features of the first keyframe are directly used for encoding, ensuring the initial accuracy of segmentation. For subsequent non-keyframes, by combining the encoded features of all previous non-keyframes, not only is the information of the keyframes inherited, but the dynamic changes of the video content are also considered, making the segmentation results more robust. This helps reduce the accumulation of errors, improves the anti-interference ability of video processing algorithms, and thus achieves more stable and reliable target segmentation performance in complex video scenes, thereby improving the overall quality of video analysis and making the segmentation results more in line with the needs of practical applications.

[0172] This approach ensures more comprehensive and accurate frame feature encoding for non-keyframes, as it incorporates key information from keyframes, thus improving the overall performance of video object segmentation. When there is only one keyframe, its encoding features are directly used, maintaining the algorithm's simplicity. However, when there are multiple keyframes, the encoding features of all keyframes are comprehensively considered, better capturing the continuity and dynamic changes of video content, thereby enhancing the robustness of non-keyframe encoding features. This encoding strategy helps improve the efficiency of video analysis because it reduces the need for complex feature extraction and fusion for each non-keyframe individually, making the entire process more efficient and suitable for processing large-scale video data.

[0173] Thus, by encoding non-key frames based on the first encoding feature to obtain the second encoding feature, and by determining the position of the non-key frame in the frame order and referencing information from preceding video frames (including both key and non-key frames), this approach not only effectively utilizes the semantic information of key frames but also maintains the temporal continuity of the video content. This improves the accuracy of non-key frame feature representation, reduces data dimensionality, optimizes the allocation of computational resources, and enhances the model's generalization ability. Furthermore, this position-based encoding strategy facilitates a smoother transition in video analysis and understanding, thereby improving the performance and efficiency of the entire video processing system.

[0174] In step 1042, the second coded features are decoded based on the first segmentation result to obtain the second segmentation result of the non-key frame.

[0175] In some embodiments, the first segmentation result provides target segmentation information in keyframes, while the second coded feature captures the correlation between non-keyframes and keyframes, as well as information about the non-keyframes themselves. Combining these two allows for more accurate target segmentation on non-keyframes. Since the second coded feature is generated based on keyframe information and the positional information of non-keyframes, it provides additional contextual information when segmenting non-keyframes, helping to improve segmentation accuracy, especially in scenes where target objects may also exist in non-keyframes. Segmentation combining the first segmentation result and the second coded feature helps maintain the continuity of video content in non-keyframes, making the overall video analysis results more coherent.

[0176] In some embodiments, step 1042 above can be implemented as follows: extracting features from the first segmentation result to obtain segmentation features of the first segmentation result; fusing the segmentation features and the second coding features to obtain a third coding feature; and decoding the third coding feature to obtain a second segmentation result of a non-keyframe.

[0177] In some embodiments, feature extraction is performed on the first segmentation result to obtain segmentation features. These features typically include boundary information between foreground and background, texture features, color features, etc., which are important foundations for target segmentation. The feature extraction process can further highlight the salient features of the target in keyframes, providing high-quality feature input for subsequent fusion and segmentation. Fusing the segmentation features with the second coded features can complement each other's information. The segmentation features provide specific morphological information of the target in non-keyframes, while the second coded features bring the correlation and contextual information between keyframes and non-keyframes. Fusion helps to build a more comprehensive and accurate feature representation, providing richer information support for the segmentation of non-keyframes. The third coded feature is a combination of the segmentation features and the second coded feature. It not only includes the local features of the target but also integrates the temporal and contextual information of the video, making the third coded feature more comprehensive and applicable to the segmentation of non-keyframes. Decoding the third coded feature is the process of restoring the encoded feature information to the specific segmentation result. This process needs to ensure the accuracy of decoding to maintain the integrity and accuracy of the segmentation features. The decoding process involves operations such as backpropagation, lookup tables, and inverse mapping to ensure that the second segmentation result of non-key frames is as close to the reality as possible. The second segmentation result of non-key frames obtained through the above process has high segmentation quality due to its deep fusion of the first segmentation result and the second encoded features.

[0178] As an example, the following is a concrete example illustrating how a second segmentation result for a non-keyframe can be obtained by performing feature extraction, fusion, and decoding on the first segmentation result: Suppose there is a video detection system that segments frames in a video to identify and detect moving objects. The following are the steps of this process: In the video, frame 10 is a keyframe, which has already been segmented using some algorithm, identifying moving objects (such as a pedestrian) and marking them as foreground. Feature extraction is performed on the segmentation result of frame 10, and the resulting segmentation features may include information such as the pedestrian's outline, texture, and color. These features are used to describe and distinguish foreground objects. Before frame 10, the system may have processed frames 1 and 5, which were also marked as keyframes. Based on the first encoded features of these two frames (such as the pedestrian's position and shape), second encoded features are generated. The segmentation features of frame 10 are fused with the second encoded features of frames 1 and 5 to obtain third encoded features. This involves a machine learning model that learns how to combine features from these different sources to optimize the segmentation result. Decoding the third coded feature may involve inverse mapping or applying a decoder model to convert the coded feature back into a segmentation mask, resulting in a second segmentation result for non-critical frames after frame 10 (e.g., frame 11). In this example, frame 11 is a non-critical frame, and its second segmentation result utilizes the segmentation features of frame 10 and the coded features of frames 1 and 5, thus accurately identifying and segmenting pedestrians in non-critical frames, even if the pedestrian is relatively static or partially occluded. This approach not only accurately segments targets in keyframes but also leverages keyframe information for high-quality segmentation in non-critical frames, improving overall performance and robustness.

[0179] Thus, through the process of feature extraction, fusion, and decoding, the segmentation features extracted from the first segmentation result are combined with the second encoded features to generate a third encoded feature. This not only enriches the feature representation of non-keyframes but also improves the accuracy and robustness of the segmentation result. It effectively integrates keyframe information with the contextual information of non-keyframes, optimizes computational efficiency, and reduces storage requirements. Finally, the high-quality second segmentation result of non-keyframes obtained through decoding provides a deeper and more comprehensive understanding of video content analysis.

[0180] In some embodiments, key frames and non-key frames in the video are ordered. Target segmentation of key frames and non-key frames is performed sequentially according to the frame order. Step 104 can also be implemented as follows: For each non-key frame in the video, the following processing is performed: the position of the non-key frame in the frame order is determined, and video frames preceding the non-key frames are determined based on the position. Video frames preceding the non-key frames include key frames preceding the non-key frames and other non-key frames preceding the non-key frames. Target segmentation is performed on the non-key frames based on the first segmentation result of the key frames preceding the non-key frames and the second segmentation result of the other non-key frames preceding the non-key frames, to obtain the second segmentation result of the non-key frames.

[0181] In some embodiments, the concepts of keyframes and non-keyframes in video processing and analysis are distinguished based on the importance of the frame content. Keyframes typically contain significant changes in the scene, such as scene transitions, the start or end of an action, while non-keyframes contain relatively few changes or are merely part of a continuous action. Frame sorting refers to the chronological arrangement of video frames. When processing video, frames are typically sorted according to their content to distinguish which are keyframes and which are non-keyframes. Target segmentation is performed on keyframes to obtain the first segmentation result. This result is usually considered the baseline because keyframes provide a clear reference for segmentation. The position of the non-keyframe in the frame sequence is determined. Based on the position of the non-keyframe, the frames preceding it are identified, including previous keyframes and non-keyframes. Using the first segmentation result of the previous keyframe and the second segmentation result of the previous non-keyframe as references, target segmentation is performed on the current non-keyframe. Through the above references, the target segmentation result of the non-keyframe, i.e., the second segmentation result, is obtained.

[0182] As an example, the video is segmented into a series of frames, each a point on the timeline. Keyframes might include frames where the person starts running, jumps, and lands. Non-keyframes are frames where there are no significant changes in movement during the running process. Frame 1: The person stands on the starting line; this is a keyframe. Target segmentation is performed on this frame to obtain the person's silhouette and other relevant features; this is the first segmentation result. Frame 5: The person starts running; this is also a keyframe. Target segmentation is performed again to update the first segmentation result. Frame 3: The person is running; this is a non-keyframe. The following are the processing steps for this non-keyframe: Determine the position of frame 3 in the frame order. Find the keyframes and non-keyframes preceding frame 3, such as frames 1 and 2. Use the first segmentation result of frame 1 (the silhouette of the person starting to run) and the second segmentation result of frame 2 (the silhouette of the person running) to assist in target segmentation of frame 3. Based on the above information, frame 3 is segmented to obtain the person's silhouette in this frame; this is the second segmentation result. This process is repeated for each non-keyframe in the video. For example, frames 4, 6, etc., will be segmented based on their position in the frame sorting, referencing the segmentation results of previous key frames and non-key frames.

[0183] In some embodiments, step 104 above can also be implemented as follows: for each of the non-key frames in the video, the following processing is performed: obtain a result library, which stores the first segmentation result of the key frame in the video that has completed the target segmentation, and the second segmentation result of other non-key frames that have completed the target segmentation; based on the first segmentation result of the key frame stored in the result library and the second segmentation result of other non-key frames stored in the result library, target segmentation is performed on the non-key frames to obtain the second segmentation result of the non-key frames.

[0184] In some embodiments, the segmentation results stored in the result library typically contain accurate segmentation information for keyframes. This information can serve as prior knowledge to guide the segmentation of non-keyframes. This approach leverages the temporal continuity of video content, improving the accuracy of non-keyframe segmentation. Since the segmentation results for keyframes are already stored in the result library, these results can be directly used when segmenting non-keyframes, avoiding redundant calculations and thus reducing overall processing complexity and time costs. Based on the segmentation results of keyframes, non-keyframes can be segmented more accurately. For example, if a target in a keyframe is still visible in a non-keyframe, the segmentation information of the keyframe can be used to assist in identifying and segmenting these targets. By referencing information in the result library, the segmentation of non-keyframes can maintain consistency with keyframes, which helps maintain consistency and coherence in target segmentation throughout the video. Utilizing existing segmentation results allows for optimized resource allocation, concentrating more computational resources on keyframe processing while non-keyframes can be processed efficiently with fewer resources. In real-time video analytics scenarios, rapid access to and utilization of information in the result library can significantly improve processing speed and meet real-time requirements. By combining the segmentation results of keyframes, the segmentation model of non-keyframes can learn more scene variations and target behavior patterns, thereby enhancing the model's generalization ability.

[0185] As an example, here's a concrete example illustrating how to read information from a results library to segment non-keyframes and obtain a second segmentation result: Suppose there's a video detection system that segments keyframes in a video and stores the segmentation results in a results library. Here are the steps of the process: In the video, frame 5 is a keyframe. The system has already accurately segmented it using an advanced segmentation algorithm and stored the segmentation results, including segmentation masks for pedestrians and bicycles. Read the segmentation result of frame 5 from the results library to obtain the mask information for pedestrians and bicycles. Next, process frame 7, a non-keyframe. Pedestrians and bicycles in frame 7 may not be as prominent or clear as in the keyframe. Use the segmentation result of frame 5 as a reference to perform target segmentation on frame 7. This might include the following steps: Motion trajectory analysis: Detect pedestrians and bicycles from frame 5 to frame 7 and predict their positions in the non-keyframe. Template matching: Use the segmentation mask of frame 5 as a template to find similar regions in frame 7. Machine Learning Model: A pre-trained model was applied, which learned how to improve the segmentation of non-keyframes based on keyframe segmentation results. This resulted in a second segmentation result for frame 7, which segmented pedestrians and bicycles. This segmentation result was more accurate and reliable than the result obtained using only information from frame 7 itself. The second segmentation result for frame 7 was stored and can be used for subsequent video analysis tasks, such as behavior analysis, traffic statistics, or to improve the segmentation algorithm.

[0186] In this way, by reading the segmentation results stored in the result library and performing target segmentation on non-key frames based on these results, the known information of key frames is effectively utilized to optimize the segmentation quality of non-key frames. This not only reduces the waste of resources caused by repeated calculations, but also improves the accuracy and efficiency of non-key frame segmentation by drawing on verified segmentation patterns, ensuring the coherence and accuracy of video content analysis, and enhancing the overall performance of the system.

[0187] In some embodiments, the first segmentation result of the key frame preceding the non-key frame, and the second segmentation result of other non-key frames preceding the non-key frame are stored in a result library.

[0188] In some embodiments, after performing target segmentation on non-key frames to obtain the second segmentation result of the non-key frames, the following processing can also be performed: based on the storage space of the result library reaching a storage threshold, the second segmentation result of at least one other non-key frame with the highest frame ranking in the result library is replaced with the second segmentation result of the non-key frame; or based on the storage space of the result library reaching a storage threshold, the second segmentation result of at least one other non-key frame with the highest frame ranking in the result library is compressed, and the second segmentation result of the non-key frame is stored in the result library.

[0189] In some embodiments, a storage threshold refers to a pre-defined storage capacity limit in a computer system or storage device. When the amount of data in the storage device reaches or exceeds this limit, specific management strategies are triggered to handle the storage of new data, preventing the storage space from being fully occupied, thereby ensuring normal operation and data security.

[0190] In some embodiments, after performing target segmentation on non-key frames to obtain the second segmentation result of the non-key frames, the processing performed aims to optimize the storage management of the result library, ensuring efficient storage of the latest segmentation results under limited storage space, while preserving valuable historical data as much as possible. When the storage space of the result library reaches the storage threshold, one of the following two strategies can be adopted: 1) Replace the second segmentation result of at least one other non-key frame with the current non-key frame's second segmentation result in the result library. This ensures that the latest segmentation result is always stored in the result library, which is very useful for application scenarios requiring real-time updates. 2) Compress the second segmentation result of at least one other non-key frame with the highest frame ranking in the result library and store the current non-key frame's second segmentation result in the result library. By compressing historical segmentation results, space can be made for new segmentation results without deleting them. This helps to preserve historical data while saving storage space.

[0191] In some embodiments, the choice between replacement and compression methods depends on specific application requirements and storage resource limitations. If real-time performance and the accuracy of the latest data are primary considerations, then a replacement strategy may be more suitable. If historical data needs to be retained for long-term analysis or backtracking, then a compression strategy may be more appropriate.

[0192] As an example, suppose there is video editing software that allows users to perform advanced editing on videos, including segmenting and replacing objects within the video. In this application scenario, after segmenting non-keyframes, the resulting second segmentation results need to be stored in a result library so that users can access and edit these results in the future. When the storage space of the result library reaches a storage threshold, one of two approaches can be taken: 1. Replace the second segmentation result of at least one other non-keyframe in the result library with the second segmentation result of the current non-keyframe. If users are editing an ongoing video project, they may be more concerned with recent edits. In this case, the replacement strategy ensures that the result library always stores the latest segmentation results, allowing users to continue editing and adjusting. 2. Compress the second segmentation result of at least one other non-keyframe in the result library and store the second segmentation result of the current non-keyframe in the result library. If users need to retain all editing history for version control or backtracking, the compression strategy ensures that space is made for new segmentation results without deleting historical data. This helps users restore to a previous editing state when needed while saving storage space. Video editing software can effectively manage segmented results in a database, ensuring users can efficiently access and edit the latest content during the editing process, while selectively retaining historical data for future use. This flexibility allows video editing software to meet the needs of different users, improving editing efficiency and user experience.

[0193] As an example, suppose there's an intelligent video editing system for processing long videos, such as movies or TV series. To improve the efficiency and quality of video editing, we need to segment the scenes in the video to better understand and edit the video content. We create a result library to store the first segmentation results of keyframes that have already undergone target segmentation and the second segmentation results of other non-keyframes that have also undergone target segmentation. These segmentation results help us quickly identify and edit scenes in the video. For each non-keyframe in the video, we perform the following processing: retrieve the first and second segmentation results stored in the result library. Use these segmentation results to perform target segmentation on the current non-keyframe. Store the segmentation results in the result library. If the result library's storage space is full, we need to free up some space to store new segmentation results. We can adopt the following strategies: replace the first segmentation result that has been stored the longest in the result library with the second segmentation result of the current non-keyframe. Or replace the second segmentation result that has been stored the longest in the result library with the second segmentation result of the current non-keyframe. This ensures that the result library always stores the latest segmentation results while freeing up some storage space to store new segmentation results.

[0194] In this way, the storage of target segmentation results can be effectively managed, ensuring that the latest segmentation information is always kept in the result library. This not only improves the efficiency of video processing, enabling the segmentation of non-key frames to be completed quickly based on existing key frame and non-key frame results, but also optimizes the use of storage space by replacing the result with the longest storage time to make room for new segmentation results.

[0195] In some embodiments, after performing target segmentation on non-key frames to obtain the second segmentation result of the non-key frames, the following processing can also be performed: obtaining the remaining storage space of the result library; when the remaining storage space cannot store the second segmentation result, compressing or deleting at least one segmentation result in the result library whose storage duration is greater than the duration threshold, to obtain a compressed or deleted segmentation result; updating the result library based on the compressed or deleted segmentation result, and storing the second segmentation result in the updated result library.

[0196] In some embodiments, before storing the second segmentation result of a non-key frame, the remaining storage space in the result repository is checked. This pre-checking helps to intelligently manage storage resources and ensure the rational use of storage space. When the remaining storage space is insufficient to store new segmentation results, the system frees up space by compressing or deleting segmentation results whose storage duration exceeds a certain threshold. This on-demand compression or deletion strategy improves storage efficiency and ensures that the result repository can continuously store new data. Compressing or deleting old segmentation results not only frees up space but also helps maintain the freshness of the data in the result repository, ensuring that the data stored in the repository is recent and more valuable information. During the process of compressing or deleting and updating the result repository, the system needs to ensure that the integrity and accessibility of the data are not affected. This means that the compression or deletion algorithm should ensure that the data can be restored to its original state after decompression, without affecting subsequent data analysis and processing. By setting a duration threshold, the system can flexibly adjust the compression or deletion strategy according to actual needs and storage space conditions to adapt to different storage requirements and video analysis scenarios. Compressing or deleting segmentation results with longer storage durations helps to reduce data redundancy and avoid storing too much unnecessary data in the result repository, thereby reducing storage costs.

[0197] As an example, suppose there is an online image processing platform that provides image segmentation services to users, enabling them to identify and segment specific objects or regions within images. After processing, the segmentation results of user-uploaded images are stored in a result library. The following is an application of this strategy: A user uploads an image containing multiple objects. The platform uses an image segmentation algorithm to identify and segment the different objects, such as vehicles and pedestrians, and stores the segmentation results in the result library. As the number of users increases and the number of processed images accumulates, the platform needs to continuously monitor the remaining storage space in the result library to prevent it from running out of space. When insufficient storage space is detected to accommodate new segmentation results, the platform initiates a storage space optimization process. This process first identifies segmentation results whose storage time exceeds a certain threshold; these results may be old data that is no longer frequently accessed. The platform compresses or deletes the selected old segmentation results to reduce their storage space usage. The compression or deletion algorithm must ensure data integrity, ensuring that the original segmentation results can be recovered without loss when needed. The compressed or deleted segmentation results replace the original data in the result library, freeing up enough storage space to accommodate new segmentation results. The new segmentation results are stored in the updated results library for users to download or perform further analysis.

[0198] In this way, by obtaining the remaining storage space in the result library and compressing or deleting old segmentation results as needed, not only is sufficient and efficient use of storage space ensured, but also storage costs and redundancy are reduced by compressing or deleting segmentation results with longer storage times. Furthermore, it ensures that new segmentation results can be stored in the library in a timely and accurate manner, thereby improving the efficiency of data processing and retrieval.

[0199] Thus, the process of encoding the frame features of non-key frames based on the first encoding features to obtain the second encoding features, and then using these features for target segmentation, results in the second encoding features having a lower dimensionality compared to the original frame features of non-key frames. This not only reduces computational resource consumption but also improves processing speed, making video analysis more efficient. By inheriting and utilizing information from key frames, the second encoding features can provide a more accurate target representation, thereby achieving more precise target segmentation on non-key frames. Target segmentation combining the first segmentation result and the second encoding features can generate a more reliable second segmentation result, improving the accuracy and robustness of video content parsing.

[0200] In step 105, based on the first segmentation result and the second segmentation result, the target segmentation result of the video is determined, and the target segmentation result is used to separate the object from the video.

[0201] In some embodiments, step 105 above can be implemented as follows: according to the order of key frames and non-key frames in the video, the first segmentation result of the key frames and the second segmentation result of the non-key frames are concatenated to obtain the target segmentation result of the video.

[0202] In some embodiments, target segmentation results refer to the output obtained from the process of separating and identifying specific target objects from video frames using video processing techniques. This process typically involves two stages: a first segmentation result and a second segmentation result. The first segmentation result usually refers to the initial segmentation performed on keyframes, which provides basic information for subsequent processing; the second segmentation result is the target segmentation performed on non-keyframes based on the first segmentation result, utilizing the information from the keyframes and possible temporal relationships to optimize the segmentation. Target segmentation results are used to accurately separate target objects of interest from the video, such as pedestrians and vehicles, for further analysis and processing of these objects.

[0203] In some embodiments, the ordering of keyframes and non-keyframes is crucial for stitching together the target segmentation results in video processing. Keyframes typically refer to frames in the video where the content changes significantly, while non-keyframes are those with relatively stable or minimally changing content. Stitching together according to the order of keyframes and non-keyframes can improve the accuracy and continuity of the target segmentation results. By stitching together the first segmentation result of the keyframes and the second segmentation result of the non-keyframes according to their order in the video, the target segmentation result is obtained. During the stitching process, it is important to maintain the continuity and consistency of the segmentation results to avoid obvious segmentation errors or discontinuities. By stitching together according to the order of keyframes and non-keyframes, the information from both keyframes and non-keyframes can be fully utilized, improving the accuracy and robustness of the target segmentation results. Simultaneously, it can reduce the workload of manual annotation and improve the efficiency of video processing.

[0204] As an example, the video includes keyframe 1, non-keyframe 2, keyframe 3, non-keyframe 4, and non-keyframe 5. According to the order of keyframes and non-keyframes in the video, the first segmentation result of the keyframes and the second segmentation result of the non-keyframes are concatenated to obtain the target segmentation result of the video: {first segmentation result of keyframe 1, second segmentation result of non-keyframe 2, first segmentation result of keyframe 3, second segmentation result of non-keyframe 4, and second segmentation result of non-keyframe 5}.

[0205] Thus, by sorting keyframes and non-keyframes, performing a first segmentation on keyframes and a second segmentation on non-keyframes, and then concatenating the segmentation results, key and auxiliary information in the video can be effectively integrated. This ensures that the main events or objects in the keyframes are accurately and thoroughly segmented, thereby highlighting the core value of the video content; the second segmentation of non-keyframes provides background and contextual information, enhancing the coherence and completeness of the video content.

[0206] Thus, by acquiring the frame features of keyframes in the video to be processed and the trajectory features of the target object in the video, the frame features of the keyframes and the trajectory features are fused to obtain fused features. Based on the fused features, target segmentation is performed on the keyframes to obtain the first segmentation result of the keyframes. Based on the first segmentation result, target segmentation is performed on the non-keyframes in the video to obtain the second segmentation result of the non-keyframes. Based on the first and second segmentation results, the target segmentation result of the video is determined. Since the trajectory features, which represent the trajectory of the target object in the video, reflect the segmentation requirements of the video, while the frame features of the keyframes reflect the spatial information within the keyframes, fusing the frame features and trajectory features of the keyframes allows the obtained fused features to comprehensively reflect both the spatial information and segmentation requirements of the keyframes. By performing target segmentation on the keyframes based on the fused features, the first segmentation result of the keyframes is obtained, thereby effectively improving the segmentation accuracy of the keyframes. Because objects in keyframes and non-keyframes in a video exhibit dynamic trajectory relationships, the non-keyframes in the video are segmented based on the first segmentation result of the keyframes, resulting in a second segmentation result for the non-keyframes. This process fully considers the dynamic trajectory relationships between non-keyframes and keyframes when segmenting non-keyframes, making the second segmentation result for non-keyframes more accurate. Consequently, the target segmentation result of the video determined based on the first and second segmentation results is more accurate, effectively improving the accuracy of target segmentation in the video.

[0207] The following will describe an exemplary application of the embodiments of this application in a real-world video segmentation scenario.

[0208] Video segmentation is the process of segmenting each frame of a video sequence. Since video is a continuous sequence of frames, video segmentation needs to consider not only the image features within a single frame but also the temporal relationships between frames. In video segmentation, each frame can be segmented independently, and then the segmentation results are stitched together to improve the accuracy and coherence of the segmentation. Video segmentation is widely used in video editing, content analysis, detection, and autonomous driving.

[0209] In recent years, the demand for video creation has grown rapidly, and accurate content segmentation capabilities are key to improving creation efficiency. The system in this application can quickly and accurately segment target objects in a video after the user provides natural language instructions, such as marking a running dog in the video, and generate a dynamic mask. This mask can be directly used for video editing, object replacement, background blurring, and other functions, significantly reducing the time cost of manual operation. This technology can be widely integrated into mainstream video editing software to form intelligent video editing tools, suitable for individual creators, short video platform users, and professional film and television post-production workers.

[0210] In modern industrial settings, users can directly specify dynamic targets for materials or equipment using natural language commands. For example, production line operators can input "inspect all unsealed boxes on the packaging line," and the system can accurately identify unsealed target boxes using the technology described in this application, and monitor the dynamic trajectories of these targets in real time, helping workers quickly locate problem areas. Furthermore, in robotic arm operation scenarios, users can input "grab downward-facing parts on the conveyor belt," and the system will segment matching parts through semantic understanding, guiding the robotic arm to complete high-precision grasping tasks in real time. This flexibility based on human commands significantly reduces operational difficulty, improves task efficiency, and enables rapid adaptation to different production needs.

[0211] These two application scenarios demonstrate the powerful capabilities of the embodiments of this application in intelligent content processing and dynamic target analysis. By implementing this technology in specific products, such as intelligent video editors, industrial automation equipment, and intelligent inspection systems, the potential of the embodiments of this application can be maximized, providing new impetus for technological upgrades in related fields.

[0212] In daily life, video has become an important form of recording and expressing information, such as home videos, short videos on social media, and video analysis. With technological advancements, people desire more intelligent video manipulation, such as enabling computers to locate certain objects in a video based on voice commands, like selecting all the children playing on a slide in an amusement park or marking the fastest runner in a video. This demand has driven the development of inference-based video object segmentation technology, which aims to help computers find and segment corresponding target objects in videos using human natural language commands. A major challenge in video processing is the dynamic nature and spatiotemporal consistency of video objects. Unlike targets in static images, objects in videos typically appear as trajectories across multiple frames, accompanied by complex dynamic behaviors and characteristics. This dynamism makes it difficult for computers to accurately capture the complete trajectory of target objects when receiving human commands. Furthermore, objects in videos are closely related in space and time, requiring the generated segmentation results to maintain accuracy not only in a single frame but also throughout the entire video. If these two problems cannot be solved simultaneously, the system's trajectory perception capability will be severely limited, leading to unreliable or inaccurate segmentation results. To address the aforementioned challenges, this application employs a Multimodal Large Language Model (MLLM) to focus on understanding and perceiving trajectories. Specifically, a bidirectional alignment mechanism is introduced, which not only captures the trajectory of targets in videos based on human natural language instructions but also generates related text descriptions by analyzing trajectory information. This bidirectional interaction enables the model to more comprehensively understand the dynamics of trajectories and their underlying meaning. To ensure the accuracy and consistency of segmentation in each frame, a novel module is designed to combine the trajectory information of the target throughout the video with detailed frame-level spatial features, thereby enhancing the model's target understanding capabilities in the spatiotemporal dimensions. This design allows the model to more effectively handle dynamic target segmentation tasks in complex video scenes. Finally, to simplify the segmentation process, a unified end-to-end mask generator is designed. This design processes all video frames through a unified decoder, significantly reducing the complexity of training and deployment compared to traditional methods that require separate optimization of two models, while ensuring the consistency and efficiency of segmentation results. This application's embodiments can bring significant convenience and efficiency improvements in home video editing, short video production, intelligent detection, and autonomous driving. In simple terms, this is a new tool that helps computers understand and interpret video content, making it as simple and intuitive as using a search engine to process dynamic video objects.

[0213] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram illustrating the principle of the video target segmentation method provided in the embodiments of this application. Figure 1This application's embodiments enhance MLLM's ability to understand dynamic trajectory behavior by introducing a bidirectional text-to-trajectory alignment mechanism (Text-to-Trajectory + Trajectory-to-Text). This is achieved by designing a Frame-Content Integration (FCI) module that fuses keyframe spatial information with trajectory information. Figure 8 The fusion shown in the diagram expands the trajectory-level token into a fine-grained token (Frame-level TargetToken) with frame-level spatial information, further enhancing the model's understanding and perception of target trajectories. Simultaneously, a unified target mask generator (Unified Mask Generator) is constructed. Figure 8 The generator shown in the figure realizes the functional unification of the two separation models, thereby enabling the segmentation process to achieve efficient end-to-end processing.

[0214] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram illustrating the principle of the video target segmentation method provided in the embodiments of this application. Figure 2 See Figure 9 (a) The overall technical framework of this application embodiment includes a multimodal large language model (MLLM), a trajectory encoder, and a keyframe spatial information and trajectory information fusion module (Frame Content Integration, FCI). Figure 9 (a) shows the fusion) and a unified target mask generator ( Figure 9 (a) shows the generator, which consists of four main modules. The input is human instructions and a video sequence; the output is the mask of the target corresponding to the human instructions. The segmentation process is as follows: Figure 9 As shown in (a), firstly, a set of keyframes is extracted from the input video using a uniform sampling method, and then input into the MLLM along with natural language instructions to generate a special feature representing the target trajectory information. Figure 9 (a) shows the trajectory. Subsequently, this trajectory-level token is deeply fused with the visual features of all key frames through the FCI module to generate a fine-grained token (frame-level target token) with spatial information for each frame. Finally, a unified mask generator generates target masks (segmentation results) for all frames based on these tokens, including segmentation of key frames based on tokens and segmentation of non-key frames based on memory modules, thus ensuring the spatiotemporal consistency of targets throughout the video.

[0215] In some embodiments, see Figure 10 , Figure 10 This is a schematic diagram illustrating the principle of the bidirectional alignment mechanism provided in this application embodiment, which introduces a text-trajectory bidirectional alignment mechanism. Specifically, the form of the training data is expanded. In addition to the conventional data of text instructions to target trajectories, descriptive training samples that generate corresponding text based on the trajectory are also constructed. For example, if the input is: Please describe trajectory 1 in the video, the output is: Of course, a child wearing a helmet is riding a horse, trajectory (trajectory 1); if the input is: Please describe trajectory 2 in the video, the output is: Of course, a brown horse is being ridden by a girl, trajectory (trajectory 2).

[0216] In some embodiments, at the model level, this application design includes a trajectory encoder (training phase) to output the trajectory features of the target in the video. Specifically, such as... Figure 9 (a) As shown in the left half, the trajectory encoder uses the ground truth coordinates of the target to be segmented in each frame (derived from the sample's ground truth) as input. It extracts features of the target's region from the complete visual features using the ROI-Align method, then linearly projects these features and concatenates them in the temporal dimension to obtain trajectory-level target features. Next, the model inputs these features along with suggestive text ("Can you segment the kitten that is bullying another kitten in this video?") into the MLLM and generates a text description containing the dynamic trajectory ("The kitten that is bullying another kitten."). By supervising this output text to train the model's understanding of dynamic target trajectories, this bidirectional supervision mechanism allows the MLLM to better capture the dynamic behavior and spatiotemporal consistency of trajectories, significantly improving the model's ability to perceive target trajectories.

[0217] In some embodiments, see Figure 9 Keyframe Spatial Information and Trajectory Information Fusion Module (FCI) Figure 9 (a) The fusion shown is a crucial step in connecting trajectory-level information with segmentation details for each frame. The trajectory-level tokens output by MLLM <trj>While containing global semantic information about the target trajectory and relevance to human commands, this module lacks spatial details specific to each video frame. Therefore, it uses an attention mechanism to fuse trajectory-level tokens with visual features, generating fine-grained tokens with spatial information for each keyframe. Specifically, by calculating the similarity between the token and the visual features, the spatial information most relevant to the target is weighted and integrated into the token. This results in a new token (Frame-level target token) that not only retains trajectory-level semantic information but also incorporates precise spatial details from the corresponding frame, significantly improving segmentation accuracy.

[0218] In some embodiments, a unified mask generator is constructed based on the decoder of the open-source model SAM2. Figure 9 (a) The generator in the lower right part is used to achieve unified optimization of keyframe and non-keyframe segmentation, and overcomes the complexity problem caused by multi-model separation optimization in traditional methods. Figure 9 As shown in (b), the designed mask generator comprises two prompt encoders (a key prompt encoder and a non-key prompt encoder) and a mask decoder. The former receives and encodes different prompt information for keyframes and non-keyframes respectively, while the latter predicts the target mask for the current frame based on the encoded prompt information. Specifically: in keyframe segmentation, the prompt information is encoded based on fine-grained tokens generated by the FCI module. The tokens enhanced by the FCI module contain both trajectory-level semantic information and frame-level spatial information of the target, enabling more accurate reconstruction of the target trajectory on the keyframe in both time and space. In non-keyframe segmentation, the prompt information is encoded based on a memory mechanism that records the target information of the segmented frames to provide dynamically consistent prompts.

[0219] In some embodiments, after the prompt encoder outputs the prompt information, a unified mask decoder can ensure that the mask generation process for both keyframes and non-keyframes is performed within an end-to-end framework: such as... Figure 9 As shown in (c), this schematic diagram illustrates how the Mask Decoder processes consecutive video frames (including keyframes and non-keyframes): First, the first frame starts with a keyframe and is segmented using keyframe cues. After obtaining the segmentation result, a dynamic memory bank is initialized based on the target mask output from the first frame and used in the segmentation of subsequent frames. When processing subsequent frames, for non-keyframes, the memory bank provides memory information as cues. After obtaining the current segmentation result, the memory bank is updated based on the target mask of that frame to ensure spatial consistency of the target. Simultaneously, to control computational costs, the memory bank selectively deletes target information from earlier video frames to ensure that the memory information is closely related to the target state near the current frame. For keyframes, the memory bank stores all prediction results because the prediction results of keyframes contain semantic information about the target, allowing for timely correction of the memory bank in case of target errors during segmentation. Based on this unified design, the embodiments of this application not only simplify the training and inference processes but also fully utilize the information interaction between frames, ensuring the spatiotemporal continuity and instruction relevance of the entire video segmentation result.

[0220] Furthermore, to further optimize the performance of the embodiments of this application, the current uniform sampling strategy for keyframes may limit the system's performance in high-dynamic video scenes. Therefore, an adaptive sampling method based on target motion features can be introduced to dynamically adjust the selection of keyframes, thereby better capturing changes in the target trajectory and improving segmentation accuracy. To enhance the memory mechanism for non-keyframe mask generation, a multi-level memory storage architecture can be designed, which retains detailed information of the most recent frame while performing low-dimensional compression storage of information from distant frames, thus maintaining higher segmentation accuracy and spatiotemporal consistency when processing long videos. The combination of these extended schemes will effectively improve the overall performance of the system and provide a new direction for the further development of dynamic video segmentation technology. In summary, the embodiments of this application, through multi-level technological innovation, achieve efficient and accurate segmentation of video target objects, providing a novel solution for the analysis and processing of dynamic scenes.

[0221] In some embodiments, see Figure 11 , Figure 11 This is a schematic diagram comparing the video target segmentation method provided in this application with related technologies. See also: Figure 11 Example 1, as shown, aims to segment out the cat in the video that is unwilling to share its toy with the black (cat / object). The segmentation results of related techniques differ from the expected segmentation results in Example 1, while the segmentation results of this application are the same as the expected segmentation results in Example 1. See also... Figure 11 Example 2, as shown, aims to segment primates from the video. The segmentation results from related techniques differ from the expected results in Example 2, while the segmentation results from this application are the same as the expected results in Example 2. See also... Figure 11 As shown in Example 3, the segmentation expectation is to segment the object moving to the right in the video. The segmentation results of related technologies are different from the expected segmentation results in Example 3, while the segmentation results of this application are the same as the expected segmentation results in Example 3.

[0222] Referring to Table 1 below, which is a performance comparison table of the embodiments of this application and related technologies, the embodiments of this application demonstrate a significant performance improvement in the inference-based video object segmentation task. Related technologies are often limited by insufficient segmentation accuracy and inference ability when processing complex instructions. However, this application effectively fuses trajectory-level object features with visual features through a keyframe spatial information and trajectory information fusion module (FCI), significantly improving the model's segmentation accuracy and spatial detail capture capability. Experimental results on this task dataset show that the overall performance of the model in the embodiments of this application reaches 51.1, and the inference performance reaches 46.1. This not only significantly outperforms similar related technologies of the same scale (7B), such as related technologies 4 and 5, but also surpasses related technologies of larger scale models (such as related technology 7).

[0223] Table 1. Performance Comparison of Embodiments and Related Technologies in this Application

[0224]

[0225] It is understood that in the embodiments of this application, data such as videos to be processed are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0226] The following description continues to illustrate the exemplary structure of the video target segmentation device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the target segmentation device 455 storing the video in the memory 450 may include: an acquisition module 4551, used to acquire frame features of key frames in the video to be processed, and acquire trajectory features of the target object in the video, wherein the trajectory features are used to characterize the trajectory of the target object in the video; a fusion module 4552, used to fuse the frame features of the key frames with the trajectory features to obtain fused features; a segmentation module 4553, used to perform target segmentation on the key frames based on the fused features to obtain a first segmentation result of the key frames; and to perform target segmentation on non-key frames in the video based on the first segmentation result to obtain a second segmentation result of the non-key frames; and a determination module 4554, used to determine the target segmentation result of the video based on the first segmentation result and the second segmentation result, wherein the target segmentation result is used to separate the target object from the video.

[0227] In some embodiments, the acquisition module is further configured to acquire initial description information of the video, the initial description information being used to describe the behavior of the target object; and to obtain trajectory features of the target object based on the video and the initial description information.

[0228] In some embodiments, the acquisition module is further configured to obtain a prompt based on the initial description information, the prompt being used to instruct the trajectory recognition model to output the trajectory features of the target object based on the initial description information; and to identify the trajectory of the target object in the video based on the video and the prompt, thereby obtaining the trajectory features of the target object.

[0229] In some embodiments, the acquisition module is further configured to acquire a first sample prompt for the video sample, the first sample prompt being obtained based on the first sample description information and used to instruct the initial trajectory recognition model to output sample trajectory features of the sample target object based on the first sample description information, the first sample description information being used to describe the behavior of the sample target object in the video sample; based on the video sample and the first sample prompt, the initial trajectory recognition model is used to identify the trajectory of the sample target object in the video sample to obtain the sample trajectory features of the sample target object; based on the sample trajectory features, the initial trajectory recognition model is trained to obtain the trajectory recognition model.

[0230] In some embodiments, the acquisition module is further configured to acquire a second sample prompt from the video sample, the second sample prompt being obtained based on the location information of the target object in the video sample, and used to instruct the initial trajectory recognition model to output the sample behavior features of the target object based on the location information, the location information being used to indicate the position of the target object in multiple video frames in the video sample; based on the video sample and the second sample prompt, the initial trajectory recognition model is used to identify the behavior of the target object in the video sample to obtain the sample behavior features of the target object; based on the sample behavior features, the initial trajectory recognition model is trained to obtain the trajectory recognition model.

[0231] In some embodiments, the segmentation module is further configured to encode the frame features of the key frame to obtain a first encoded feature; and to decode the first encoded feature based on the fusion feature to obtain a first segmentation result of the key frame.

[0232] In some embodiments, the segmentation module is further configured to encode the frame features of the non-key frame to obtain a second encoded feature; and to decode the second encoded feature based on the first segmentation result to obtain a second segmentation result of the non-key frame.

[0233] In some embodiments, the frame features of the key frame include multiple sub-frame features. The fusion module is further configured to determine the similarity between each of the sub-frame features of the key frame and the trajectory features; and to perform weighted fusion of each sub-frame feature and the trajectory features using each similarity as a weight to obtain the fused features corresponding to the key frame.

[0234] In some embodiments, the keyframes and non-keyframes in the video are ordered, and the target segmentation of the keyframes and the target segmentation of the non-keyframes are performed sequentially according to the frame order. The segmentation module is further configured to perform the following processing on each of the non-keyframes in the video: determine the position of the non-keyframe in the frame order, and determine the video frames preceding the non-keyframes based on the position, wherein the video frames preceding the non-keyframes include keyframes preceding the non-keyframes and other non-keyframes preceding the non-keyframes; and perform target segmentation on the non-keyframes based on the first segmentation result of the keyframes preceding the non-keyframes and the second segmentation result of the other non-keyframes preceding the non-keyframes to obtain the second segmentation result of the non-keyframes.

[0235] In some embodiments, the first segmentation result of the key frame preceding the non-key frame and the second segmentation results of other non-key frames preceding the non-key frame are stored in a result library; the segmentation module is further configured to, based on the storage space of the result library reaching a storage threshold, replace the second segmentation result of at least one other non-key frame with the second segmentation result of the non-key frame that is ranked first in the result library; or, based on the storage space of the result library reaching a storage threshold, compress the second segmentation result of at least one other non-key frame that is ranked first in the result library and store the second segmentation result of the non-key frame in the result library.

[0236] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video target segmentation method described in this application.

[0237] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the video target segmentation method provided in this application. For example, ... Figure 3 The video target segmentation method is shown.

[0238] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEP ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of electronic devices including one or any combination of the above-mentioned memories.

[0239] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0240] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0241] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0242] In summary, the embodiments of this application have the following beneficial effects:

[0243] (1) By acquiring the frame features of keyframes in the video to be processed and the trajectory features of the target object in the video, the frame features of the keyframes and the trajectory features are fused to obtain fused features. Based on the fused features, target segmentation is performed on the keyframes to obtain the first segmentation result of the keyframes. Based on the first segmentation result, target segmentation is performed on the non-keyframes in the video to obtain the second segmentation result of the non-keyframes. Based on the first and second segmentation results, the target segmentation result of the video is determined. Thus, since the trajectory features represent the trajectory of the target object in the video, which reflects the segmentation requirements of the video, while the frame features of the keyframes reflect the spatial information in the keyframes, by fusing the frame features of the keyframes and the trajectory features, the obtained fused features can comprehensively reflect the spatial information and segmentation requirements of the keyframes. By performing target segmentation on the keyframes based on the fused features, the first segmentation result of the keyframes is obtained, thereby effectively improving the segmentation accuracy of the keyframes. Because objects in keyframes and non-keyframes in a video exhibit dynamic trajectory relationships, the non-keyframes in the video are segmented based on the first segmentation result of the keyframes, resulting in a second segmentation result for the non-keyframes. This process fully considers the dynamic trajectory relationships between non-keyframes and keyframes when segmenting non-keyframes, making the second segmentation result for non-keyframes more accurate. Consequently, the target segmentation result of the video determined based on the first and second segmentation results is more accurate, effectively improving the accuracy of target segmentation in the video.

[0244] (2) The system can accurately pinpoint important frames in a video, i.e., keyframes, which is crucial for video content understanding, retrieval, and efficient storage. Identifying an object as a keyframe when its parameter value exceeds a set threshold ensures that the extracted information represents the most significant or important parts of the video content, thereby improving the targeting and efficiency of video analysis. Feature extraction from keyframes yields more specific and useful video feature information, providing effective data support for subsequent video processing tasks such as object detection, behavior recognition, and content classification. Feature extraction not only improves the accuracy of recognition and classification but also reduces the computational load of subsequent processing. Identifying frames that do not meet the parameter threshold as non-keyframes helps process redundant information and optimize resource utilization. This differentiated processing strategy allows the system to focus on key information while ignoring frames with less impact on the overall analysis, thus improving processing speed and system efficiency while ensuring analysis quality.

[0245] (3) By combining the initial descriptive information with the video content, trajectory features are predicted, which significantly improves the richness and accuracy of the descriptive information. When describing the behavior of an object, introducing detailed information about the movement trajectory not only makes the description more comprehensive, but also helps users understand the context in which the behavior occurs, such as the speed, direction and path of the movement, thus providing deeper insights.

[0246] (4) By acquiring the first sample prompt from the video sample and using the initial trajectory recognition model to identify the trajectory features of the target object, the accuracy of trajectory recognition can be improved. At the same time, training the initial trajectory recognition model based on the sample trajectory features can further optimize the model performance and improve its ability to recognize the trajectory of the target object.

[0247] (5) By acquiring second sample prompts based on location information, the initial trajectory recognition model can be more effectively guided to learn the behavioral and trajectory features of the target object. This not only improves the model's accuracy in recognizing the trajectory of the target object, but also enhances the model's adaptability to complex scenarios.

[0248] (6) By generating prompts based on initial descriptive information and using a trajectory recognition model to identify the trajectory features of the target object based on these prompts and video data, the accuracy and efficiency of trajectory recognition can be significantly improved. This technology not only enhances the model's ability to understand the behavior of the target object, but also improves the ability to predict the movement trajectory of the target object in complex scenes.

[0249] (7) The similarity between the features of each subframe of the keyframe and the trajectory features is determined, and weighted fusion is performed using these similarities as weights, which significantly improves the accuracy and representativeness of the keyframe feature representation. By calculating the similarity, the subframe features most relevant to the trajectory features can be identified, thereby highlighting these important information in the keyframe and filtering out noise or irrelevant information. Weighted fusion ensures that the feature vector of the keyframe not only contains the core information of the video content, but also that the contribution of each subframe feature is proportional to its correlation with the trajectory features. This makes the final fused features more accurate and robust input when used for tasks such as video recognition, classification, or retrieval, thereby improving the overall performance of the model and the reliability of predictions.

[0250] (8) By mapping the fused features to keyframes one-to-one and encoding each keyframe based on its fused features, the resulting first encoded features not only accurately capture the core information of each keyframe, but also significantly improve the overall efficiency of video analysis due to the optimization of frame sorting and the number of keyframes. The encoded feature vectors are not only lower in dimensionality, making them easier to store and process quickly, but also improve the performance and generalization ability of target segmentation while ensuring accuracy because they filter and retain key information.

[0251] (9) By obtaining the remaining storage space of the result library and compressing or deleting the segmentation results that exceed the time threshold when space is insufficient, this method effectively realizes dynamic management of storage resources. The compressed or deleted segmentation results not only release the necessary storage space to accommodate the new first segmentation results, but also optimize storage efficiency and extend the lifespan of the result library.

[0252] (10) By encoding non-key frames based on the first encoding feature to obtain the second encoding feature, and by determining the position of the non-key frame in the frame order and referring to the information of the preceding video frames (including key frames and non-key frames), not only is the semantic information of the key frames effectively utilized, but the temporal continuity of the video content is also maintained. This improves the accuracy of non-key frame feature representation, reduces data dimensionality, optimizes the allocation of computing resources, and enhances the generalization ability of the model. At the same time, this position-based encoding strategy also helps to achieve a smoother transition in video analysis and understanding, thereby improving the performance and efficiency of the entire video processing system.

[0253] (11) Through the process of feature extraction, fusion, and decoding, the segmentation features extracted from the first segmentation result are combined with the second encoded features to generate a third encoded feature that not only enriches the feature representation of non-key frames but also improves the accuracy and robustness of the segmentation result. It effectively integrates key frame information with the contextual information of non-key frames, optimizes computational efficiency, and reduces storage requirements. Finally, the high-quality non-key frame second segmentation result obtained through decoding provides a deeper and more comprehensive understanding of video content analysis.

[0254] (12) Read the segmentation results stored in the result library and perform target segmentation on non-key frames based on these results. This effectively utilizes the known information of key frames to optimize the segmentation quality of non-key frames. This not only reduces the waste of resources caused by repeated calculations, but also improves the accuracy and efficiency of non-key frame segmentation by referencing verified segmentation patterns. This ensures the coherence and accuracy of video content analysis and enhances the overall performance of the system.

[0255] (13) By adopting the method of obtaining the remaining storage space of the result library and compressing or deleting old segmentation results as needed, not only is the sufficiency and efficient use of storage space ensured, but also the storage cost and redundancy are reduced by compressing or deleting segmentation results with a long storage time. Furthermore, it ensures that new segmentation results can be stored in the library in a timely and accurate manner, thereby improving the efficiency of data processing and retrieval.

[0256] (14) By sorting keyframes and non-keyframes, performing a first segmentation on keyframes and a second segmentation on non-keyframes respectively, and then splicing the segmentation results together, key and auxiliary information in the video can be effectively integrated. This ensures that the main events or objects in the keyframes are accurately and in detail segmented, thereby highlighting the core value of the video content; the second segmentation of non-keyframes provides background and contextual information, enhancing the coherence and integrity of the video content.

[0257] (15) It can ensure that the frame feature encoding of non-key frames is more comprehensive and accurate because it integrates the key information in the key frames, which helps to improve the overall performance of video target segmentation. When there is only one key frame, the encoding feature of the key frame is used directly for encoding, which maintains the simplicity of the algorithm. When there are multiple key frames, the encoding features of all key frames are considered in combination, which can better capture the continuity and dynamic changes of video content, thereby enhancing the robustness of non-key frame encoding features. This encoding strategy helps to improve the efficiency of video analysis because it reduces the need for complex feature extraction and fusion for each non-key frame individually, making the whole process more efficient and suitable for processing large-scale video data.

[0258] (16) It effectively utilizes the core information in keyframes and enhances the continuity and accuracy of target segmentation in video analysis by recursively incorporating the coding features of non-keyframes. For the first non-keyframe, the coding features of the first keyframe are directly used for encoding, ensuring the initial accuracy of segmentation; while for subsequent non-keyframes, by combining the coding features of all previous non-keyframes, not only is the information of the keyframes inherited, but the dynamic changes of the video content are also considered, making the segmentation results more robust. This helps reduce the accumulation of errors, improves the anti-interference ability of video processing algorithms, and thus achieves more stable and reliable target segmentation performance in complex video scenes, thereby improving the overall quality of video analysis and making the segmentation results more in line with the needs of practical applications.

[0259] (17) It ensures the maximum space utilization efficiency of the feature repository. By monitoring and adjusting the storage space in real time, the algorithm can intelligently handle situations where space is insufficient, avoiding feature data loss or processing interruption due to space limitations. When storage space is sufficient, the first encoded features of keyframes can be stored directly, maintaining the continuity of processing; while when space is insufficient, by compressing or deleting encoded features that have not been used for a long time, not only is storage space freed up, but the storage structure is also optimized, enabling the feature repository to operate continuously and efficiently. It improves the robustness and flexibility of the video processing system, providing strong support for real-time analysis and large-scale data processing, while ensuring the efficient and accurate execution of video analysis tasks.

[0260] (18) To further optimize the performance of the embodiments of this application, the current uniform sampling strategy for keyframes may limit the system's performance in high-dynamic video scenes. Therefore, an adaptive sampling method based on target motion features can be introduced to dynamically adjust the selection of keyframes, thereby better capturing changes in target trajectories and improving segmentation accuracy. To enhance the memory mechanism for non-keyframe mask generation, a multi-level memory storage architecture can be designed, which retains detailed information of the most recent frames while performing low-dimensional compression or deletion storage of information from distant frames, thereby maintaining higher segmentation accuracy and spatiotemporal consistency when processing long videos. The combination of these extended schemes will effectively improve the overall performance of the system and provide a new direction for the further development of dynamic video segmentation technology. In summary, the embodiments of this application, through multi-level technological innovation, achieve efficient and accurate segmentation of video target objects, providing a brand-new solution for the analysis and processing of dynamic scenes.

[0261] (19) The first encoded features of the keyframe are concatenated with the frame features of the non-keyframe. The concatenation operation combines the elements of the two feature sets into a longer feature vector. The purpose of concatenation is to combine the encoded information of the keyframe and the original frame information of the non-keyframe to more comprehensively represent the content of the non-keyframe. The concatenated feature vector constitutes the initial encoded features of the non-keyframe. These initial encoded features contain information inherited from the keyframe and visual information of the non-keyframe itself, and are the starting point for subsequent processing. Since the concatenated feature vector may have a high dimension and contain a lot of information, it may be redundant, so dimensionality reduction is required. The dimensionality-reduced feature vector forms the second encoded features of the non-keyframe. These features are more compact, reducing the computational burden, while still retaining information that is crucial for tasks such as target segmentation. The lower dimension of the second encoded features helps to improve the efficiency and accuracy of subsequent target segmentation.

[0262] (20) The process of concatenating the fused features and the frame features of the keyframes, and further obtaining the first encoded features through dimensionality reduction, not only integrates the visual and non-visual information of the keyframes to form a more comprehensive initial encoded feature, but also effectively simplifies the feature dimensions through dimensionality reduction, thereby reducing computational complexity and storage requirements while maintaining key information. This processing not only improves the running efficiency of video analysis algorithms, but also enhances the generalization ability and robustness of the model, enabling tasks such as video target segmentation and recognition to be performed more accurately and efficiently, and overall improving the performance of video content understanding and analysis.

[0263] (21) By acquiring behavioral and trajectory description information from video samples and processing it using an initial extended trajectory recognition model, this approach effectively integrates multi-dimensional information from video content, enhancing the model's understanding of object behavior and its ability to predict trajectories. This not only improves the accuracy of video analysis but also enriches the training data through the generation of extended information, making model training more efficient. Ultimately, a more accurate and practical extended trajectory recognition model is obtained, bringing significant performance improvements and broader application potential to the field of video understanding.

[0264] (22) Combining the object's trajectory information with the initial description forms a more comprehensive and detailed descriptive information. This process not only enriches the connotation of the descriptive information but also improves the information's representational ability through expansion and feature extraction, thereby providing more accurate and in-depth insights for video analysis and understanding. This enables subsequent tasks such as automatic recognition, behavior prediction, and decision support to be based on richer trajectory features, significantly improving the performance and practicality of the video processing system.

[0265] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / trj>

Claims

1. A method for target segmentation in video, characterized in that, The method includes: Obtain the frame features of keyframes in the video to be processed; The trajectory features of the target object in the video are obtained, and the trajectory features are used to characterize the trajectory of the target object in the video. The frame features of the keyframes are fused with the trajectory features to obtain fused features; Based on the fusion features, target segmentation is performed on the key frame to obtain the first segmentation result of the key frame; Based on the first segmentation result, target segmentation is performed on the non-key frames in the video to obtain the second segmentation result of the non-key frames; Based on the first segmentation result and the second segmentation result, a target segmentation result is determined for the video, and the target segmentation result is used to separate the target object from the video.

2. The method according to claim 1, characterized in that, The step of obtaining the trajectory features of the target object in the video includes: Obtain initial description information of the video, the initial description information being used to describe the behavior of the target object; Based on the video and the initial description information, the trajectory features of the target object are obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the trajectory features of the target object based on the video and the initial description information includes: Based on the initial description information, a prompt is obtained, which is used to instruct the trajectory recognition model to output the trajectory features of the target object based on the initial description information; Based on the video and the prompt, the trajectory of the target object in the video is identified using a trajectory recognition model to obtain the trajectory features of the target object.

4. The method according to claim 3, characterized in that, The method further includes: The first sample prompt is obtained from the video sample. The first sample prompt is obtained based on the first sample description information and is used to instruct the initial trajectory recognition model to output the sample trajectory features of the sample target object based on the first sample description information. The first sample description information is used to describe the behavior of the sample target object in the video sample. Based on the video sample and the first sample prompt, the trajectory of the target object in the video sample is identified using an initial trajectory recognition model to obtain the sample trajectory features of the target object; Based on the sample trajectory features, the initial trajectory recognition model is trained to obtain the trajectory recognition model.

5. The method according to claim 4, characterized in that, The method further includes: The second sample prompt is obtained from the video sample. The second sample prompt is obtained based on the position information of the sample target object in the video sample and is used to instruct the initial trajectory recognition model to output the sample behavior features of the sample target object based on the position information. The position information is used to indicate the position of the sample target object in multiple video frames in the video sample. Based on the video sample and the second sample prompt, the behavior of the target object in the video sample is identified through an initial trajectory recognition model to obtain the sample behavior features of the target object; Based on the sample behavior features, the initial trajectory recognition model is trained to obtain the trajectory recognition model.

6. The method according to claim 1, characterized in that, The step of segmenting the keyframe based on the fused features to obtain a first segmentation result of the keyframe includes: The frame features of the keyframe are encoded to obtain the first encoded features; The first encoding feature is decoded based on the fused feature to obtain the first segmentation result of the keyframe.

7. The method according to claim 1, characterized in that, The step of performing target segmentation on non-key frames in the video based on the first segmentation result to obtain a second segmentation result for the non-key frames includes: The frame features of the non-key frames are encoded to obtain the second encoded features; Based on the first segmentation result, the second encoded feature is decoded to obtain the second segmentation result of the non-key frame.

8. The method according to claim 1, characterized in that, The frame features of the keyframe include multiple subframe features. The process of fusing the frame features of the keyframe with the trajectory features to obtain fused features includes: Determine the similarity between the sub-frame features of the keyframe and the trajectory features; Using the similarity as the weight, the features of each subframe and the trajectory features are weighted and fused to obtain the fused features corresponding to the keyframe.

9. The method according to claim 1, characterized in that, The key frames and non-key frames in the video are ordered. The step of performing target segmentation on non-key frames in the video based on the first segmentation result to obtain a second segmentation result for the non-key frames includes: For each of the non-keyframes in the video, the following processing is performed: The position of the non-key frame in the frame sorting is determined, and the video frames preceding the non-key frame are determined based on the position. The video frames preceding the non-key frame include key frames preceding the non-key frame and other non-key frames preceding the non-key frame. Based on the first segmentation result of the key frame preceding the non-key frame, and the second segmentation result of other non-key frames preceding the non-key frame, target segmentation is performed on the non-key frame to obtain the second segmentation result of the non-key frame.

10. The method according to claim 9, characterized in that, The first segmentation result of the key frame preceding the non-key frame, and the second segmentation result of other non-key frames preceding the non-key frame are stored in the result library; After performing target segmentation on non-key frames in the video based on the first segmentation result to obtain a second segmentation result for the non-key frames, the method further includes: When the storage space of the result library reaches the storage threshold, the second segmentation result of at least one other non-key frame with the highest frame ranking in the result library is replaced with the second segmentation result of the non-key frame. Alternatively, if the storage space of the result library reaches a storage threshold, the second segmentation result of at least one other non-key frame with the highest frame ranking in the result library is compressed, and the second segmentation result of the non-key frame is stored in the result library.

11. A target segmentation device for video, characterized in that, The device includes: The acquisition module is used to acquire the frame features of key frames in the video to be processed, and to acquire the trajectory features of the target object in the video, wherein the trajectory features are used to characterize the trajectory of the target object in the video; The fusion module is used to fuse the frame features of the key frame with the trajectory features to obtain fused features; The segmentation module is used to perform target segmentation on the keyframe based on the fusion features to obtain a first segmentation result of the keyframe; and to perform target segmentation on non-keyframes in the video based on the first segmentation result to obtain a second segmentation result of the non-keyframes. A determining module is configured to determine a target segmentation result of the video based on the first segmentation result and the second segmentation result, wherein the target segmentation result is used to separate the target object from the video.

12. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video target segmentation method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the video target segmentation method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the video target segmentation method according to any one of claims 1 to 10 is implemented.

Images