Video processing model training method and video processing method

Abstract

The embodiment of the invention provides a video processing model training method and a video processing method.The video processing model training method comprises the steps that target sample data is obtained, and the target sample data comprises a sample video, sample entity identification information, a sample entity description text and a sample video processing result; according to the sample video, the sample entity identification information and the sample entity description text, an initial video processing model is obtained, the initial video processing model is used for generating an entity description text corresponding to the video, and the entity description text represents features of an entity to be recognized in the video; acquiring a video identification layer, and generating a reference video processing model based on the initial video processing model and the video identification layer; and training a reference video processing model according to the sample entity identification information, the sample video and the sample video processing result to obtain a target video processing model. The video processing accuracy is improved when the target video processing model is used for processing the corresponding video subsequently.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular to a video processing model training method and a video processing method. Background Art

[0002] With the development of video recognition technology, video recognition in complex scenes has made significant progress and can better meet the semantic analysis needs of the overall scene. However, the accuracy of video recognition is crucial for many real-world applications.

[0003] Currently, when using a multimodal large language model to recognize a video, it is possible to identify and locate a specific object or area through visual markers or textual prompts. However, the accuracy of using a multimodal large language model to recognize a video in combination with the meaning of the object itself and the corresponding object area in the video is still low. Therefore, in order to solve the above problem, a video processing model training method is urgently needed so that the model obtained by the above training can have a higher accuracy when performing video recognition. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a video processing model training method, a video processing method, a video processing method applied to a terminal device, and a video processing method applied to a cloud device. One or more embodiments of this specification also involve a video processing model training device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a video processing model training method is provided, including:

[0006] Acquire target sample data, wherein the target sample data includes a sample video, sample entity identification information, a sample entity description text, and a sample video processing result;

[0007] Obtaining an initial video processing model according to the sample video, the sample entity identification information and the sample entity description text, wherein the initial video processing model is used to generate an entity description text corresponding to the video, and the entity description text represents the characteristics of the entity to be identified in the video;

[0008] Acquire a video recognition layer, and generate a reference video processing model based on the initial video processing model and the video recognition layer;

[0009] The reference video processing model is trained according to the sample entity identification information, the sample video and the sample video processing result to obtain a target video processing model.

[0010] According to a second aspect of the embodiments of this specification, a video processing method is provided, including:

[0011] Obtain the video to be processed and the identification information of the entity to be identified;

[0012] The video to be processed and the identification information of the entity to be recognized are input into the target video processing model to obtain the description text of the entity to be recognized and the video processing result, wherein the target video processing model is trained according to the above-mentioned video processing model training method, and the video processing result is generated according to the description text of the entity to be recognized, and the description text of the entity to be recognized represents the characteristics of the entity corresponding to the identification information of the entity to be processed in the video to be processed.

[0013] According to a third aspect of an embodiment of this specification, a video processing method is provided, which is applied to a terminal device, including:

[0014] Obtaining a video to be processed and annotation information of entities to be identified drawn by a user for the video to be processed;

[0015] Determining identification information of an entity to be identified based on the annotation information of the entity to be identified, wherein the identification information of the entity to be identified represents a description of the entity to be selected by the user in the video to be processed;

[0016] Generate a video processing instruction according to the video to be processed and the identification information of the entity to be identified, and send the video processing instruction to a cloud-side device, so that the cloud-side device generates a description text of the entity to be identified and a video processing result corresponding to the video processing instruction according to a target video processing model, wherein the target video processing model is obtained by training according to the above-mentioned video processing model training method;

[0017] Receive the entity description text and video processing results to be identified sent by the cloud-side device.

[0018] According to a fourth aspect of the embodiments of this specification, a video processing method is provided, which is applied to a cloud-side device, including:

[0019] A video processing instruction sent by a receiving terminal device, wherein the video processing instruction includes the video to be processed and the identification information of the entity to be identified;

[0020] Input the video to be processed and the identification information of the entity to be identified into a target video processing model to obtain a description text of the entity to be identified and a video processing result, wherein the target video processing model is obtained by training according to the above-mentioned video processing model training method, and the video processing result is generated according to the description text of the entity to be identified, and the description text of the entity to be identified represents the characteristics of the entity corresponding to the identification information of the entity to be processed in the video to be processed;

[0021] The description text of the entity to be identified and the video processing result are sent to the terminal side device.

[0022] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0023] Memory and processor;

[0024] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned video processing model training method, a video processing method, a video processing method applied to an end-side device, and a video processing method applied to a cloud-side device are implemented.

[0025] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of a video processing model training method, a video processing method, a video processing method applied to an end-side device, and a video processing method applied to a cloud-side device are implemented.

[0026] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned video processing model training method, a video processing method, a video processing method applied to an end-side device, and a video processing method applied to a cloud-side device.

[0027] An embodiment of the present specification implements the steps of obtaining target sample data, wherein the target sample data includes a sample video, sample entity identification information, a sample entity description text, and a sample video processing result; obtaining an initial video processing model based on the sample video, the sample entity identification information, and the sample entity description text, wherein the initial video processing model is used to generate an entity description text corresponding to the video, and the entity description text represents features of an entity to be identified in the video; obtaining a video recognition layer, and generating a reference video processing model based on the initial video processing model and the video recognition layer; and training the reference video processing model based on the sample entity identification information, the sample video, and the sample video processing result to obtain a target video processing model.

[0028] By applying the solution of the embodiments of this specification, it is possible to accurately identify and locate the object area in the video when the corresponding video is subsequently processed using the target video processing model, thereby improving the recognition ability combined with the meaning of the object, thereby improving the accuracy of video processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of a video processing model training method provided by an embodiment of this specification;

[0030] Figure 2 is a structural schematic diagram of a target video processing model provided by an embodiment of this specification;

[0031] Figure 3 is a flow chart of a video processing method provided by an embodiment of this specification;

[0032] Figure 4 is a flowchart of a video processing method applied to a terminal device provided by an embodiment of this specification;

[0033] Figure 5 is a flow chart of a video processing method applied to a cloud-side device provided by an embodiment of this specification;

[0034] Figure 6 is an architecture diagram of a video processing system provided by an embodiment of this specification;

[0035] Figure 7 is a processing flow chart of a video segmentation model training method provided by an embodiment of this specification;

[0036] Figure 8 It is a structural schematic diagram of a video processing model training device provided by an embodiment of this specification;

[0037] Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0038] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0039] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0041] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0042] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model (Foundation Model 1). The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (Large Language Model 1, LLM), a multi-modal pre-training model (multi-modal pre-training model 1), etc.

[0043] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0044] First, the terms involved in one or more embodiments of this specification are explained.

[0045] The Multimodal Large Language Model (MLLM) is a language model that can process text, images, and other types of data simultaneously. It is trained by fusing data from multiple modalities, thus having the ability to understand and generate different types of content. With the combination of multimodal data, the model can achieve comprehensive analysis of complex scenes and is widely used in natural language processing, computer vision, and other fields. It shows strong adaptability and flexibility in tasks that require collaborative understanding of text and vision.

[0046] In this specification, a video processing model training method, a video processing method, a video processing method applied to an end-side device, and a video processing method applied to a cloud-side device are provided. This specification also involves a video processing model training device, a computing device, and a computer-readable storage medium, a computer program product, which are described in detail one by one in the following embodiments.

[0047] Most current video understanding methods focus on scene-level video understanding, while ignoring fine-grained spatial object or region-level understanding, which is critical for real-world applications. In multimodal large language models, how to achieve spatial understanding of the model is a fundamental research problem. This problem derives two important capabilities, reference and localization.

[0048] Reference requires the model to accurately understand the semantics of a specific object / region, for example, by giving some visual markers such as dots, boxes, graffiti lines, etc. in a certain frame of a video to identify the meaning of a specific object or scene. On the other hand, localization requires the model to locate and segment related areas based on given conditions (such as text prompts or visual clues).

[0049] Essentially, both reference and localization rely on the same type of knowledge: spatial information, temporal information, and alignment between semantics. However, the current multimodal large language model has weak reference and localization capabilities, so the accuracy of using multimodal large language models to identify videos based on the meaning of the object itself and the corresponding object area in the video is still low.

[0050] See also Figure 1 , Figure 1 A flowchart of a video processing model training method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0051] Step 102: Acquire target sample data, wherein the target sample data includes sample video, sample entity identification information, sample entity description text, and sample video processing results.

[0052] In practical applications, the target sample data is any one of the sample data sets, the sample data set is a data set used to obtain the target video processing model, the sample video is a video used to obtain the target video processing model, the sample entity identification information is the identification information corresponding to a specific entity in the sample video, the sample entity description text is the text describing the specific entity in the sample video, and the sample video processing result is a video in which a specific entity in the sample video is marked or a specific entity is cut out.

[0053] Exemplarily, target sample data can be understood as a set of data instances in a data set used to analyze, test or train a specific model. For example, in the training of a video processing model, a target sample data may be a specific video clip used to train the data processing model to obtain a target data processing model with higher data processing accuracy. Sample videos can be understood as videos with a prominent subject in order to achieve accurate recognition and positioning of object areas in the video using the target video processing model. For example, a video of a man rowing a boat or a video of a cat running on the grass.

[0054] The sample entity identification information can be understood as a unique identifier used to distinguish a specific entity in the video. This identification information can be text or a label, or it can be a line segment, a point or a bounding box that covers a specific entity in the starting frame of the video, etc., indicating a specific object in the video, which is used to identify the entity during model training or testing. The sample entity description text can be understood as text information that provides a detailed explanation of a specific entity in the sample video. It usually contains a description of the entity's features, behaviors, or locations to help the model understand the characteristics of the entity. For example, a man rowing on the sea. The sample video processing result can be understood as a sample video processed by the model, containing the annotation or cutting results of a specific entity.

[0055] By obtaining target sample data including sample videos, sample entity identification information, sample entity description text and sample video processing results, the sample entity identification information and sample entity description text can be effectively used to enable the model to have the ability to focus on specific entities in the video during the subsequent model training process, thereby making the target video processing model obtained through training have higher accuracy in the subsequent video processing process.

[0056] Furthermore, target sample data is obtained, including:

[0057] Acquire an initial data set, wherein the initial data set includes at least one initial video and video description text corresponding to each initial video;

[0058] Determining at least one sample video in each initial video based on the description text of each initial video;

[0059] Target sample data is obtained according to each sample video and the sample video description text corresponding to each sample video.

[0060] In practical applications, the initial data set is a data set containing original videos and related descriptions used for model training. The initial video is the original video material that has not been processed in the data set, and the video description text is a detailed text description of the initial video content, which is used to help the model understand the content in the video.

[0061] Exemplarily, the initial data set can be understood as containing a variety of data sources for training models, including initial videos and video description texts. It plays an important data support role in the training of video processing models. It consists of a large amount of video data and is accompanied by text descriptions of these videos. The initial video can be understood as the video material in the initial data set. The initial video usually retains the complete scene and action without any cropping, segmentation or other data processing. For example, the initial video can be the complete process of a person fishing on a boat, including every action and environmental change of his fishing, or it can be a picture of multiple animals running in the grass.

[0062] Video description text can be understood as a detailed text description of the initial video content, which is used to help the model better understand the specific content in the video. These descriptions are usually written based on information such as the scene, character movements, and object appearance in the video. For example, in a video of a person fishing, the video description text may include: "A man in a camouflage shirt stands by the boat fishing. He concentrates on watching the water and quickly catches the fish when it jumps out of the water." By providing this detailed text description, the model is able to combine the image information in the video with the language information, thereby achieving more accurate video understanding and analysis.

[0063] It should be noted that determining at least one sample video in each initial video based on the description text of each initial video can be understood as screening out sample videos for training the target video processing model through the description text of the initial video so that the target video processing model has the ability to focus on specific entities in the video. The specific method can be to obtain the number of subjects in the initial video description text corresponding to each initial video, and determine at least one sample video based on the number of subjects; it can also be determined whether the video description text contains a series of actions or only describes one frame, and the initial video with the description text describing one frame is eliminated to determine the sample video; it can also be based on the scene label in the initial video description text, and screen sample videos with the same or similar scenes. This specification does not impose any restrictions on this.

[0064] It should be noted that obtaining target sample data based on each sample video and the sample video description text corresponding to each sample video can be understood as extracting data suitable for training the target data processing model by analyzing and processing the sample video and its description text. The specific method can be to obtain the bounding box of each entity in the video, and convert the bounding box into a mask to finally obtain the sample entity annotation information and sample entity description data corresponding to the subject; it can also generate a detailed description of the video through the video description information, and highlight the area of ​​the object corresponding to the subject, obtain a detailed description of the object, and then generate corresponding question and answer pairs including basic questions and answers and reasoning questions and answers; it can also first extract the subject in the video, and then describe the action and appearance in detail, confirm and screen through the corresponding area, and finally generate sample entity description information, etc. This manual does not impose any restrictions on this.

[0065] In one embodiment provided in this specification, the initial video is a man fishing on a boat and catching a big fish. The description text of the video is "The man is fishing on a boat and catches a big fish." First, the main description object is determined by extracting the subject (i.e., "man"), and a detailed action and appearance description is generated for this subject. The action description includes: the man stands by the boat, holding a fishing rod, wearing a camouflage long-sleeved shirt and a blue baseball cap, concentrating on looking at the water surface, and there are fish jumping and splashing in the water. The man bends down and reaches out to grab the struggling fish, splashing water. In the end, he successfully caught the fish and held it up in front of the camera to show it. In the appearance description, the man's clothing is recorded in detail, such as wearing a camouflage long-sleeved shirt, gray shorts and a blue baseball cap, and wearing sunglasses.

[0066] Then, each video frame is processed, a frame is randomly selected, a bounding box of the target subject (man) is generated, and the bounding box is converted into a mask to obtain a specific description of the corresponding area. Then, feature information such as the subject wearing camouflage clothes is confirmed based on the brief description at the region level. Finally, a structured sample entity description information is generated: "The man stands by the boat, wearing a camouflage long-sleeved shirt, gray shorts and a blue baseball cap, looking at the water intently, reaching out to grab the struggling fish and successfully lifting it to show it to the camera, showing concentration and calmness throughout the process."

[0067] By selecting sample data for training the target video processing model from the initial data set so that the target video processing model has the ability to focus on specific entities in the video, the model's ability to identify and understand key objects can be effectively improved, thereby enhancing its application effect in complex video scenes. This screening process ensures that the model's training data is highly targeted, allowing it to more accurately capture the characteristics and behaviors of the target entity, while reducing interference caused by irrelevant information, improving the overall performance and reliability of the model, and thus having greater adaptability and stability in diverse video content.

[0068] Further, determining at least one sample video in each initial video based on the description text of each initial video includes:

[0069] Obtain the number of subjects in the initial video description text corresponding to each initial video;

[0070] At least one sample video is determined based on the number of subjects corresponding to each initial video.

[0071] In practical applications, the number of subjects is the number of core characters or entities in the initial video description text, which is used to help the system identify key objects in the video and screen them to ensure that the model focuses on the main characters or entities in the video.

[0072] Exemplarily, the number of subjects can be understood as the number of nouns used in the description text to refer to the main entities in the video. For example, in the text describing "a man fishing by the river and successfully catching a fish", "man" belongs to the subject, so the number of subjects in this description is 1; in the text describing "a man running on the grass with two dogs", "man" and "dog" are subjects, so the number of subjects in this description is 2. The analysis of the number of subjects helps the system select suitable sample videos in the training data, especially in multi-entity scenarios, which can help the model distinguish different roles or objects. By determining the number of subjects, the system can give priority to videos containing the target number of subjects as samples to ensure that the model can effectively learn multi-role scenarios. For example, in traffic monitoring, the system can filter out videos containing multiple vehicles or pedestrians to improve the model's recognition ability in multi-entity scenarios.

[0073] Step 104: Obtain an initial video processing model based on the sample video, the sample entity identification information and the sample entity description text, wherein the initial video processing model is used to generate an entity description text corresponding to the video, and the entity description text represents the characteristics of the entity to be identified in the video.

[0074] In practical applications, the initial video processing model is a model built based on target sample data, which is used to generate description text of entities in the video. The initial video processing model can be understood as a model trained based on target sample data to generate description text of entities to be identified in the video. By learning the entity features in the sample video, the initial video processing model can generate description text of the entity corresponding to the sample entity identification information in the sample video, and characterize the appearance or behavior characteristics of the specified entity in the video.

[0075] Exemplarily, obtaining the initial video processing model can be understood as training a preliminary model that can identify and describe specific entities in the video by combining the sample video and its corresponding entity information and description text. The specific method can be to obtain a first video encoding unit and a text description unit, train the first video encoding unit through the sample video and the sample video description text, generate a second video encoding unit for identifying and describing the video content, and use the second video encoding unit and the text description unit to generate an initial video processing model; the key frames of the sample video can also be aligned with the sample entity identification information to train the model to recognize the features of different entities, and the label is enhanced through the sample entity description text, thereby improving the recognition effect of the model; based on the sample video and the sample entity description text, deep learning technology can be used to extract the feature vectors of the entities in the video, and the model can be trained under the guidance of the sample entity identification information so that it can generate the corresponding entity description text. This specification does not impose any restrictions on this.

[0076] By obtaining an initial video processing model for generating descriptive text of entities to be identified in the video, the recognition accuracy of the model for specific entities in the video can be effectively improved, so that the model can more accurately extract the key features of the entity, thereby generating descriptive text that meets actual needs, and then a target video processing model with higher accuracy can be obtained in the subsequent period.

[0077] Considering that the model needs to understand the video and accurately generate description text of entities in the video, the target sample data includes sample video description text;

[0078] Further, obtaining an initial video processing model according to the sample video, the sample entity identification information and the sample entity description text includes:

[0079] Obtaining a first video encoding unit and a text description unit;

[0080] Training the first video encoding unit based on the sample video and the sample video description text to generate a second video encoding unit;

[0081] An initial video processing model is acquired according to the sample entity description text and the sample entity identification information, as well as the second video encoding unit and the text description unit.

[0082] In practical applications, the first video encoding unit is an encoder that preliminarily processes sample videos, aiming to convert video content into processable feature data; the text description unit is a unit that processes and generates video content descriptions, parsing text and image information to generate detailed descriptions; the second video encoding unit is the output unit after training the first video encoding unit, which can enable the model to understand the content in the video more accurately for further analysis and processing.

[0083] Exemplarily, the first video encoding unit can be understood as being mainly responsible for receiving and preliminarily processing sample videos, and encoding the video content into feature vectors for subsequent analysis. The encoding unit converts the frame content in the video into data suitable for model understanding, and the second video encoding unit can be understood as an encoder formed after training and optimizing the first video encoding unit, which can provide higher-precision feature extraction in video processing. Compared with the first video encoding unit, the second video encoding unit has been trained with sample videos and descriptive texts, and can make the model understand the content in the video more accurately. For example, in a sample video recording forest animals, the second video encoding unit can identify basic objects such as animals and plants in the video, providing a basis for further feature extraction. Through the encoding of this unit, the video content is simplified into feature data, which is more suitable for machine processing.

[0084] The text description unit can be understood as a module that generates and processes text descriptions of video content. It generates accurate content descriptions by combining video encoding information with text. The role of this unit is to generate detailed semantic information for key entities and actions in the video so that the model can understand the video content more comprehensively. For example, when processing a video showing a landscape, the text description unit generates description information such as "mountains and trees next to the lake", enabling the model to generate more contextual interpretations based on its understanding of the video. The text description unit not only improves the semantic richness of the video content, but also provides the model with more detailed reference information.

[0085] The second video encoding unit is obtained by training with target sample data, and then combined with the text description unit to generate an initial video processing model. The initial video processing model can generate a text description based on the video, so that entity description text of a specific entity in the video can be generated subsequently.

[0086] Further, training the first video encoding unit based on the sample video and the sample video description text to generate a second video encoding unit includes:

[0087] Inputting the sample video into the first video encoding unit to obtain predicted video feature information generated by the first video encoding unit;

[0088] Inputting the predicted video feature information into the text description unit, and obtaining the predicted video description text generated by the text description unit;

[0089] The first video encoding unit is trained according to the sample video description text and the predicted video description text to generate a second video encoding unit.

[0090] In practical applications, the predicted video feature information is feature data about the video content extracted by the first video encoding unit, which is used to describe key attributes of the video such as vision and action; the predicted video description text is text data generated by the text description unit, which is used to describe key information in the predicted video feature information and provide a linguistic explanation of the video content.

[0091] The predicted video feature information can be understood as the feature vector or data representation generated by the first video encoding unit after inputting the sample video. These feature information include important visual features in the video, such as the color, shape, motion trajectory, etc. of the main object, which helps the model to effectively understand the core of the video content. The predicted video description text can be understood as a textual description generated by the text description unit based on the predicted video feature information, which provides a detailed language description of the video content. The predicted video description text not only provides an easy-to-understand explanation for the model, but also enables the video content to be expressed in natural language. For example, in the example of an athlete's competition video, the predicted video description text can generate a description of "the athlete sprints quickly and jumps to the finish line", which intuitively summarizes the athlete's actions and behaviors.

[0092] It should be noted that, considering the accuracy of the generated entity text description, the text description unit can be any large language model, and the video encoding unit can be any deep learning model for extracting video features. Since training large language models and deep learning models for extracting video features requires relatively high costs, in order to improve the training efficiency of the model, a mapping subunit can be added after the deep learning model for extracting video features, so that only the mapping subunit needs to be adjusted when the subsequent training obtains the second video encoding unit.

[0093] Obtaining the second video encoding unit by training the first video encoding unit helps further training of the model to generate more accurate and detailed feature explanations and improve the expressiveness of the model in video understanding tasks.

[0094] Considering that when performing video recognition, it is necessary to receive the user's annotation information for a specific entity, the sample entity identification information includes the sample entity annotation information;

[0095] Acquiring an initial video processing model according to the sample entity description text and the sample entity identification information, as well as the second video encoding unit and the text description unit, including:

[0096] Obtaining a first entity annotation coding unit;

[0097] Processing the sample video and the sample entity annotation information according to the first entity annotation encoding unit, the second video encoding unit and the text description unit to generate a predicted entity description text;

[0098] Based on the predicted entity description text and the sample entity description text, training the first entity annotation encoding unit to generate a second entity annotation encoding unit;

[0099] An initial video processing model is obtained according to the second video encoding unit, the second entity annotation encoding unit and the text description unit.

[0100] In practical applications, the sample entity annotation information is the annotation data used to identify specific entities in the video, including the specific position or outline of the corresponding entity; the first entity annotation encoding unit is a unit that preliminarily processes and encodes the sample entity annotation information, and is used to extract key information in order to generate a more accurate entity description; the predicted entity description text is text information generated by the model, which is used to describe the characteristics of the annotated entity in detail; the second entity annotation encoding unit is a trained and optimized encoder with higher accuracy for further processing of entity annotation information.

[0101] Sample entity annotation information can be understood as the annotation data marked in the sample video for identifying specific entities, including the coverage information of the entity's shape, position, or other visual features. For example, in a video recording someone running in a park, the sample entity annotation information may include a bounding box around the runner, allowing the model to accurately lock on to the target object in the video. This annotation information provides basic data for subsequent encoding and description, ensuring that the model focuses on specific targets in multi-entity scenes.

[0102] The first entity annotation encoding unit can be understood as an encoder unit responsible for preliminary processing of sample entity annotation information, which is used to extract and encode key information in the annotation data. The first entity annotation encoding unit converts the annotation information into feature data so that the model can recognize the details of a specific entity in the video. The second entity annotation encoding unit can be understood as a trained and optimized entity annotation encoding unit, which has higher accuracy and recognition ability than the first entity annotation encoding unit. After multiple rounds of training, this encoding unit can extract and process annotation information more accurately, thereby improving the segmentation and recognition capabilities of the model in complex scenes.

[0103] The predicted entity description text can be understood as a text description generated according to the feature information generated by the first entity annotation encoding unit, which is used to present the features of the entity in a linguistic way. The predicted entity description text will describe the features of the annotated object in detail, including appearance, location and action.

[0104] Since the initial video processing model includes the second entity annotation coding unit obtained through training, the model can have higher accuracy and stability when identifying and processing specific entities in the video, thereby generating more targeted and detailed entity description text. Through the optimization of this coding unit, the model can effectively segment and identify target entities in diverse and complex scenarios, reducing background interference and the possibility of misidentification, ensuring that the output of the model is more in line with actual needs.

[0105] Step 106: Obtain a video recognition layer, and generate a reference video processing model based on the initial video processing model and the video recognition layer.

[0106] In practical applications, the video recognition layer is a model structure used for video processing, which is used to complete video processing tasks to identify specific objects in video content; the reference video processing model is a model obtained by splicing the video recognition layer on the basis of the initial model, which is used to verify and optimize the recognition ability of the model.

[0107] It should be noted that the application scenario of video processing can be any application scenario for video recognition. Taking the application scenario of a humanoid robot as an example, the obtained target data processing model is used to identify the video acquired by the robot, and the entities that need to be paid attention to and their movement conditions in the video can be accurately determined, so that the robot can complete the corresponding task more effectively. In the application scenario of intelligent monitoring, the obtained target data processing model is used to identify the images taken by the monitoring camera, and the camera can make judgments based on the key objects and their dynamic changes in the real-time video, thereby providing accurate basis for subsequent execution operations and realizing intelligent operation of the monitoring system. Taking the application scenario of autonomous driving as an example, the obtained target data processing model is used to identify the road condition video collected by the vehicle camera, and the dynamic information of key entities in the video, such as pedestrians, vehicles and traffic signals, can be analyzed in real time, so that the autonomous driving system can more accurately judge the road condition and perform corresponding driving operations, thereby improving driving safety and reaction speed. Taking the application scenario of virtual reality as an example, the obtained target data processing model is used to identify the virtual environment of user interaction, and the interactive objects and scene details in the virtual scene can be accurately located, thereby enhancing the user's immersive experience, and enabling the virtual reality system to make corresponding real-time feedback according to the user's actions, to achieve a more interactive virtual experience, etc. This manual does not impose any restrictions on this.

[0108] Exemplarily, the video recognition layer can be understood as a neural network structure for accurately segmenting and marking key objects in a video. The reference video processing model can be understood as an improved model generated by combining the video recognition layer on the basis of the initial video processing model, for realizing video recognition. The reference video processing model combines the basic recognition capabilities of the initial model with the segmentation and marking capabilities of the video recognition layer to significantly improve the recognition accuracy in complex scenes.

[0109] Further, generating a reference video processing model based on the initial video processing model and the video recognition layer includes:

[0110] Acquire a video entity description layer based on the initial video processing model;

[0111] A reference video processing model is generated according to the video entity description layer and the video recognition layer.

[0112] In practical applications, the video entity description layer is a neural network layer used to generate entity description text of a specific entity in a video. Acquiring the video entity description layer based on the initial video processing model can be understood as using the initial video processing model to generate entity description text of a specific entity in a video. Therefore, the method of determining the video entity description layer can directly determine the initial video processing model as the video entity description layer, or can quantize the initial video processing model as the video entity description layer, etc. This specification does not impose any restrictions on this.

[0113] Step 108: Train the reference video processing model according to the sample entity identification information, the sample video and the sample video processing result to obtain a target video processing model.

[0114] In practical applications, the target video processing model is the final model used in practical applications, which has been fully trained to meet the requirements of identifying specific entities in the video. The target video processing model can be understood as the final model version after comprehensive training, which has the ability to identify entities in specific video content, can be stably applied in practical environments, has higher recognition ability and generalization, and can accurately identify specific entities in the video.

[0115] Considering that the target video processing model needs to realize video recognition, the reference video processing model includes a video entity description layer and a video recognition layer;

[0116] Further, training the reference video processing model according to the sample entity identification information, the sample video and the sample video processing result to obtain a target video processing model includes:

[0117] Processing the sample video and the sample entity identification information based on the video entity description layer to obtain a reference entity description text;

[0118] Inputting the sample video and the reference entity description text into the video recognition layer to generate a reference prediction video processing result;

[0119] According to the reference prediction video processing result and the sample video processing result, training the video entity description layer to obtain an intermediate video processing model;

[0120] A target video processing model is obtained based on the intermediate video processing model.

[0121] In practical applications, the reference entity description text is the description text for specific entities in the sample video generated by the model in the process of obtaining the target video processing model; the reference predicted video processing result is the video processing result for the sample video generated by the model in the process of obtaining the target video processing model; the intermediate video processing model is a neural network model that can perform preliminary video processing.

[0122] Exemplarily, the reference entity description text can be understood as a textual description generated by the model after identifying a specific entity in a sample video when generating a target video processing model. This type of description text provides a specific language description of the appearance, action or position of the entity, which helps the model to deepen its understanding of the entity characteristics during the training process. The reference prediction video processing result can be understood as the preliminary processing result of the sample video generated by the model during the training process of the target video processing model, which is usually used to verify the recognition and segmentation effect of the model. The reference prediction video processing result can provide basic information and area annotations of the entity in the video. The predicted entity description text can be understood as a detailed text description of a specific entity generated by the model based on the sample video, which further enriches the characteristic information of the entity. This type of description text not only describes the appearance and position of the entity, but may also cover its dynamic behavior. The intermediate video processing model can be understood as a neural network model formed after partial training, which has basic video processing and analysis capabilities and is used for the gradual optimization of the target model. When training the target video processing model, the intermediate video processing model plays a bridging role, allowing the model to gradually learn and adapt to different video features.

[0123] The method of obtaining the target video processing model based on the intermediate video processing model can directly determine the intermediate video processing model as the target video processing model, or it can be to train the intermediate video processing model with richer sample data to obtain the target video processing model, etc. This manual does not impose any limitations on this.

[0124] By gradually training the intermediate models, the target video processing model can gradually improve its accuracy and adaptability to complex scenes on a stable basis.

[0125] Considering that questions and answers are performed on videos in actual applications, the sample entity identification information includes sample entity annotation information and / or sample question information;

[0126] Processing the sample video and the sample entity identification information based on the video entity description layer to obtain a reference entity description text includes:

[0127] The sample video, the sample entity annotation information and / or the sample question information are input into the video entity description layer to obtain a reference entity description text.

[0128] In practical applications, the sample question information is a question text for a specific entity in the model. The sample question information can be understood as question information describing a specific task for a specific entity in the video, such as "What is the man in the video doing?", "Please mark the man in the video", etc. This manual does not impose any restrictions on this.

[0129] Exemplarily, the structure of the target video processing model is as follows Figure 2 As shown, Figure 2 It is a structural diagram of a target video processing model provided by an embodiment of the present specification, wherein a text description unit in a video entity description layer generates a video entity description text according to video feature information generated by a video encoding unit in the video entity description layer according to a video, entity annotation feature information generated by an entity annotation encoding unit in the video entity description layer, and question information corresponding to the video, and then a video recognition layer generates a video recognition result according to the video entity description text and the video.

[0130] The target video processing model obtained by training with sample entity annotation information and / or the sample question information can support both regional understanding and visual positioning in the subsequent video recognition process. By introducing the representation of the target region, the original global information and text information are used as inputs of the large language model to generate a fine-grained understanding of the region. And while the large language model generates text, it generates tags for specific targets, which are mapped to the positioning model and end-to-end fine-tuned to achieve regional positioning.

[0131] Further, obtaining a target video processing model based on the intermediate video processing model includes:

[0132] Acquire a model adjustment unit, and generate an intermediate video entity description layer according to the model adjustment unit and the video entity description layer in the intermediate video processing model;

[0133] Inputting the sample video and the sample entity identification information into the intermediate video entity description layer to obtain an intermediate entity description text;

[0134] Inputting the intermediate entity description text and the sample video into the video recognition layer to obtain an intermediate prediction video processing result;

[0135] The model adjustment unit in the intermediate video entity description layer and the video recognition layer are trained based on the intermediate prediction video processing result and the sample video processing result to obtain a target video processing model.

[0136] In practical applications, the model adjustment unit is a model layer that is further optimized after the initial training is completed.

[0137] Exemplarily, the model adjustment unit can be understood as a structured layer used by the model to capture the overall characteristics of the data during training, which is commonly represented in the form of a matrix. The main function of this layer is to help the model learn the basic pattern of the data. Since the model adjustment unit focuses on extracting more relevant information from the input data when processing data, rather than processing all data dimensions, it is preferred that the rank of the model adjustment unit is smaller than the rank of the intermediate data processing model, so that the model adjustment unit can capture important features more efficiently and reduce computational complexity. This structure not only reduces the resource consumption of the model, but also improves the performance of the model in specific tasks, especially when processing high-dimensional, noisy data. A lower rank means that the model can focus on key data dimensions, thereby reducing the interference of redundant information, and thus can improve the accuracy of the model in identifying specific objects in the video while reducing the consumption of model training.

[0138] By applying the solution of the embodiments of this specification, the deep alignment of each module in space, time and semantic features is achieved by gradually training the video encoding unit, the entity annotation encoding unit and the text description unit, strengthening the information exchange and sharing between modules, and further improving the recognition and segmentation accuracy of the object meaning and the corresponding area. In addition, a multi-stage training strategy is adopted to gradually optimize the performance of each module, so that the model can better cope with fine-grained regional understanding and target positioning tasks in complex scenes. Based on the constructed fine-grained regional understanding dataset, the generalization ability of the model in complex video scenes is significantly enhanced, making it more adaptable and reliable when processing different video tasks, thereby improving the accuracy of identifying specific objects in the video.

[0139] Corresponding to the above method embodiment, this specification also provides a video processing method embodiment, see Figure 3 , Figure 3 A flow chart of a video processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0140] Step 302: Obtain the video to be processed and the identification information of the entity to be identified.

[0141] Step 304: Input the video to be processed and the identification information of the entity to be identified into the target video processing model to obtain the description text of the entity to be identified and the video processing result, wherein the target video processing model is trained according to the above-mentioned video processing model training method, and the video processing result is generated according to the description text of the entity to be identified, and the description text of the entity to be identified represents the characteristics of the entity corresponding to the identification information of the entity to be processed in the video to be processed.

[0142] In practical applications, the video to be processed is the original video data that requires entity recognition, such as a scene containing a man rowing; the entity identification information to be identified is the identification information that describes the specific object that needs to be identified in the video, specifically the "man" and its related actions and the identification information of the boat, which is used to guide the model to identify and recognize the areas of the man and the boat; the entity description text to be identified is the descriptive text paragraph of the object that needs to be identified in the video; the video processing result is the output of the target video processing model after segmenting the video content, presenting the independent areas or outlines of the man and the boat.

[0143] For example, the video to be processed can be understood as the original video material containing the objects to be identified and segmented, such as a complete video recording a man rowing a boat in a lake. The video content may not only contain the man and the boat, but also the surrounding lake water, sky and other background scenes. By processing the video to be processed, the model can separate the man and the boat from the background and provide separate objects for subsequent detailed analysis.

[0144] The identification information of the entity to be identified can be understood as the identification information of the target entity to be focused on in the video to be processed, such as the specific description of the "man" and "boat" in the video. The identification information may include the man's posture, the rowing action, the shape of the boat, etc. It can also be the annotation information generated by the user by clicking a point or drawing a box on any frame in the video to help the segmentation model accurately identify and separate the man and the boat area.

[0145] The video processing result can be understood as follows: after receiving the video to be processed and the identification information of the entity to be identified, the target video processing model first obtains the entity description text that describes the performance of the user-specified entity in the video, and then segments the video according to the above entity description text and outputs it. For example, the entity description text "a man standing by the boat, wearing a camouflage long-sleeved shirt, gray shorts and a blue baseball cap, looking at the water intently, reaching out to grab the struggling fish and successfully lifting it up, showing it to the camera, showing concentration and composure throughout the process" is used to perform video segmentation to separate independent areas or outlines of the man and the boat.

[0146] The above is a schematic scheme of a video processing method of this embodiment. It should be noted that the technical scheme of the video processing method and the technical scheme of the video processing model training method described above are of the same concept, and the details not described in detail in the technical scheme of the video processing method can be found in the description of the technical scheme of the video processing model training method described above.

[0147] By applying the solution of the embodiments of this specification, by inputting the video to be processed and the entity identification information into the target video processing model, the target video processing model can make full use of the spatial, temporal and semantic features obtained during the training process to achieve accurate video segmentation. In addition, based on multi-stage optimization training, the model has a higher ability to adapt to complex scenes and fine-grained understanding ability, which can significantly improve the recognition and segmentation accuracy of object areas in the video to be processed, especially when combining object semantics and entity identification information, it has a higher parsing ability, thereby improving the accuracy of segmenting specific entities in the video.

[0148] Corresponding to the above method embodiment, this specification also provides a video processing method embodiment applied to a terminal device, see Figure 4 , Figure 4 A flowchart of a video processing method applied to a terminal device according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0149] Step 402: Obtain the video to be processed and the annotation information of the entity to be identified drawn by the user for the video to be processed.

[0150] Step 404: Determine identification information of the entity to be identified based on the annotation information of the entity to be identified, wherein the identification information of the entity to be identified represents a description of the entity to be selected by the user in the video to be processed.

[0151] Step 406: Generate a video processing instruction based on the video to be processed and the entity identification information to be identified, and send the video processing instruction to the cloud side device, so that the cloud side device generates the entity description text to be identified and the video processing result corresponding to the video processing instruction according to the target video processing model, wherein the target video processing model is obtained by training according to the above-mentioned video processing model training method.

[0152] Step 408: Receive the description text of the entity to be identified and the video processing result sent by the cloud-side device.

[0153] In actual applications, cloud-side devices are remote servers that provide powerful computing power and data storage, and are used to process and store large-scale video data; end-side devices are local devices that perform video acquisition and preliminary analysis, such as smartphones, personal computing devices, etc., which usually interact with cloud-side devices; video processing instructions are specific commands sent by end-side devices, which contain the video to be processed and the entity identification information to be identified, and are used to instruct cloud-side devices to process the video.

[0154] Cloud-side devices can be understood as remote servers with high computing power and storage capacity, which are specially used to process large amounts of data, especially playing a core role in complex video processing tasks. Cloud-side devices are able to receive video processing instructions from end-side devices and use their powerful computing power to perform detailed analysis of the processed video. For example, in an intelligent monitoring system, cloud-side devices can receive video data from multiple cameras and process them centrally to generate accurate segmentation and identification information of different objects, providing users with real-time monitoring results.

[0155] The end-side device can be understood as a local device that is responsible for video acquisition, preliminary processing, and interaction with cloud-side devices, such as a smartphone, camera, or smart home device. The entity annotation information to be identified can be understood as the annotation information used to identify the object specified by the user. It can be a point at any frame point in the input video, or a drawn box, a drawn curve, etc. to specify the object of interest.

[0156] The end-side device first collects the video content and the entity annotation information drawn by the user, then uses the entity annotation information to generate entity identification information, and generates video processing instructions based on the above video and entity annotation information, and sends the instructions to the cloud-side device for further processing. The end-side device is particularly critical in low-resource scenarios because it can perform preliminary tasks locally while relying on cloud-side devices for high-precision video calculations. For example, after a smart camera detects that someone has entered the monitoring range, it can use the video clip as the video to be processed and automatically generate entity identification information. Then, it generates video processing instructions based on the above video to be processed and entity identification information, and sends the video processing instructions to the cloud for subsequent processing.

[0157] The video processing instruction can be understood as a specific operation instruction generated by the terminal device and sent to the cloud device, which contains the video to be processed and the identification information of the entity to be identified. The instruction is used to request the cloud device to perform video processing on the video content to obtain the independent area of ​​the target entity and the description text of the user-specified entity. For example, the terminal device may detect that there are multiple characters in the video, and send the video clip and the identification information of the specific character to be processed to the cloud through the video processing instruction, ensuring that the cloud device can generate text describing the character corresponding to the task identification information in the video clip, and then quickly locate and segment the outline and action of the key character based on the text to generate the video processing result.

[0158] The above is a schematic scheme of a video processing method applied to a terminal device of this embodiment. It should be noted that the technical scheme of the video processing method applied to the terminal device belongs to the same concept as the technical scheme of the above video processing method. For details not described in detail in the technical scheme of the video processing method applied to the terminal device, please refer to the description of the technical scheme of the above video processing method.

[0159] The solution of the embodiment of this specification is applied to effectively combine the entity identification information annotated by the user with the content of the video to be processed, so that the cloud-side device can generate accurate entity description text and video processing results based on its multi-stage optimized training model. Because this method relies on the powerful computing power of the cloud-side device, it can efficiently separate the target entity in complex scenes and generate fine-grained description text, which helps to meet the data interaction needs between multiple devices in real time, and ultimately ensures that the recognition and processing accuracy of key entities in the video reaches a high level.

[0160] Corresponding to the above method embodiment, this specification also provides a video processing method embodiment applied to a cloud-side device, see Figure 5 , Figure 5 A flowchart of a video processing method applied to a cloud-side device provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0161] Step 502: receiving a video processing instruction sent by a terminal-side device, wherein the video processing instruction includes the video to be processed and the identification information of the entity to be identified.

[0162] Step 504: Input the video to be processed and the identification information of the entity to be identified into the target video processing model to obtain the description text of the entity to be identified and the video processing result, wherein the target video processing model is trained according to the above-mentioned video processing model training method, and the video processing result is generated according to the description text of the entity to be identified, and the description text of the entity to be identified represents the characteristics of the entity corresponding to the identification information of the entity to be processed in the video to be processed.

[0163] Step 506: Send the description text of the entity to be identified and the video processing result to the terminal side device.

[0164] The above is a schematic scheme of a video processing method applied to a cloud-side device in this embodiment. It should be noted that the technical scheme of the video processing method applied to the cloud-side device and the technical scheme of the above-mentioned video processing method belong to the same concept, and the details of the technical scheme of the video processing method applied to the cloud-side device that are not described in detail can be referred to the description of the technical scheme of the above-mentioned video processing method.

[0165] The solution of the embodiments of this specification can respond quickly to the video processing instructions of the receiving end device, significantly improving the recognition and segmentation accuracy of the object area in the video to be processed. Combining the object semantics and entity identification information, the target video processing model makes full use of the multi-dimensional features obtained during the training process to achieve efficient video processing. In addition, since the target video processing model is obtained based on multi-stage optimization training in the cloud, it has a stronger ability to handle complex scenes and understand fine-grained areas, thereby improving the accuracy of segmenting specific entities in the video.

[0166] See also Figure 6 , Figure 6 An architecture diagram of a video processing system provided by an embodiment of the present specification is shown, and the video processing system may include a client 100 and a server 200;

[0167] The client 100 is used to obtain a video to be processed and annotation information of an entity to be identified drawn by a user for the video to be processed; determine identification information of an entity to be identified based on the annotation information of the entity to be identified, wherein the identification information of the entity to be identified represents a description of the entity to be selected by the user in the video to be processed; generate a video processing instruction according to the video to be processed and the identification information of the entity to be identified, and send the video processing instruction to the server 200;

[0168] The server 200 is used to receive a video processing instruction sent by the client 100, wherein the video processing instruction includes a video to be processed and identification information of an entity to be identified; input the video to be processed and the identification information of the entity to be identified into a target video processing model, and obtain a description text of the entity to be identified and a video processing result, wherein the target video processing model is obtained by training according to the above-mentioned video processing model training method, and the video processing result is generated according to the description text of the entity to be identified, and the description text of the entity to be identified represents the characteristics of the entity corresponding to the identification information of the entity to be processed in the video to be processed; and send the description text of the entity to be identified and the video processing result to the client 100;

[0169] The client 100 is also used to receive the description text of the entity to be identified and the video processing result sent by the server 200.

[0170] By applying the solution of the embodiments of this specification, the collaborative work of the client and the server can achieve a rapid response to video processing instructions. The server uses the target video processing model to effectively process video processing requests from the client, and combines object semantics and entity identification information to significantly improve the recognition and segmentation accuracy of object areas in the video to be processed. In addition, the target video processing model based on multi-stage optimization training on the server has high adaptability and fine-grained understanding capabilities for complex scenes. Even in multi-object interactions and complex environments, the system still shows excellent stability and processing capabilities.

[0171] The video processing system may include multiple clients 100 and a server 200, wherein the client 100 may be referred to as a terminal device and the server 200 may be referred to as a cloud device. Multiple clients 100 may establish a communication connection through the server 200. In the video processing scenario, the server 200 is used to provide video processing services between multiple clients 100. Multiple clients 100 may serve as a sender or a receiver, respectively, and realize communication through the server 200.

[0172] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the video processing scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a description text of the entity to be identified and a video processing result according to the data stream, and pushes the description text of the entity to be identified and the video processing result to other clients that have established communication.

[0173] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0174] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device, and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0175] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that provide support for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content De l ivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0176] It is worth noting that the video processing method provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the video processing method provided in the embodiments of this specification. In other embodiments, the video processing method provided in the embodiments of this specification may also be jointly executed by the client and the server.

[0177] The following combination Figure 7 Taking the application of the video processing model training method provided in this specification in the video segmentation model training as an example, the video processing model training method is further described. Figure 7 A processing flow chart of a video segmentation model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0178] Step 702: Acquire an initial data set including a plurality of initial videos and description texts corresponding to each initial video.

[0179] Step 704: Use the large model to extract the subject of the description text corresponding to each initial video, and remove the initial videos corresponding to multiple subjects to obtain at least one reference video.

[0180] Step 706: Use the large model to obtain at least one sample video by matching the subject to a reference video that only describes one frame.

[0181] Step 708: Apply the video segmentation model to obtain the bounding boxes corresponding to each entity in the sample video, and only retain the bounding box corresponding to the subject.

[0182] Step 710: convert the bounding box corresponding to the subject into a mask, obtain sample entity annotation information corresponding to each sample video, and construct a sample data set including multiple sample data.

[0183] Step 712: Freeze the modules except the video encoding unit to train the video encoding unit to achieve alignment between the video features and the text description unit.

[0184] Step 714: Freeze the modules except the entity annotation encoding unit to train the entity annotation encoding unit to achieve alignment between the regional features and the text description unit.

[0185] Step 716: Obtain a region bounding box using the sample entity annotation information, and apply a mask tool to convert it into a region mask.

[0186] Step 718: Input the region mask into the entity annotation encoding unit so that the sample entity annotation information is combined with the information of other frames of the target sample video for temporal fusion to generate a visual mark.

[0187] Step 720: input the visual mark and the video feature information encoded by the video encoding unit into the text description unit, and perform semantic analysis to generate entity description text corresponding to the target entity in the sample video.

[0188] Step 722: parse the output of the text description unit and the target sample video through the video recognition layer to generate a predicted video segmentation result.

[0189] Step 724: Use the predicted video segmentation results and the sample video segmentation results to train the video encoding unit, entity annotation encoding unit and text description unit in the reference video segmentation model to generate an intermediate video segmentation model to achieve the video reference and positioning functions.

[0190] Step 726: Use the intermediate video segmentation model to obtain the intermediate prediction video segmentation result, use the sample video segmentation result and the intermediate prediction segmentation result to adjust the text description unit and the video recognition layer in the intermediate video segmentation model, and finally obtain the target video segmentation model.

[0191] By applying the scheme of the embodiments of this specification, it is possible to make full use of the spatial, temporal and semantic features in the initial data set, gradually optimize the training of the video encoding unit, the entity annotation encoding unit and the text description unit, and achieve a detailed understanding and effective segmentation of the video content. In particular, through the module training and time-series fusion operations at each stage, the model has stronger adaptability in complex scenarios, and can significantly improve the ability to accurately identify and segment target objects and regions in sample videos. In addition, combined with multiple rounds of adjustment and optimization, the final target video segmentation model has higher accuracy and robustness when processing the positioning and segmentation tasks of specific entities in the video, and is suitable for various complex practical application scenarios, thereby improving the reliability and efficiency of the overall video segmentation process.

[0192] Corresponding to the above method embodiment, this specification also provides a video processing model training device embodiment, Figure 8 FIG. 1 is a schematic diagram showing the structure of a video processing model training device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0193] The sample acquisition module 802 is configured to acquire target sample data, wherein the target sample data includes a sample video, sample entity identification information, a sample entity description text, and a sample video processing result;

[0194] The model acquisition module 804 is configured to obtain an initial video processing model according to the sample video, the sample entity identification information and the sample entity description text, wherein the initial video processing model is used to generate an entity description text corresponding to the video, and the entity description text represents the characteristics of the entity to be identified in the video;

[0195] The model generation module 806 is configured to obtain a video recognition layer and generate a reference video processing model based on the initial video processing model and the video recognition layer;

[0196] The model training module 808 is configured to train the reference video processing model according to the sample entity identification information, the sample video and the sample video processing result to obtain a target video processing model.

[0197] Optionally, the sample acquisition module 802 is further configured to:

[0198] Acquire an initial data set, wherein the initial data set includes at least one initial video and video description text corresponding to each initial video;

[0199] Determining at least one sample video in each initial video based on the description text of each initial video;

[0200] Target sample data is obtained according to each sample video and the sample video description text corresponding to each sample video.

[0201] Optionally, the sample acquisition module 802 is further configured to:

[0202] Obtain the number of subjects in the initial video description text corresponding to each initial video;

[0203] At least one sample video is determined based on the number of subjects corresponding to each initial video.

[0204] Optionally, the target sample data includes sample video description text;

[0205] The model acquisition module 804 is further configured to:

[0206] Obtaining a first video encoding unit and a text description unit;

[0207] Training the first video encoding unit based on the sample video and the sample video description text to generate a second video encoding unit;

[0208] An initial video processing model is acquired according to the sample entity description text and the sample entity identification information, as well as the second video encoding unit and the text description unit.

[0209] Optionally, the model acquisition module 804 is further configured to:

[0210] Inputting the sample video into the first video encoding unit to obtain predicted video feature information generated by the first video encoding unit;

[0211] Inputting the predicted video feature information into the text description unit, and obtaining the predicted video description text generated by the text description unit;

[0212] The first video encoding unit is trained according to the sample video description text and the predicted video description text to generate a second video encoding unit.

[0213] Optionally, the sample entity identification information includes sample entity annotation information;

[0214] The model acquisition module 804 is further configured to:

[0215] Obtaining a first entity annotation coding unit;

[0216] Processing the sample video and the sample entity annotation information according to the first entity annotation encoding unit, the second video encoding unit and the text description unit to generate a predicted entity description text;

[0217] Based on the predicted entity description text and the sample entity description text, training the first entity annotation encoding unit to generate a second entity annotation encoding unit;

[0218] An initial video processing model is obtained according to the second video encoding unit, the second entity annotation encoding unit and the text description unit.

[0219] Optionally, the model generation module 806 is further configured to:

[0220] Acquire a video entity description layer based on the initial video processing model;

[0221] A reference video processing model is generated according to the video entity description layer and the video recognition layer.

[0222] Optionally, the reference video processing model includes a video entity description layer and a video recognition layer;

[0223] The model training module 808 is further configured to:

[0224] Processing the sample video and the sample entity identification information based on the video entity description layer to obtain a reference entity description text;

[0225] Inputting the sample video and the reference entity description text into the video recognition layer to generate a reference prediction video processing result;

[0226] According to the reference prediction video processing result and the sample video processing result, training the video entity description layer to obtain an intermediate video processing model;

[0227] A target video processing model is obtained based on the intermediate video processing model.

[0228] Optionally, the sample entity identification information includes sample entity annotation information and / or sample question information;

[0229] The model training module 808 is further configured to:

[0230] The sample video, the sample entity annotation information and / or the sample question information are input into the video entity description layer to obtain a reference entity description text.

[0231] Optionally, the model training module 808 is further configured to:

[0232] Acquire a model adjustment unit, and generate an intermediate video entity description layer according to the model adjustment unit and the video entity description layer in the intermediate video processing model;

[0233] Inputting the sample video and the sample entity identification information into the intermediate video entity description layer to obtain an intermediate entity description text;

[0234] Inputting the intermediate entity description text and the sample video into the video recognition layer to obtain an intermediate prediction video processing result;

[0235] The model adjustment unit in the intermediate video entity description layer and the video recognition layer are trained based on the intermediate prediction video processing result and the sample video processing result to obtain a target video processing model.

[0236] Applying the scheme of the embodiments of this specification, the sample acquisition module, the model acquisition module, the model generation module and the model training module work together to ensure the systematicness and efficiency of the training process. On the basis of screening the initial data layer by layer, the sample acquisition module ensures the high-quality input of the target sample data, thereby providing a reliable foundation for model training. The model acquisition module generates the initial and reference video processing models through multi-stage feature alignment, thereby enhancing the multi-dimensional understanding of the video content. Through multiple rounds of optimization and adjustment, the model training module enables the target video processing model to have stronger fine-grained understanding and target positioning capabilities in complex video scenes, especially under multi-object interactions and complex semantics, and achieves high-accuracy segmentation of specific entities. The modular design of the overall device greatly improves the system's ability to refine the processing of video content and the response efficiency, and is suitable for a variety of practical application scenarios.

[0237] The above is a schematic scheme of a video processing model training device of this embodiment. It should be noted that the technical scheme of the video processing model training device and the technical scheme of the video processing model training method described above belong to the same concept, and the details not described in detail in the technical scheme of the video processing model training device can be found in the description of the technical scheme of the video processing model training method described above.

[0238] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0239] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC).

[0240] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0241] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0242] Among them, the processor 920 is used to execute the following computer-executable instructions, which, when executed by the processor, can implement the steps of the above-mentioned video processing model training method, a video processing method, a video segmentation method applied to an end-side device, and a video processing method applied to a cloud-side device.

[0243] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the above-mentioned video processing model training method, video processing method, video processing method applied to terminal-side device and video processing method applied to cloud-side device belong to the same concept. For the details not described in detail in the technical scheme of the computing device, please refer to the description of the technical schemes of the above-mentioned video processing model training method, video processing method, video processing method applied to terminal-side device and video processing method applied to cloud-side device.

[0244] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned video processing model training method, a video processing method, a video processing method applied to an end-side device, and a video processing method applied to a cloud-side device.

[0245] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the above-mentioned video processing model training method, video processing method, video processing method applied to terminal-side device and video processing method applied to cloud-side device belong to the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical schemes of the above-mentioned video processing model training method, video processing method, video processing method applied to terminal-side device and video processing method applied to cloud-side device.

[0246] An embodiment of the present specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-mentioned video processing model training method, a video processing method, a video processing method applied to an end-side device, and a video processing method applied to a cloud-side device.

[0247] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical schemes of the above-mentioned video processing model training method, video processing method, video processing method applied to terminal-side device and video processing method applied to cloud-side device belong to the same concept. For details not described in detail in the technical scheme of the computer program, please refer to the description of the technical schemes of the above-mentioned video processing model training method, video processing method, video processing method applied to terminal-side device and video processing method applied to cloud-side device.

[0248] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0249] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0250] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0251] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0252] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A video processing model training method, comprising: Acquire target sample data, wherein the target sample data includes a sample video, sample entity identification information, a sample entity description text, and a sample video processing result; Obtaining an initial video processing model according to the sample video, the sample entity identification information and the sample entity description text, wherein the initial video processing model is used to generate an entity description text corresponding to the video, and the entity description text represents the characteristics of the entity to be identified in the video; Acquire a video recognition layer, and generate a reference video processing model based on the initial video processing model and the video recognition layer; The reference video processing model is trained according to the sample entity identification information, the sample video and the sample video processing result to obtain a target video processing model.

2. The method according to claim 1, obtaining target sample data, comprising: Acquire an initial data set, wherein the initial data set includes at least one initial video and video description text corresponding to each initial video; Determining at least one sample video in each initial video based on the description text of each initial video; Target sample data is obtained according to each sample video and the sample video description text corresponding to each sample video.

3. The method according to claim 2, wherein determining at least one sample video in each initial video based on the description text of each initial video comprises: Obtain the number of subjects in the initial video description text corresponding to each initial video; At least one sample video is determined based on the number of subjects corresponding to each initial video.

4. The method according to claim 1, wherein the target sample data comprises sample video description text; Obtaining an initial video processing model according to the sample video, the sample entity identification information and the sample entity description text, including: Obtaining a first video encoding unit and a text description unit; Training the first video encoding unit based on the sample video and the sample video description text to generate a second video encoding unit; An initial video processing model is acquired according to the sample entity description text and the sample entity identification information, as well as the second video encoding unit and the text description unit.

5. The method according to claim 4, training the first video encoding unit based on the sample video and the sample video description text to generate the second video encoding unit, comprising: Inputting the sample video into the first video encoding unit to obtain predicted video feature information generated by the first video encoding unit; Inputting the predicted video feature information into the text description unit, and obtaining the predicted video description text generated by the text description unit; The first video encoding unit is trained according to the sample video description text and the predicted video description text to generate a second video encoding unit.

6. The method according to claim 4, wherein the sample entity identification information comprises sample entity annotation information; Acquiring an initial video processing model according to the sample entity description text and the sample entity identification information, as well as the second video encoding unit and the text description unit, including: Obtaining a first entity annotation encoding unit; Processing the sample video and the sample entity annotation information according to the first entity annotation encoding unit, the second video encoding unit and the text description unit to generate a predicted entity description text; Based on the predicted entity description text and the sample entity description text, training the first entity annotation encoding unit to generate a second entity annotation encoding unit; An initial video processing model is obtained according to the second video encoding unit, the second entity annotation encoding unit and the text description unit.

7. The method of claim 1, generating a reference video processing model based on the initial video processing model and the video recognition layer, comprising: Acquire a video entity description layer based on the initial video processing model; A reference video processing model is generated according to the video entity description layer and the video recognition layer.

8. The method of claim 1, wherein the reference video processing model comprises a video entity description layer and a video recognition layer; Training the reference video processing model according to the sample entity identification information, the sample video and the sample video processing result to obtain a target video processing model includes: Processing the sample video and the sample entity identification information based on the video entity description layer to obtain a reference entity description text; Inputting the sample video and the reference entity description text into the video recognition layer to generate a reference prediction video processing result; According to the reference prediction video processing result and the sample video processing result, training the video entity description layer to obtain an intermediate video processing model; A target video processing model is obtained based on the intermediate video processing model.

9. The method according to claim 8, wherein the sample entity identification information includes sample entity annotation information and / or sample question information; Processing the sample video and the sample entity identification information based on the video entity description layer to obtain a reference entity description text includes: The sample video, the sample entity annotation information and / or the sample question information are input into the video entity description layer to obtain a reference entity description text.

10. The method according to claim 8, obtaining a target video processing model based on the intermediate video processing model, comprising: Acquire a model adjustment unit, and generate an intermediate video entity description layer according to the model adjustment unit and the video entity description layer in the intermediate video processing model; Inputting the sample video and the sample entity identification information into the intermediate video entity description layer to obtain an intermediate entity description text; Inputting the intermediate entity description text and the sample video into the video recognition layer to obtain an intermediate prediction video processing result; The model adjustment unit in the intermediate video entity description layer and the video recognition layer are trained based on the intermediate prediction video processing result and the sample video processing result to obtain a target video processing model.

11. A video processing method, comprising: Obtain the video to be processed and the identification information of the entity to be identified; The video to be processed and the identification information of the entity to be recognized are input into the target video processing model to obtain a description text of the entity to be recognized and a video processing result, wherein the target video processing model is trained according to the method described in any one of claims 1 to 10, and the video processing result is generated based on the description text of the entity to be recognized, and the description text of the entity to be recognized represents the characteristics of the entity corresponding to the identification information of the entity to be processed in the video to be processed.

12. A video recognition method, applied to a terminal device, comprising: Obtaining a video to be processed and annotation information of entities to be identified drawn by a user for the video to be processed; Determining identification information of an entity to be identified based on the annotation information of the entity to be identified, wherein the identification information of the entity to be identified represents a description of the entity to be selected by the user in the video to be processed; Generate a video processing instruction according to the video to be processed and the identification information of the entity to be identified, and send the video processing instruction to a cloud-side device, so that the cloud-side device generates a description text of the entity to be identified and a video processing result corresponding to the video processing instruction according to a target video processing model, wherein the target video processing model is obtained by training according to the method described in any one of claims 1 to 10; Receive the entity description text and video processing results to be identified sent by the cloud-side device.

13. A video processing method, applied to a cloud-side device, comprising: A video processing instruction sent by a receiving terminal device, wherein the video processing instruction includes the video to be processed and the identification information of the entity to be identified; Input the video to be processed and the identification information of the entity to be identified into a target video processing model to obtain a description text of the entity to be identified and a video processing result, wherein the target video processing model is trained according to the method described in any one of claims 1 to 10, and the video processing result is generated according to the description text of the entity to be identified, and the description text of the entity to be identified represents the characteristics of the entity corresponding to the identification information of the entity to be processed in the video to be processed; The description text of the entity to be identified and the video processing result are sent to the terminal side device.

14. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 13 are implemented.

15. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

16. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images