Video analysis method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a video analysis method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a target video sequence and a text prompt word of a user; the target video sequence comprises a plurality of images, and each image comprises a to-be-analyzed target selected by a user; performing feature extraction on the target video sequence to obtain visual features of the target video sequence; performing feature extraction on the text cue words to obtain text features of the text cue words; and inputting the visual features and the text features into a pre-trained large language model to obtain a description text of the to-be-analyzed target output by the large language model. Through the above mode, the analysis and description of the specific target in the video can be completed, and the interested content of the user can be focused.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video analysis method, device, equipment, storage medium and program product. Background Art

[0002] With the development of artificial intelligence technology, the method of using large language models for video analysis has gradually been promoted.

[0003] In some related technologies, a global image feature sequence can be directly extracted from the original video using a visual encoder, and the global image feature sequence is mapped to the semantic space of a large language model to obtain a video description text output by the large language model.

[0004] However, existing video analysis methods can only extract global image features of videos, but have difficulty extracting local image features of videos. Therefore, in scenarios where only specific targets in the video need to be described, existing video analysis methods find it difficult to complete the analysis and description of specific targets in the video and cannot focus on content that users are interested in. Summary of the invention

[0005] The embodiments of the present application provide a video analysis method, apparatus, device, storage medium and program product to solve the technical problem that existing video analysis methods are difficult to complete the analysis and description of specific targets in the video and cannot focus on the content of interest to the user.

[0006] In a first aspect, an embodiment of the present application provides a video analysis method, comprising: obtaining a target video sequence and a user's text prompt words; the target video sequence comprises a plurality of images, each image comprising a target to be analyzed selected by the user; performing feature extraction on the target video sequence to obtain visual features of the target video sequence; performing feature extraction on the text prompt words to obtain text features of the text prompt words; inputting the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0007] In one embodiment, feature extraction is performed on a target video sequence to obtain visual features of the target video sequence, including: determining a target mask, depth features, video history features, and coding features of a first frame image; the first frame image is the first frame image of the target video sequence; the next frame image of the first frame image is used as the current frame image; based on the target mask of the current frame image and the previous frame image of the current frame image, the depth features of the current frame image are determined; based on the depth features of the current frame image and the video history features of the previous frame image of the current frame image, the coding features of the current frame image are determined; the coding features of the current frame image are input into a mask decoding network; The invention relates to a method for obtaining a target mask of a current frame image output by a mask decoding network; determining the video history feature of the current frame image based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image; taking the next frame image of the current frame image as the current frame image, returning the target mask based on the current frame image and the previous frame image of the current frame image, and determining the depth feature of the current frame image, until the target mask, depth feature, video history feature and coding feature of each image in the target video sequence are obtained; and taking the video history feature of the last frame image in the target video sequence as the visual feature of the target video sequence.

[0008] In one embodiment, the target mask, depth features, video history features and coding features of the first frame image are determined, including: determining the user's interaction information; inputting the first frame image and the interaction information into a segmentation model to obtain the target mask of the first frame image output by the segmentation model; splicing the first frame image and the target mask of the first frame image to obtain a spliced ​​image; performing feature extraction on the spliced ​​image based on a visual feature extraction network to obtain the depth features of the first frame image and the video history features of the first frame image; and extracting the coding features of the first frame image.

[0009] In one embodiment, after obtaining the target mask, depth features, video history features and encoding features of each image in the target video sequence, it also includes: based on the target mask of each image, performing target segmentation on each image respectively to obtain the target segmentation result of each image; and returning the target segmentation result of each image to the user.

[0010] In one embodiment, obtaining a target video sequence and a user's text prompt words includes: obtaining an initial video; responding to user interaction information, determining a target to be analyzed in the initial video and a processing time selected by the user based on the interaction information; determining a target video sequence based on the target to be analyzed in the initial video and a processing time selected by the user; and obtaining the user's text prompt words.

[0011] In one embodiment, visual features and text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model, including: dimensionally adjusting the visual features so that the visual features and the text features have the same feature dimensions; inputting the dimensionally adjusted visual features and text features into the pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0012] In a second aspect, an embodiment of the present application provides a video analysis device, comprising: an acquisition module, used to acquire a target video sequence and a user's text prompt words; the target video sequence comprises multiple images, each image contains a target to be analyzed selected by the user; a visual feature extraction module, used to extract features from the target video sequence to obtain visual features of the target video sequence; a text feature extraction module, used to extract features from text prompt words to obtain text features of the text prompt words; an analysis module, used to input visual features and text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-mentioned video analysis methods is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned video analysis methods.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned video analysis methods when executed by a processor.

[0016] The video analysis method, device, equipment, storage medium and program product provided by the embodiment of the present application obtain a target video sequence and a user's text prompt word; the target video sequence includes multiple images, each image contains a target to be analyzed selected by the user; feature extraction is performed on the target video sequence to obtain the visual features of the target video sequence; feature extraction is performed on the text prompt word to obtain the text features of the text prompt word; the visual features and text features are input into the pre-trained large language model to obtain the description text of the target to be analyzed output by the large language model. Through the above method, the user is allowed to select the target to be analyzed by himself, feature extraction is performed on the target video sequence containing the target to be analyzed selected by the user, visual features are obtained, and then feature extraction is performed on the user's text prompt word to obtain text features. The large language model is used to analyze the visual features and text features, and the description text of the target to be analyzed can be obtained, thereby completing the analysis and description of specific targets in the video and focusing on the content of interest to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the video analysis method provided in the embodiment of the present application.

[0019] Figure 2 This is the second flow chart of the video analysis method provided in the embodiment of the present application.

[0020] Figure 3 It is a flowchart of visual feature encoding provided in an embodiment of the present application.

[0021] Figure 4 It is a flowchart of feature fusion processing provided in an embodiment of the present application.

[0022] Figure 5 It is a structural schematic diagram of a video analysis device provided in an embodiment of the present application.

[0023] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] See also Figure 1 , Figure 1 This is one of the flow charts of the video analysis method provided in the embodiment of the present application. In the embodiment of the present application, the video analysis method is applied to the visual understanding system, and the video analysis method includes steps S110 to S140, and each step is as follows: S110: Obtain a target video sequence and a user's text prompt word.

[0026] The target video sequence includes multiple images, each of which contains a target to be analyzed selected by the user.

[0027] See also Figure 2 , Figure 2 This is the second flow chart of the video analysis method provided in the embodiment of the present application.

[0028] like Figure 2 As shown, the visual understanding system can be used to describe the content in a picture or video. In this embodiment, the visual understanding system can provide interaction with the user, and the user can select the target to be analyzed on the video by himself through the visual understanding system, and the visual understanding system can directly output the description of the target to be analyzed in the picture or video.

[0029] Specifically, the visual understanding system includes a user interaction processing module, a visual feature encoding module, a target mask decoding module and a graphic feature fusion module.

[0030] The user interaction processing module can obtain the target video sequence and the user's text prompt words.

[0031] S120: Extract features of the target video sequence to obtain visual features of the target video sequence.

[0032] S130: Extract features of the text prompt words to obtain text features of the text prompt words.

[0033] Specifically, the user's text prompt word prompt is input into the text encoding network to obtain the text feature R generated by the text encoding network T (i.e., text sequence features), R TThe dimension is [textlength, d], where textlength is the length of the text word (token) and d is the dimension of each word.

[0034] S140: Input the visual features and text features into the pre-trained large language model to obtain the description text of the target to be analyzed output by the large language model.

[0035] The video analysis method provided in the embodiment of the present application obtains a target video sequence and a user's text prompt word; the target video sequence includes multiple images, each image contains a target to be analyzed selected by the user; feature extraction is performed on the target video sequence to obtain the visual features of the target video sequence; feature extraction is performed on the text prompt word to obtain the text features of the text prompt word; the visual features and text features are input into a pre-trained large language model to obtain the description text of the target to be analyzed output by the large language model. Through the above method, the user is allowed to select the target to be analyzed by himself, feature extraction is performed on the target video sequence containing the target to be analyzed selected by the user to obtain visual features, and then feature extraction is performed on the user's text prompt word to obtain text features. The large language model is used to analyze the visual features and text features to obtain the description text of the target to be analyzed, thereby completing the analysis and description of specific targets in the video and focusing on the content of interest to the user.

[0036] In some embodiments, feature extraction is performed on a target video sequence to obtain visual features of the target video sequence, including: determining a target mask, depth features, video history features, and coding features of a first frame image; the first frame image is the first frame image of the target video sequence; the next frame image of the first frame image is used as the current frame image; based on the target mask of the current frame image and the previous frame image of the current frame image, the depth features of the current frame image are determined; based on the depth features of the current frame image and the video history features of the previous frame image of the current frame image, the coding features of the current frame image are determined; the coding features of the current frame image are input into a mask decoding network; The invention relates to a method for obtaining a target mask of a current frame image output by a mask decoding network; determining the video history feature of the current frame image based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image; taking the next frame image of the current frame image as the current frame image, returning the target mask based on the current frame image and the previous frame image of the current frame image, and determining the depth feature of the current frame image, until the target mask, depth feature, video history feature and coding feature of each image in the target video sequence are obtained; and taking the video history feature of the last frame image in the target video sequence as the visual feature of the target video sequence.

[0037] Please continue reading Figure 2The user interaction processing module is used to convert the interactive operations of the user's selected targets into system executable signals. The core of this module is the SAM model (Segment Anything Model).

[0038] Specifically, the visual understanding system can play the initial video (i.e., the original video), which may contain multiple targets, which may be people, objects, animals, plants, specific areas, etc.; while the user is watching the video, the user can pause the video at any time, and the picture will stay on the image of that moment frame. The user can interact with the system, select a target of interest in the image as the target to be analyzed, and determine the processing time, thereby generating interactive information; the processing time refers to the length of the video that the user wants the system to analyze.

[0039] Furthermore, the visual understanding system responds to the user's interactive information and can determine the target to be analyzed in the initial video and the processing time selected by the user based on the interactive information, and determine the target video sequence based on the target to be analyzed in the initial video and the processing time selected by the user.

[0040] For example, if the initial video is half an hour long, the user can select the target to be analyzed on the image at the 10th minute of the initial video and set the processing time to 15 minutes. That is, the user wants the system to analyze a video of 15 minutes in length. The visual understanding system can then extract the target video sequence based on the video clips from the 10th to the 25th minute of the initial video.

[0041] Understandably, the visual understanding system also needs to obtain the user's text prompts.

[0042] by Figure 2 For example, the user's text prompt is "What is the man doing?"

[0043] Optionally, the user may select the target to be analyzed by left-clicking the mouse or selecting a frame.

[0044] It should be noted that, in this embodiment, when the user selects a target to be analyzed on an image, the frame image is used as the first frame image I0 in the target video sequence, and the first frame image I0 is the first frame image of the target video sequence.

[0045] Furthermore, the user's interaction information is used as a prompt, and the first frame image I0 and the interaction information are input into the SAM model of the user interaction processing module to obtain the target mask mask0 of the first frame image I0 output by the SAM model.

[0046] The visual feature encoding module is used to generate visual features. The visual features should contain the following two types of information: one is the fusion of the mask information of the target to be analyzed, so that the description of the specified target can be achieved; the other is to contain all the image frame information so that the model has the ability to describe the entire video. With the idea of ​​semi-supervised tasks, the visual feature encoding module can store historical image features, combine historical results with the current image, and predict the current result. The specific approach is: store the historical features of the video frame in the memory, merge them with the previous target mask results and the current image features, update the historical features of the video and the current image features, and then decode the current image features to obtain the target mask result of the current image. In this process, the historical features of the video merge each frame of the image and its target mask information, which can be used to understand and describe specific targets in the video.

[0047] Furthermore, the visual feature encoding module performs splicing processing on the first frame image I0 and the target mask mask0 of the first frame image: the first frame image I0 and the target mask mask0 of the first frame image are spliced ​​in dimension to obtain a spliced ​​image; the spliced ​​image is input into a visual feature extraction network (such as a visual feature encoder ViT) for processing, and the visual feature extraction network can perform feature extraction processing on the spliced ​​image to obtain a depth feature R0 of the first frame image, and use the depth feature R0 of the first frame image as the video history feature RH0 of the first frame image; extract the encoding feature RN0 of the first frame image, and store the above-extracted feature in the server memory.

[0048] Furthermore, the next frame image of the first frame image is used as the current frame image.

[0049] See also Figure 3 , Figure 3 It is a flowchart of visual feature encoding provided in an embodiment of the present application.

[0050] like Figure 3 As shown, for each current frame image I n , can be based on the current frame image I n With the current frame image I n The previous frame image I n-1 The target mask mask n-1 , determine the current frame image I n The deep features R n .

[0051] Specifically, the current frame image I n With the current frame image I n The previous frame image I n-1 The target mask mask n-1 The spliced ​​images are spliced ​​in the dimension, and the visual feature extraction network is used to extract features from the spliced ​​images to obtain the current frame image I nThe deep features R n .

[0052] Further, based on the current frame image I n The deep features R n and the current frame image I n The previous frame image I n-1 Video history features RH n-1 , determine the current frame image I n The encoding feature RN n .

[0053] Specifically, Figure 3 As shown, the current frame image I n The deep features R n As the index vector Q, the previous frame image I n-1 Video history features RH n-1 As the key vector K and value vector V, the three are input into the encoding feature update network to obtain the current frame image I output by the encoding feature update network n The encoding feature RN n , current frame image I n The encoding feature RN n Fusion of the previous frame image I n-1 Video history features.

[0054] Furthermore, the current frame image I n The encoding feature RN n Input to the mask decoding network to obtain the target mask of the current frame image output by the mask decoding network n .

[0055] Specifically, Figure 2 As shown, the current frame image I n The encoding feature RN n , as the mask decoding feature, is input into the mask decoding network of the target mask decoding module (the mask decoding network can adopt structures such as deconvolution and upsampling), and the target mask mask of the current frame image output by the mask decoding network is obtained. n .

[0056] The target mask of the current frame image n Can be used to generate the next frame image I n+1 The target mask mask n+1 , and then the masks {mask1,...,mask n}, and acts on the original video frame image to realize the visualization of the target segmentation result and provide the visualization result to the user.

[0057] Further, based on the current frame image I n The encoding feature RN n and the current frame image I n The previous frame image I n-1 Video history features RH n-1 , the current frame image I can be determined n Video history features RH n .

[0058] Specifically, Figure 3 As shown, the previous frame image I n-1 Video history features RH n-1 As the index vector Q, the current frame image I n The encoding feature RN n As the key vector K and value vector V, the three are input into the historical feature update network to obtain the current frame image I output by the historical feature update network n Video history features RH n , current frame image I n Video history features RH n Fusion of the current frame image I n The encoding feature RN n .

[0059] Furthermore, the next frame image of the current frame image is taken as the current frame image, and the target mask based on the current frame image and the previous frame image of the current frame image is returned to determine the depth feature of the current frame image, until the target mask, depth feature, video history feature and encoding feature of each image in the target video sequence are obtained.

[0060] It can be understood that for a target video sequence of length n (n is determined by the duration and video frame rate selected by the user), its encoding features can be expressed as {RN0, RN1, ..., RN n}, which combines the information of historical target mask, historical frame and current frame, and can be used to predict the target mask of the current frame, while the video history features {RH0,RH1,...,RH n} contains both the target mask information and the video history information of the previous frame. The final video history feature RH n Contains the historical information of the entire video.

[0061] Furthermore, the video history features of the last frame image in the target video sequence are used as the visual features of the target video sequence.

[0062] In this embodiment, assuming that the length of the target video sequence is n, the last frame image I in the target video sequence is n The video history feature is RH n, which is used as the visual feature of the target video sequence.

[0063] The video analysis method provided in the embodiment of the present application provides a user interaction function, where the user selects the target to be analyzed, and uses the SAM model as an interaction bridge to convert the user's interactive operation into the segmentation of a visually specific target, thereby achieving the selection of the target of interest. Based on the user interaction results, video-level understanding of local targets, video-level understanding of global information, image-level understanding of local targets, and image-level understanding of global information can be achieved respectively, and the system architecture has stronger compatibility.

[0064] At the same time, the video analysis method provided in the embodiment of the present application adopts a semi-supervised task framework to extract visual features, which not only integrates the target mask information, but also covers the video history information, so that the video history features can be used to generate a video-level description of the specified target, realizing the video analysis function without increasing the feature dimension, thereby reducing the computational cost.

[0065] In addition, this embodiment extracts the video history features of the last frame image in the target video sequence as the visual features. The visual features contain the main information of the entire video. On the premise of using the single-frame visual features, the information of the entire video can be integrated, thereby improving the efficiency of feature extraction and saving computing resources.

[0066] In some embodiments, determining the target mask, depth features, video history features and coding features of the first frame image includes: determining the user's interaction information; inputting the first frame image and the interaction information into a segmentation model to obtain the target mask of the first frame image output by the segmentation model; splicing the first frame image and the target mask of the first frame image to obtain a spliced ​​image; performing feature extraction on the spliced ​​image based on a visual feature extraction network to obtain the depth features of the first frame image and the video history features of the first frame image; and extracting the coding features of the first frame image.

[0067] Among them, the segmentation model is the SAM model.

[0068] Specifically, the visual understanding system can play the initial video (i.e., the original video), which may contain multiple targets, which may be people, objects, animals, plants, specific areas, etc.; while the user is watching the video, the user can pause the video at any time, and the picture will stay on the image of that moment frame. The user can interact with the system, select a target of interest in the image as the target to be analyzed, and determine the processing time, thereby generating interactive information; the processing time refers to the length of the video that the user wants the system to analyze.

[0069] Furthermore, the visual understanding system responds to the user's interactive information and can determine the target to be analyzed in the initial video and the processing time selected by the user based on the interactive information, and determine the target video sequence based on the target to be analyzed in the initial video and the processing time selected by the user.

[0070] It should be noted that, in this embodiment, when the user selects a target to be analyzed on an image, the frame image is used as the first frame image I0 in the target video sequence, and the first frame image I0 is the first frame image of the target video sequence.

[0071] Furthermore, the user's interaction information is used as a prompt, and the first frame image I0 and the interaction information are input into the SAM model of the user interaction processing module to obtain the target mask mask0 of the first frame image I0 output by the SAM model.

[0072] In some embodiments, after obtaining the target mask, depth features, video history features and encoding features of each image in the target video sequence, it also includes: based on the target mask of each image, performing target segmentation on each image respectively to obtain the target segmentation result of each image; and returning the target segmentation result of each image to the user.

[0073] After obtaining the target mask of each image, target segmentation is performed on each image based on the target mask of each image to obtain the target segmentation result of each image; the target segmentation result of each image is returned to the user to realize the visualization of the target segmentation result, which is convenient for the user to intuitively experience.

[0074] In some embodiments, obtaining a target video sequence and a user's text prompt words includes: obtaining an initial video; responding to user interaction information, determining a target to be analyzed in the initial video and a processing time selected by the user based on the interaction information; determining a target video sequence based on the target to be analyzed in the initial video and a processing time selected by the user; and obtaining the user's text prompt words.

[0075] It should be noted that if the user has not selected a target to be analyzed, the system may not perform segmentation processing operations with user interaction. At this time, the dimension of the mask in the encoding feature update network is set to 0, and the description text output by the system will not be for a specific target in the video, but for the global image in the video; if the user only selects the target to be analyzed but does not determine the processing time, the system only processes the current frame image and outputs a specific target description result for the current frame image; if the user neither selects a target to be analyzed nor determines the processing time, the system outputs a global description text for the current frame image.

[0076] The video analysis method provided in the embodiment of the present application can adjust the system functions in a timely manner according to interactive settings such as the target and duration selected by the user, and can realize the description of pictures / videos and local / global information.

[0077] In some embodiments, visual features and text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model, including: dimensionally adjusting the visual features so that the visual features and the text features have the same feature dimensions; inputting the dimensionally adjusted visual features and text features into the pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0078] Please continue reading Figure 2 The core of the image-text feature fusion module is the pre-trained Large Language Model (LLM), which fuses visual features and text features and inputs them into the large language model. The large language model can perform causal reasoning based on visual features and text features and output a description text of the target to be analyzed.

[0079] See also Figure 4 , Figure 4 It is a flowchart of feature fusion processing provided in an embodiment of the present application.

[0080] Specifically, Figure 4 As shown in the figure, the user's text prompt word prompt is input into the text encoding network to obtain the text feature R generated by the text encoding network. T (Right now Figure 4 The text sequence features in R T The dimension is [textlength, d], where textlength is the length of the text word (token) and d is the dimension of each word.

[0081] In this embodiment, assuming that the length of the target video sequence is n, the last frame image I in the target video sequence is n The video history feature is RH n , which is used as the visual feature of the target video sequence, that is, the visual feature is RH n .

[0082] It can be understood that in order to make the visual feature RH n With text features R T With the same feature dimension, the MLP structure (generally using several layers of linear layers) can be used to adjust the visual feature RH n Dimension, visual features RH after dimension adjustment nThe dimension is [M×M, d], where M = ImageSize (image size) / PatchSize (ViT divided image window size).

[0083] It should be noted that if the video history feature RH is not used n Instead of using the existing method as a visual feature, the depth features {R0, R1, ..., R n} as visual features, the dimension of the visual features will increase to [n × M × M, d], which will increase the system's computational overhead. In addition, for long videos, the dimension of the visual features may exceed the maximum processing token length of the large language model, thus affecting the generation of description text.

[0084] Furthermore, the visual features and text features that have been dimensionally adjusted are concatenated in the column dimension, combined with rotational position encoding, and input into the pre-trained large language model to obtain the descriptive text of the target to be analyzed output by the large language model, thereby realizing the description of the selected target in the video.

[0085] by Figure 2 For example, the user's text prompt is "What is the man doing?", and the target to be analyzed selected by the user is the "man" in the video. Then the description text of the target to be analyzed output by the large language model is "He is squatting and talking on the phone. At the beginning, he is facing the robot on the left. As the call progresses, he gradually moves behind the cabinet."

[0086] The present application also provides a video analysis device. Figure 5 , Figure 5 5 is a schematic diagram of the structure of the video analysis device provided in the embodiment of the present application. In the embodiment of the present application, the video analysis device includes an acquisition module 510, a visual feature extraction module 520, a text feature extraction module 530 and an analysis module 540.

[0087] The acquisition module 510 is used to acquire the target video sequence and the user's text prompt words.

[0088] The target video sequence includes multiple images, each of which contains a target to be analyzed selected by the user.

[0089] The visual feature extraction module 520 is used to extract features from the target video sequence to obtain visual features of the target video sequence.

[0090] The text feature extraction module 530 is used to extract features from the text prompt words to obtain text features of the text prompt words.

[0091] The analysis module 540 is used to input the visual features and text features into the pre-trained large language model to obtain the descriptive text of the target to be analyzed output by the large language model.

[0092] In some embodiments, the visual feature extraction module 520 is used to determine the target mask, depth feature, video history feature and coding feature of the first frame image; the first frame image is the first frame image of the target video sequence; the next frame image of the first frame image is used as the current frame image; based on the target mask of the current frame image and the previous frame image of the current frame image, the depth feature of the current frame image is determined; based on the depth feature of the current frame image and the video history feature of the previous frame image of the current frame image, the coding feature of the current frame image is determined; the coding feature of the current frame image is input into the mask decoding network to obtain the target mask of the current frame image output by the mask decoding network; based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image, the video history feature of the current frame image is determined; the next frame image of the current frame image is used as the current frame image, and the target mask based on the current frame image and the previous frame image of the current frame image is returned to determine the depth feature of the current frame image, until the target mask, depth feature, video history feature and coding feature of each image in the target video sequence are obtained; the video history feature of the last frame image in the target video sequence is used as the visual feature of the target video sequence.

[0093] In some embodiments, the visual feature extraction module 520 is used to determine the user's interaction information; input the first frame image and the interaction information into the segmentation model to obtain the target mask of the first frame image output by the segmentation model; splice the first frame image and the target mask of the first frame image to obtain a spliced ​​image; based on the visual feature extraction network, perform feature extraction on the spliced ​​image to obtain the depth feature of the first frame image and the video history feature of the first frame image; extract the coding feature of the first frame image.

[0094] In some embodiments, the visual feature extraction module 520 is used to perform target segmentation on each image based on the target mask of each image to obtain the target segmentation result of each image; and return the target segmentation result of each image to the user.

[0095] In some embodiments, the acquisition module 510 is used to acquire an initial video; in response to user interaction information, determine the target to be analyzed in the initial video and the processing time selected by the user based on the interaction information; determine the target video sequence based on the target to be analyzed in the initial video and the processing time selected by the user; and acquire the user's text prompt words.

[0096] In some embodiments, the analysis module 540 is used to perform dimension adjustment processing on the visual features so that the visual features and the text features have the same feature dimensions; the dimensionally adjusted visual features and text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0097] The present application also provides an electronic device, Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the video analysis method.

[0098] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0099] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video analysis method provided by the above methods is implemented.

[0100] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video analysis methods provided by the above methods.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video analysis method, characterized in that: include: Acquire a target video sequence and a user's text prompt; the target video sequence includes a plurality of images, each of which includes a target to be analyzed selected by the user; Extracting features from the target video sequence to obtain visual features of the target video sequence; Performing feature extraction on the text prompt word to obtain text features of the text prompt word; The visual features and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

2. The video analysis method according to claim 1, characterized in that: The extracting features of the target video sequence to obtain visual features of the target video sequence includes: Determine a target mask, depth features, video history features, and encoding features of a first frame image; the first frame image is the first frame image of the target video sequence; The next frame image of the first frame image is used as the current frame image; Determining a depth feature of the current frame image based on a target mask of the current frame image and a frame image previous to the current frame image; Determine the encoding feature of the current frame image based on the depth feature of the current frame image and the video history feature of the previous frame image of the current frame image; Inputting the encoded features of the current frame image into a mask decoding network to obtain a target mask of the current frame image output by the mask decoding network; Determine the video history feature of the current frame image based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image; Taking the next frame image of the current frame image as the current frame image, returning the target mask based on the current frame image and the previous frame image of the current frame image, and determining the depth feature of the current frame image, until the target mask, depth feature, video history feature and coding feature of each image in the target video sequence are obtained; The video history features of the last frame of the target video sequence are used as the visual features of the target video sequence.

3. The video analysis method according to claim 2, characterized in that: The step of determining the target mask, depth features, video history features, and encoding features of the first frame image includes: Determining interaction information of the user; Inputting the first frame image and the interaction information into a segmentation model to obtain a target mask of the first frame image output by the segmentation model; Performing splicing processing on the first frame image and the target mask of the first frame image to obtain a spliced ​​image; Based on a visual feature extraction network, feature extraction processing is performed on the spliced ​​image to obtain a depth feature of the first frame image and a video history feature of the first frame image; Extract the encoding features of the first frame image.

4. The video analysis method according to claim 2, characterized in that: After obtaining the target mask, depth feature, video history feature and coding feature of each image in the target video sequence, the method further includes: Based on the target mask of each of the images, perform target segmentation on each of the images to obtain a target segmentation result for each of the images; The target segmentation result of each of the images is returned to the user.

5. The video analysis method according to claim 1, characterized in that: The step of obtaining the target video sequence and the user's text prompt words includes: Get the initial video; In response to the interaction information of the user, determining the target to be analyzed in the initial video and the processing duration selected by the user based on the interaction information; Determining the target video sequence based on the target to be analyzed in the initial video and the processing duration selected by the user; Obtain the text prompt word of the user.

6. The video analysis method according to claim 1, characterized in that: The step of inputting the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model includes: Performing dimension adjustment processing on the visual feature so that the visual feature and the text feature have the same feature dimension; The visual features that have been dimensionally adjusted and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

7. A video analysis device, characterized in that: include: An acquisition module, used to acquire a target video sequence and a user's text prompt word; the target video sequence includes a plurality of images, each of which includes a target to be analyzed selected by the user; A visual feature extraction module, used to extract features from the target video sequence to obtain visual features of the target video sequence; A text feature extraction module, used to extract features from the text prompt words to obtain text features of the text prompt words; The analysis module is used to input the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the video analysis method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video analysis method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video analysis method according to any one of claims 1 to 6 is implemented.

Images