Intelligent picture and video scene splitting method
By using Transformer structure and learnable Query for segmentation in the picture and video scene storyboard task, the problems of low accuracy and unstable classification in the existing technology are solved, and efficient picture and video scene storyboard is achieved.
Patent Information
- Application Number
- CN202411128491.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art has problems of low accuracy, unstable classification and poor clustering effects in storyboard tasks for picture and video scenes, especially when dealing with complex and similar scenarios.
The Transformer structure is used to learn image feature representation, and the learnable Query is used to segment the image feature sequence, thereby realizing storyboarding between pictures and video scenes. This solution injects image features into Query through Transformer Decoder, so that the model can adapt to different picture sets to predict appropriate storyboard results.
This solution overcomes the shortcomings of the traditional two-stage solution, realizes end-to-end storyboard effect, can better handle complex scenarios, and improves the accuracy and stability of storyboards.
Smart Images
Figure CN119516420A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of storyboarding, and in particular to a storyboarding method for pictures and video scenes. Background Art
[0002] The purpose of photo storyboarding is to divide a group of photos or video sequences into several groups according to the scenes in which they were shot. For video sequences, this is generally called storyboarding. Since the shooting conditions and scenes of photos vary greatly from one another and there are many categories, this task is very challenging. Currently, the solutions to this task mainly include classification-based methods and two-stage methods based on feature extraction and clustering.
[0003] The classification-based method predefines possible shooting scenes of the pictures, such as airports, museums, parks, etc., classifies and labels the pictures, and obtains labeled data through a large amount of data collection and manual annotation. For example, the Google Place365 project divides pictures into 365 categories according to the scene, and collects a total of about 62 million pictures, with an average of 40,000 pictures per scene. The project also does a very detailed classification of photos, including airport runways, aircraft cabins, archives, art schools, etc. Even with such a detailed classification, due to the complexity of real pictures, even the same scene has very large changes. On the other hand, outdoor scenes have many similarities. Therefore, it is difficult for the model to learn accurate classification using the classification method, and the accuracy is low. On the other hand, because it is a single-image classification method, it is easy for pictures of the same scene to be unable to stably divide the pictures due to unstable classification.
[0004] Another idea is to extract features from images and then perform clustering. This is a two-stage solution. The strength of the features and the effectiveness of the clustering method directly affect the final effect of the picture storyboard. Picture features are divided into traditional picture features and deep features. Traditional picture features mainly include some hash algorithms, such as average hashing, perceptual hashing, differential hashing, wavelet hashing, color hashing, and hashing algorithms that combine the above hash algorithms with cropping robustness. These hash algorithms can withstand small disturbances caused by certain picture disturbances, but are completely incapable of handling scenes with large differences in photo storyboard tasks. Deep picture features are a relatively effective solution in the current industry.
[0005] Learning the deep features of images can be divided into implicit learning and explicit learning. Implicit learning means that when training the image feature encoder, no loss design and supervision is performed for downstream tasks. For photo storyboarding scenarios, we can use image classification to implicitly learn features. Or use the CLIP scheme to learn the feature encoder of the image through multimodal self-supervision. However, the implicit scheme has the problem of misalignment with the learning target and the storyboard, and it is impossible to learn features suitable for storyboards, and it is difficult to achieve satisfactory results in the second stage clustering.
[0006] Explicit learning generally uses Metric Learning to train the feature encoder of an image. And the learning data is no longer classified data, but data labeled according to the storyboard standard. Metric Learning is already a relatively mature technical field in the industry. According to the type of loss function, it can be divided into comparison-based loss functions, such as ConstrasticLoss, Triplet Loss, Cross-batch Memory Loss, and Lifted Structure Loss; there are also losses based on Softmax functions, such as sphere loss, CosFace, ArcFace, and Circle Loss. These losses, supplemented by a large amount of data, can generally learn relatively strong image representations for subsequent characters. It is already quite mature for solutions such as face recognition.
[0007] The effectiveness of the two-stage solution also depends largely on the choice of clustering method. In the case of photo storyboards, due to the number of storyboards, only clustering methods that automatically determine the number of clusters can be selected, such as density-based clustering methods such as the DBSCAN series of methods and hierarchical clustering methods. These methods have their own problems and cannot solve the task of photo storyboards well. Summary of the invention
[0008] In view of this, the purpose of the present invention is to propose an intelligent method for storyboarding of pictures and video scenes. Based on the above research and summary, this patent proposes a completely different solution from the above solution. This solution is a single-stage solution, but it is not based on classification, but regards the problem of storyboarding as a segmentation problem. Inspired by the DETR and MaskFormer solutions, this patent uses the Transformer structure to learn the feature representation of the picture, and uses the learnable Query to segment the picture feature sequence, so as to achieve the effect of photo storyboarding. This solution overcomes the disadvantage that the two-stage solution requires clustering, and solves the problem of photo storyboarding well. The present invention can use the Transformer Decoder to inject the current picture features into the Query, so that the solution can be adaptively adjusted according to different atlases and predict appropriate storyboarding results.
[0009] According to one aspect of the present invention, there is provided an intelligent method for storyboarding of pictures and video scenes, the method being based on a Transformer network structure, the Transformer network structure comprising at least one Transformer Encoder and at least one Transformer Decoder;
[0010] Obtain a sequence of images to be processed, and set a query embedding vector corresponding to the sequence of images to be processed;
[0011] Preprocessing the image sequence to be processed to obtain a first sequence feature;
[0012] Encode the sequence feature using Transformer Encoder to obtain a second sequence feature;
[0013] Using Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature;
[0014] Aggregate the second sequence features with the third sequence features to obtain the split-mirror features;
[0015] Based on the storyboard feature, a storyboard result is obtained.
[0016] Among the above technical solutions, the current mainstream solution in the industry is to learn the feature vectors of the picture frame and then perform clustering by measuring the distance between the vectors. This two-stage approach does not reflect the learning goal of the model in training: 1) It does not reflect that the learning goal of the model is clustering; 2) The single distance measurement (generally cosine distance cannot reflect the correlation between images); 3) Due to reasons 1 and 2, the threshold hyperparameter of the distance in the clustering process is difficult to determine, or the threshold cannot be adaptively adjusted for different scenes, resulting in poor storyboarding effect. This solution is different from the existing picture storyboarding solutions in the industry. Most of the existing solutions are based on classification and clustering. These methods have poor effects and cannot be done end-to-end. This solution is an end-to-end solution that can divide the picture into multiple shooting scenes at one time. This solution uses a combination of single-image Image Encoder and Transformer to overcome the shortcomings of the single-image solution that cannot obtain a global view when learning representation and the features are not comprehensive enough. The current picture features are injected into the query using Transformer Decoder, so that the solution can be adaptively adjusted according to different atlases and predict appropriate storyboarding results, which is not available in other solutions.
[0017] In some embodiments, preprocessing the image sequence to be processed to obtain a first sequence feature includes:
[0018] Normalizing each image in the image sequence to be processed, and
[0019] A feature extraction operation is performed on the normalized image sequence to be processed to obtain a feature sequence of length L, where L is the number of images in the image sequence to be processed.
[0020] In the above technical solution, Image Encoder quickly reduces the dimensionality of high-dimensional images. On the one hand, it greatly reduces the computational complexity of the subsequent Transformer module. On the other hand, Image Encoder can effectively extract the semantic information of the image, which is helpful for the reasoning of subsequent modules.
[0021] In some embodiments, the image sequence information to be processed is injected into the query embedding vector using a Transformer Decoder to obtain a third sequence feature, and the calculation expression is as follows:
[0022] S,Q=TransformerDecoder(A)
[0023] Among them, the dimension of A is N x D, where N represents the number of queries and D represents the dimension of the query. The output S is the score of the corresponding query, and Q is the third sequence feature.
[0024] In the above technical solution, the storyboard query is effectively interacted with the information of the current image sequence or video sequence through the Transformer Decoder, so that the storyboard query can be expressed more precisely, which is conducive to the model outputting storyboard results for different images.
[0025] In some embodiments, the second sequence feature and the third sequence feature are aggregated to obtain the storyboard feature, including:
[0026] The inner product of the second sequence feature and the third sequence feature is used to calculate the mirror mask of each query. The calculation is as follows
[0027] M=sigmoid(E*Q)
[0028] Where * represents the inner product, E represents the second sequence characteristic, and Q represents the third sequence characteristic.
[0029] In the above technical solution, the storyboard problem is transformed into an instance segmentation problem by segmentation, which innovatively turns the storyboard problem into an end-to-end one, avoiding the shortcomings of multi-stage solution in adjusting parameters, feature training and storyboard clustering cannot share information, and cannot achieve global optimization.
[0030] According to another aspect of the present invention, a training method for a Transformer network is provided. The method is based on a Transformer network structure, wherein the Transformer network structure includes at least one Transformer Encoder and at least one Transformer Decoder.
[0031] Obtain a sample image sequence, and set a query embedding vector corresponding to the sample image sequence and a storyboard label corresponding to the sample image sequence;
[0032] Preprocessing the sample image sequence to obtain first sequence features and segmentation confidence;
[0033] Encode the sequence feature using Transformer Encoder to obtain a second sequence feature;
[0034] Using Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature;
[0035] Aggregate the second sequence features with the third sequence features to obtain the split-mirror features;
[0036] Based on the storyboard feature, the storyboard prediction label is obtained.
[0037] Calculate the storyboard loss value based on the storyboard label and the storyboard prediction label;
[0038] Calculate the confidence loss value based on the sample image sequence and the confidence of the storyboard;
[0039] The Transformer network is trained by performing gradient descent based on the segmentation loss value and the confidence loss value.
[0040] Among the above technical solutions, the Transformer architecture has been proven to be a powerful model architecture. Its sequence processing characteristics are naturally suitable for processing image sequences or video sequences. The Transformer architecture can allow information to flow between images and summarize the differences and connections between images. Through joint training, the storyboard standard is integrated into the model learning to achieve end-to-end learning of storyboards, avoiding the situation where the two-stage model hyperparameters cannot adapt to different types of images.
[0041] According to another aspect of the present invention, a Transformer network is provided, characterized in that it includes:
[0042] Obtain a sequence of images to be processed, and set a query embedding vector corresponding to the sequence of images to be processed;
[0043] A first feature module is configured to preprocess the image sequence to be processed to obtain a first sequence feature;
[0044] An encoder is configured to encode the sequence feature using a Transformer Encoder to obtain a second sequence feature;
[0045] A decoder is configured to use a Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature;
[0046] An aggregation layer, configured to aggregate the second sequence feature with the third sequence feature to obtain a storyboard feature;
[0047] The discriminator is configured to obtain a storyboard result based on the storyboard feature.
[0048] In the above technical solution, the Transformer architecture can be used to connect the correlations between images, and the correlations between images can be discovered even if the image viewing angle is larger.
[0049] According to another aspect of the present invention, there is provided an intelligent image and video scene storyboard device, based on the above method; comprising:
[0050] An acquisition module is used to acquire a sequence of images to be processed, and to set a query embedding vector corresponding to the sequence of images to be processed;
[0051] A first feature module, used for preprocessing the image sequence to be processed to obtain a first sequence feature;
[0052] A second feature module, used to encode the sequence feature using a Transformer Encoder to obtain a second sequence feature;
[0053] A third feature module, used to inject the image sequence information to be processed into the query embedding vector using a Transformer Decoder to obtain a third sequence feature;
[0054] An aggregation module, used for aggregating the second sequence features with the third sequence features to obtain the storyboard features;
[0055] The discrimination module is used to obtain the storyboard result based on the storyboard feature.
[0056] In the above technical scheme, in order to better use the above method, the present application proposes an image segmentation device based on a MaskFormer network, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.
[0057] According to another aspect of the present invention, there is provided an intelligent picture and video scene storyboard device, comprising:
[0058] at least one processor and a memory communicatively coupled to the at least one processor;
[0059] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.
[0060] In the above technical solution, in order to better run and process the method, the above method is stored in a memory, and the stored method is executed by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.
[0061] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, wherein the computer program implements the above method when executed by a processor.
[0062] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0064] Figure 1 It is a flow chart of an embodiment of an intelligent method for storyboarding pictures and video scenes of the present invention;
[0065] Figure 2 It is a schematic diagram of a framework of an embodiment of a Transformer network of the present invention;
[0066] Figure 3 It is a flow chart of an embodiment of a training method of a Transformer network of the present invention;
[0067] Figure 4 It is a schematic diagram of a framework of an embodiment of a training method for a Transformer network of the present invention;
[0068] Figure 5 It is a structural schematic diagram of an embodiment of an intelligent image and video scene storyboard device of the present invention. DETAILED DESCRIPTION
[0069] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.
[0070] The present invention provides an intelligent picture and video scene storyboarding method, which can use TransformerDecoder to inject current picture features into Query, so that the solution can be adaptively adjusted according to different atlases and predict appropriate storyboarding results.
[0071] Embodiment 1
[0072] See also Figure 1, an intelligent method for storyboarding of pictures and video scenes, the method is based on a Transformer network structure, the Transformer network structure includes at least one Transformer Encoder and at least one Transformer Decoder;
[0073] S11, obtaining a sequence of images to be processed, and setting a query embedding vector corresponding to the sequence of images to be processed;
[0074] S12, preprocessing the image sequence to be processed to obtain a first sequence feature;
[0075] In this embodiment, S12, preprocessing the image sequence to be processed to obtain a first sequence feature, specifically includes:
[0076] S121, normalizing each image in the image sequence to be processed, and,
[0077] S122, performing feature extraction operation on the normalized image sequence to be processed to obtain a feature sequence of length L, where L is the number of images in the image sequence to be processed.
[0078] S13, using Transformer Encoder to encode the sequence feature to obtain a second sequence feature;
[0079] S14, using Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature;
[0080] In this embodiment, S14, using Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain the third sequence feature; the calculation expression is as follows:
[0081] S,Q=TransformerDecoder(A)
[0082] Among them, the dimension of A is N x D, where N represents the number of queries and D represents the dimension of the query. The output S is the score of the corresponding query, and Q is the third sequence feature.
[0083] S15, aggregating the second sequence features and the third sequence features to obtain the storyboard features;
[0084] In this embodiment, S15, aggregating the second sequence features and the third sequence features to obtain the storyboard features, includes:
[0085] The inner product of the second sequence feature and the third sequence feature is used to calculate the mirror mask of each query. The calculation is as follows
[0086] M=sigmoid(E*Q)
[0087] Among them, * represents the inner product, E represents the second sequence characteristics, and Q represents the third sequence characteristics.
[0088] S16. Obtaining a storyboard result based on the storyboard feature.
[0089] Among the above technical solutions, the current mainstream solution in the industry is to learn the feature vectors of the picture frame and then perform clustering by measuring the distance between the vectors. This two-stage approach does not reflect the learning goal of the model in training: 1) It does not reflect that the learning goal of the model is clustering; 2) The single distance measurement (generally cosine distance cannot reflect the correlation between images); 3) Due to reasons 1 and 2, the threshold hyperparameter of the distance in the clustering process is difficult to determine, or the threshold cannot be adaptively adjusted for different scenes, resulting in poor storyboarding effect. This solution is different from the existing picture storyboarding solutions in the industry. Most of the existing solutions are based on classification and clustering. These methods have poor effects and cannot be done end-to-end. This solution is an end-to-end solution that can divide the picture into multiple shooting scenes at one time. This solution uses a combination of single-image Image Encoder and Transformer to overcome the shortcomings of the single-image solution that cannot obtain a global view when learning representation and the features are not comprehensive enough. The current picture features are injected into the query using Transformer Decoder, so that the solution can be adaptively adjusted according to different atlases and predict appropriate storyboarding results, which is not available in other solutions.
[0090] Next, the specific working process of the present invention will be described based on the principle, which is as follows:
[0091] See also Figure 2 The scheme is roughly divided into two parts. The ae part is the encoder part of the image, and its main function is to learn the representation of the picture. The fg part is the query learning part, which learns the query vectors of different storyboards. Finally, each query vector outputs a storyboard score and a storyboard mask. If the mask is higher than the threshold, it means that the corresponding image belongs to the current storyboard, and if it is lower than the threshold, it means that it does not belong to the current storyboard. The following is a detailed description of each module.
[0092] a) Input images are uniformly scaled to 256x256 before entering Image Encoder, and the value range is normalized to [-1,1].
[0093] b) is a single image encoder. This patent adopts the architecture of resnet101. Other encoder architectures can be used. Image Encoder uniformly encodes the image into a 512 feature vector.
[0094] c) After the Image Encoder encodes the image into a feature vector of length 512, the feature vectors of different images are concatenated to form a feature sequence of length L, where L represents the number of images. N varies with the size of the image set. Due to the limitation of video memory during training, we set L to 256.
[0095] d) Transformer Encoder, taking advantage of Transformer’s ability to process sequence features, transforms the feature sequence of (c) to enhance its representation information, making it more suitable for storyboard segmentation.
[0096] The hyperparameter settings of Transformer Encoder are shown in the following table:
[0097] Table 1 Transformer Encoder hyperparameter setting table
[0098]
[0099]
[0100] e) The dimension of the image features after the Transformer Encoder output transformation remains unchanged, still 512 dimensions, denoted as E.
[0101] f) The number of learnable query embedding vectors is a hyperparameter, representing the number of queries and the maximum number of supported storyboards, which can be set according to the actual application scenario. This patent sets the number to 100 and the dimension length to 512.
[0102] g) Transformer Decoder, in order to make the query embedding change accordingly according to the input image set, Transformer Decoder uses cross attention to inject the information of the current image set into the query embedding. Its calculation can be expressed as follows
[0103] S,Q=TransformerDecoder(A)
[0104] The dimension of A is N x D, where N represents the number of queries and D represents the dimension of the query. In this patent, N = 100 and D = 512. The output S is the score of the corresponding query, which is generally normalized to [0, 1] using the Sigmoid function. The dimension of S is N x 1, and the dimension of Q is N x D.
[0105] h) The mask of each query is calculated by the inner product of image features E and Q, which is calculated as follows
[0106] M=sigmoid(E*Q)
[0107] Where * represents the inner product, and the dimension of M is L x N. That is, each Query corresponds to a string of Masks, indicating the same storyboard photos.
[0108] like Figure 2 As shown in h, each query obtains a segmentation mask, which is generally binarized using a threshold, which is set to 0.5 in this patent. Thus, 0 means that the current image does not belong to the current storyboard, and 1 means that it belongs to the current storyboard.
[0109] Embodiment 2
[0110] See also Figure 3 , Figure 4 , a training method for a Transformer network, the method is based on a Transformer network structure, the Transformer network structure includes at least one Transformer Encoder, and at least one Transformer Decoder;
[0111] S1, obtaining a sample image sequence, and setting a query embedding vector corresponding to the sample image sequence and a storyboard label corresponding to the sample image sequence;
[0112] S2, preprocessing the sample image sequence to obtain a first sequence feature and a split-mirror confidence;
[0113] S3, using Transformer Encoder to encode the sequence feature to obtain a second sequence feature;
[0114] S4, using Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature;
[0115] S5, aggregating the second sequence features and the third sequence features to obtain the storyboard features;
[0116] S6. Obtaining a storyboard prediction label based on the storyboard feature;
[0117] S7, calculating a storyboard loss value based on the storyboard label and the storyboard prediction label;
[0118] S8, calculating the confidence loss value based on the sample image sequence and the confidence of the storyboard;
[0119] S9. Perform gradient descent based on the split-mirror loss value and the confidence loss value to train the Transformer network.
[0120] Among the above technical solutions, the Transformer architecture has been proven to be a powerful model architecture. Its sequence processing characteristics are naturally suitable for processing image sequences or video sequences. The Transformer architecture can allow information to flow between images and summarize the differences and connections between images. Through joint training, the storyboard standard is integrated into the model learning to achieve end-to-end learning of storyboards, avoiding the situation where the two-stage model hyperparameters cannot adapt to different types of images.
[0121] Embodiment 3
[0122] See also Figure 2 , a Transformer network, characterized by comprising:
[0123] Obtain a sequence of images to be processed, and set a query embedding vector corresponding to the sequence of images to be processed;
[0124] A first feature module is configured to preprocess the image sequence to be processed to obtain a first sequence feature;
[0125] An encoder is configured to encode the sequence feature using a Transformer Encoder to obtain a second sequence feature;
[0126] A decoder is configured to use a Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature;
[0127] An aggregation layer, configured to aggregate the second sequence feature with the third sequence feature to obtain a storyboard feature;
[0128] The discriminator is configured to obtain a storyboard result based on the storyboard feature.
[0129] In the above technical solution, the Transformer architecture can be used to connect the correlations between images, and the correlations between images can be discovered even if the image viewing angle is larger.
[0130] Embodiment 4
[0131] See also Figure 5 , an intelligent image and video scene storyboard device, based on the above method; comprising:
[0132] An acquisition module is used to acquire a sequence of images to be processed, and to set a query embedding vector corresponding to the sequence of images to be processed;
[0133] A first feature module, used for preprocessing the image sequence to be processed to obtain a first sequence feature;
[0134] A second feature module, used to encode the sequence feature using a Transformer Encoder to obtain a second sequence feature;
[0135] A third feature module, used to inject the image sequence information to be processed into the query embedding vector using a Transformer Decoder to obtain a third sequence feature;
[0136] An aggregation module, used for aggregating the second sequence features with the third sequence features to obtain the storyboard features;
[0137] The discrimination module is used to obtain the storyboard result based on the storyboard feature.
[0138] In the above technical scheme, in order to better use the above method, the present application proposes an image segmentation device based on a MaskFormer network, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.
[0139] Embodiment 5
[0140] An intelligent picture and video scene storyboard device, comprising:
[0141] at least one processor and a memory communicatively coupled to the at least one processor;
[0142] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in embodiment one or embodiment two.
[0143] In the above technical solution, in order to better run and process the method described in Embodiment 1 or Embodiment 2, the above method is stored in a memory, and a processor is used to execute the stored method. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.
[0144] Embodiment 6
[0145] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method described in one or two embodiments is implemented.
[0146] In the above technical solution, in order to better run and use the method described in Embodiment 1 or Embodiment 2, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.
[0147] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. An intelligent method for storyboarding pictures and video scenes, characterized in that: The method is based on a Transformer network structure, which includes at least one Transformer Encoder and at least one Transformer Decoder; Obtain a sequence of images to be processed, and set a query embedding vector corresponding to the sequence of images to be processed; Preprocessing the image sequence to be processed to obtain a first sequence feature; Encode the sequence feature using Transformer Encoder to obtain a second sequence feature; Using Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature; Aggregate the second sequence features with the third sequence features to obtain the split-mirror features; Based on the storyboard feature, a storyboard result is obtained.
2. The intelligent method for storyboarding pictures and video scenes according to claim 1, characterized in that: Preprocessing the image sequence to be processed to obtain a first sequence feature includes: Normalizing each image in the image sequence to be processed, and A feature extraction operation is performed on the normalized image sequence to be processed to obtain a feature sequence of length L, where L is the number of images in the image sequence to be processed.
3. The intelligent method for storyboarding pictures and video scenes as claimed in claim 1, characterized in that: Transformer Decoder is used to inject the image sequence information to be processed into the query embedding vector to obtain the third sequence feature, which is calculated and expressed as follows: S,Q=TransformerDecoder(A) Among them, the dimension of A is N x D, where N represents the number of queries and D represents the dimension of the query. The output S is the score of the corresponding query, and Q is the third sequence feature.
4. The intelligent method for storyboarding pictures and video scenes according to claim 1, characterized in that: Aggregate the second sequence features with the third sequence features to obtain the storyboard features, including: The inner product of the second sequence feature and the third sequence feature is used to calculate the mirror mask of each query. The calculation is as follows M=sigmoid(E*Q) Where * represents the inner product, E represents the second sequence characteristic, and Q represents the third sequence characteristic.
5. A training method for a Transformer network, characterized in that: The method is based on a Transformer network structure, which includes at least one Transformer Encoder and at least one Transformer Decoder; Obtain a sample image sequence, and set a query embedding vector corresponding to the sample image sequence and a storyboard label corresponding to the sample image sequence; Preprocessing the sample image sequence to obtain first sequence features and segmentation confidence; Encode the sequence feature using Transformer Encoder to obtain a second sequence feature; Using Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature; Aggregate the second sequence features with the third sequence features to obtain the split-mirror features; Based on the storyboard feature, the storyboard prediction label is obtained. Calculate the storyboard loss value based on the storyboard label and the storyboard prediction label; Calculate the confidence loss value based on the sample image sequence and the confidence of the storyboard; The Transformer network is trained by performing gradient descent based on the segmentation loss value and the confidence loss value.
6. A Transformer network, characterized in that: include: Obtain a sequence of images to be processed, and set a query embedding vector corresponding to the sequence of images to be processed; A first feature module is configured to preprocess the image sequence to be processed to obtain a first sequence feature; An encoder is configured to encode the sequence feature using a Transformer Encoder to obtain a second sequence feature; A decoder is configured to use a Transformer Decoder to inject the image sequence information to be processed into the query embedding vector to obtain a third sequence feature; An aggregation layer, configured to aggregate the second sequence feature with the third sequence feature to obtain a storyboard feature; The discriminator is configured to obtain a storyboard result based on the storyboard feature.
7. An intelligent image and video scene storyboard device, characterized in that: A method according to any one of claims 1 to 6; comprising: An acquisition module, used for acquiring a sequence of images to be processed, and setting a query embedding vector corresponding to the sequence of images to be processed; A first feature module, used for preprocessing the image sequence to be processed to obtain a first sequence feature; A second feature module, used to encode the sequence feature using a Transformer Encoder to obtain a second sequence feature; A third feature module, used to inject the image sequence information to be processed into the query embedding vector using a Transformer Decoder to obtain a third sequence feature; An aggregation module, used for aggregating the second sequence features with the third sequence features to obtain the storyboard features; The discrimination module is used to obtain the storyboard result based on the storyboard feature.
8. An intelligent picture and video scene storyboard device, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as claimed in any one of claims 1 to 5.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.