A video portrait-landscape screen conversion method and device, storage medium and electronic equipment

Abstract

The application provides a video horizontal screen portrait screen conversion method and device, a storage medium and electronic equipment, applied to the technical field of video processing, obtaining image features of a video frame sequence and target object information of each video frame; cross-modal attention fusion is performed based on the target object information and the image features to obtain multi-modal fusion features; a time sequence dependency relationship of the multi-modal fusion features is established along a time dimension, and a corresponding cutting center point of each video frame is obtained according to the time sequence dependency relationship; and the corresponding video frame is cut horizontally to portrait screen based on the cutting center point to obtain a portrait screen video sequence. The application performs cross-modal attention fusion on the target object information and the image features to focus on the area where the target object is located, and establishes a time sequence dependency relationship of the multi-modal fusion features along the time dimension, so that a smooth cutting center point is adaptively obtained, and the video horizontal screen portrait screen conversion is realized without relying on complex logical rules.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method, apparatus, storage medium, and electronic device for video landscape / portrait screen conversion. Background Technology

[0002] With the rapid development of short videos, vertical videos have become the mainstream consumption format, and horizontal videos need to be converted to vertical videos to adapt to mobile phone viewing.

[0003] However, existing methods for switching from horizontal to vertical orientation can be mainly divided into two categories: one is based on logical judgment using manual rules, which relies on human body and face detection results for cropping, but the rules are difficult to reuse and have poor adaptability to complex scenes; the other is based on deep learning methods using convolutional neural networks, which have a certain generalization ability, but do not consider the temporal relationship between video frames and fail to effectively utilize multimodal information such as audio, which can easily lead to problems such as image shaking and loss of subject tracking when switching between shots or speakers.

[0004] Therefore, how to achieve smooth and adaptive landscape-to-portrait cropping without relying on complex logical rules has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a method, apparatus, storage medium, and electronic device for video landscape / portrait screen conversion that overcomes or at least partially solves the above problems. The technical solution is as follows:

[0006] A method for converting video between portrait and landscape orientations includes:

[0007] Obtain image features of the video frame sequence and target object information for each video frame;

[0008] Cross-modal attention fusion is performed based on the target object information and the image features to obtain multimodal fusion features;

[0009] Establish the temporal dependency relationship of the multimodal fusion features along the time dimension, and obtain the cropping center point corresponding to each video frame based on the temporal dependency relationship;

[0010] Based on the cropping center point, the corresponding video frames are cropped from landscape to portrait mode to obtain a portrait video sequence.

[0011] Optionally, obtaining the image features of the video frame sequence and the target object information of each video frame includes:

[0012] Extract the image features of each video frame in the video frame sequence;

[0013] Obtain the face detection bounding box, face identifier, and speech confidence score for each video frame, and combine the bounding box information, face identifier, and speech confidence score into target object information.

[0014] Optionally, the step of performing cross-modal attention fusion based on the target object information and the image features to obtain multimodal fusion features includes:

[0015] The target object information is mapped into a query vector of the same dimension as the image features through a projection network;

[0016] The image features are used as key vectors and value vectors to determine the similarity between the query vector and the key vector. After normalizing the similarity, attention weights are obtained.

[0017] The attention weights are summed with the value vector to obtain the multimodal fusion features.

[0018] Optionally, establishing the temporal dependency relationship of the multimodal fusion features along the time dimension, and obtaining the cropping center point corresponding to each video frame based on the temporal dependency relationship, includes:

[0019] The multimodal fusion features are sorted along the time dimension so that the time dimension becomes the main dimension for time series modeling;

[0020] Attention is calculated on the sorted multimodal fusion features along the time dimension to establish the temporal dependency relationship between each video frame;

[0021] Regression prediction is performed based on the temporal dependency to obtain the cropping center point corresponding to each video frame.

[0022] Optionally, the method further includes:

[0023] Obtain the labeled cropping center points of the training video frame sequence;

[0024] Regression prediction is performed on the calculation results of the temporal dependency to obtain the predicted cropping center point and adaptive mask for each video frame;

[0025] Determine the basic positional loss between the predicted clipping center point and the labeled clipping center point;

[0026] The predicted cropping center points of each video frame are arranged in chronological order to form a sequence of predicted center points;

[0027] Obtain the acceleration sequence of the predicted center point sequence;

[0028] Multiply the adaptive mask by the corresponding element of the acceleration sequence to obtain the acceleration smoothing loss;

[0029] The sum of the basic position loss and the acceleration smoothing loss is used as the total loss to adjust the trainable parameters in the cross-modal attention fusion, temporal dependency establishment, and regression prediction processes.

[0030] Optionally, obtaining the acceleration sequence of the predicted center point sequence includes:

[0031] Perform a first-order difference operation on the predicted center point sequence to obtain the velocity sequence;

[0032] The velocity sequence is subjected to a first-order difference operation again to obtain an acceleration sequence, wherein the length of the acceleration sequence is the same as the length of the adaptive mask.

[0033] Optionally, the adaptive mask is automatically adjusted during training to reduce the weight of the acceleration sequence corresponding to the shot switching frame or speaker switching frame in the acceleration smoothing loss.

[0034] A video landscape / portrait screen conversion device includes: a video feature information acquisition unit, a multimodal fusion feature acquisition unit, a cropping center point acquisition unit, and a video cropping unit;

[0035] The video feature information acquisition unit is used to obtain the image features of the video frame sequence and the target object information of each video frame;

[0036] The multimodal fusion feature acquisition unit is used to perform cross-modal attention fusion based on the target object information and the image features to obtain multimodal fusion features;

[0037] The cropping center point acquisition unit is used to establish the temporal dependency relationship of the multimodal fusion features along the time dimension, and obtain the cropping center point corresponding to each video frame based on the temporal dependency relationship.

[0038] The video cropping unit is used to crop the corresponding video frames from horizontal to vertical screen based on the cropping center point to obtain a vertical video sequence.

[0039] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the video landscape / portrait switching method.

[0040] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the video landscape / portrait switching method.

[0041] By employing the above technical solutions, this invention provides a method, apparatus, storage medium, and electronic device for video landscape / portrait conversion. It obtains image features of a video frame sequence and target object information for each video frame; performs cross-modal attention fusion based on the target object information and image features to obtain multimodal fusion features; establishes temporal dependencies of the multimodal fusion features along the time dimension; and obtains the cropping center point corresponding to each video frame based on the temporal dependencies. Based on the cropping center points, it performs landscape-to-portrait cropping on the corresponding video frames to obtain a portrait video sequence. This invention achieves video landscape / portrait conversion without relying on complex logical rules by performing cross-modal attention fusion of target object information and image features to focus on the region where the target object is located and establishing temporal dependencies of the multimodal fusion features along the time dimension.

[0042] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0044] Figure 1 The diagram illustrates a flowchart of one embodiment of the video landscape / portrait screen conversion method provided by this invention.

[0045] Figure 2 This diagram illustrates a flowchart of one implementation of the model parameter training process provided in this invention.

[0046] Figure 3 A schematic diagram of the video landscape / portrait screen conversion device provided in an embodiment of the present invention is shown;

[0047] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0048] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0049] With the rapid development of short video platforms, vertical videos have gradually become one of the mainstream forms of user content consumption. Traditional TV programs and online videos are mostly in 16:9 or 4:3 landscape format, while mobile terminals such as smartphones typically display in portrait ratios such as 9:16 or 9:18. To improve the user experience when watching in portrait mode, it is necessary to convert landscape videos to portrait videos; this process is usually called video landscape-to-portrait conversion.

[0050] Currently available landscape / portrait screen conversion methods mainly fall into two categories: The first type uses complex logical rules to filter and jointly determine the results of upstream human detection, face detection, and saliency detection to obtain the cropping area. However, these methods suffer from difficulty in reusing logical rules across different scenarios, and the rules struggle to fully consider the information from all video frames. This results in poor adaptability in complex scenarios and excessive reliance on engineering experience. The second type employs deep learning technology, primarily relying on convolutional neural networks to further extract features from candidate subject information obtained from upstream tasks, ultimately regressing the specific subject to be retained. While this type offers improved adaptability compared to the first, it still has significant shortcomings: it doesn't consider the temporal sequence of the video, fails to establish connections between video frames, and has limited utilization of modal information. For example, it doesn't utilize audio information to guide the selection of the cropped subject, leading to issues like image shakiness and subject loss during speaker or shot transitions.

[0051] Based on this, this invention provides a method for video landscape / portrait conversion. First, image features of the video frame sequence and target object information for each frame are obtained. The target object information and image features are then fused across modal attention to obtain multimodal fusion features. Second, a temporal dependency relationship of the multimodal fusion features is established along the time dimension. Based on this temporal dependency relationship, the cropping center point of each frame is obtained, and the landscape video is cropped to portrait mode based on the cropping center point. During training, regression prediction is performed on the calculation results of the temporal dependency relationship, and the predicted cropping center point and adaptive mask are output simultaneously. The basic position loss and acceleration smoothing loss are calculated, and the sum of these two losses is used to adjust the trainable parameters in the cross-modal attention fusion, temporal dependency relationship establishment, and regression prediction processes. The adaptive mask automatically reduces the smoothing loss weight when switching shots or speakers. Therefore, this invention does not rely on complex rules and can fully utilize target object information and temporal information to adaptively achieve smooth landscape / portrait conversion. This helps solve problems such as difficulty in rule reuse, lack of temporal modeling, and image jitter in existing methods, improving the viewing experience of portrait videos and the level of video processing automation.

[0052] like Figure 1The diagram shows a flowchart of one embodiment of the video landscape / portrait screen conversion method provided by this invention. The method may include:

[0053] S100: Obtain the image features of the video frame sequence and the target object information of each video frame.

[0054] Among them, the video frame sequence refers to a collection of consecutive video frames arranged in chronological order, with each frame being an image, which constitutes the landscape video data to be processed.

[0055] Image features refer to high-dimensional vector representations extracted from video frames, used to describe the visual content of the image.

[0056] A video frame refers to a single frame of an image in a video sequence, that is, a single frame in a landscape video.

[0057] Among them, target object information refers to object-related data detected and identified from video frames to guide cropping, which may include the target object's detection box position information, identity identifier, and confidence level.

[0058] Specifically, in this embodiment of the invention, for the landscape video to be processed, it is first decoded into a sequence of video frames arranged in chronological order. For each frame, the image features of that frame are extracted; simultaneously, the target object information of that frame is obtained, including the detection box position, identification, and confidence level. The image features and target object information are then used as inputs for subsequent cross-modal attention fusion.

[0059] As examples, image features can be extracted using backbone feature extraction networks (such as Dinov2, ResNet, VGG, etc.), and the feature dimension can be represented as [B, 1196, 768], where B is the batch size (i.e., the sequential modeling length). The network parameters can be frozen during training to improve efficiency. Target object information can be obtained through face detection algorithms (such as...). The system obtains the face detection bounding boxes (center point coordinates, width and height) in each frame. It assigns a unique face ID to each detected face using a face tracking algorithm (such as ByteTrack). The face region of the frame is then cropped and input into an active speaker detection model (such as TalkNet-Asd) to obtain the speech confidence score corresponding to each detection box. Finally, the center point coordinates, width and height, face ID, and speech confidence score of the face detection boxes are combined into target object information with dimensions [B, K, 6]. Here, K is the maximum number of target objects to retain in a frame (e.g., no more than 10 by default; if more are exceeded, the number is truncated according to the speaker probability and the size of the detection box).

[0060] S110. Perform cross-modal attention fusion based on target object information and image features to obtain multimodal fusion features.

[0061] Cross-modal attention fusion refers to using information from one modality as a query vector and information from another modality as key and value vectors, calculating the similarity between the query and the key, and weighting the value vector, thereby allowing one modality to guide the feature selection process of the other modality.

[0062] Among them, multimodal fusion features refer to the feature representation obtained by fusing target object information with image features through cross-modal attention, which enhances the visual information of the region where the target object is located.

[0063] Specifically, in this embodiment of the invention, target object information (dimension [B, K, 6]) can be mapped to a query vector with the same dimensions as the image features, with the dimension specification being [B, K, 768]. Then, the extracted image features (dimension [B, 1196, 768]) are used as key and value vectors. The query vector Q, key vector K, and value vector V are substituted into the cross-modal attention fusion formula. The similarity between Q and K is calculated, normalized, and then weighted and summed with V to obtain the multimodal fused features. Essentially, this process guides the model to focus more on visual features related to the target object's location in the video frame, under the guidance of the target object information. This results in the fused features (dimension [B, K, 768]) strengthening the expression of the target object's location, providing a basis for subsequent temporal modeling and cropping center point prediction.

[0064] As examples, target object information can be mapped to query vectors through a projection network, where the projection network can be a multilayer perceptron structure containing several fully connected layers and non-linear activation functions to map the original feature dimensions (e.g., 6-dimensional) to the same feature dimensions as the image features (e.g., 768-dimensional). Cross-modal attention fusion can be achieved using a cross-attention mechanism.

[0065] S120. Establish the temporal dependency relationship of multimodal fusion features along the time dimension, and obtain the cropping center point corresponding to each video frame based on the temporal dependency relationship.

[0066] Temporal dependencies refer to the contextual relationships or dynamic change patterns between frames at different time points in a video sequence. Establishing temporal dependencies for multimodal fusion features along the time dimension allows the cropping decision for the current frame to reference information from preceding and following frames, thereby ensuring the temporal continuity and smoothness of the cropping trajectory.

[0067] The cropping center point refers to the coordinate value obtained after regression prediction based on temporal dependencies, which is used to determine the horizontal center position of the portrait screen in the original landscape video frame.

[0068] Specifically, in this embodiment of the invention, multimodal fusion features can be sorted along the time dimension, making the time dimension the primary dimension for temporal modeling. Then, attention calculation is performed on the sorted multimodal fusion features along the time dimension to establish temporal dependencies between video frames, allowing the cropping decision for the current frame to reference information from preceding and following frames. Finally, regression prediction is performed based on this temporal dependency to obtain the cropping center point coordinates for each video frame, thereby utilizing temporal information to ensure that the cropping center point is continuous and smooth in time, avoiding abrupt changes between frames.

[0069] As examples, the original dimensions of the multimodal fusion features are [B, K, 768], which are rearranged to [K, B, 768], making the temporal dimension (i.e., the original B dimension) the primary sequence dimension for attention computation. The rearranged features are simultaneously input as query, key, and value into a multi-head attention layer. The attention mechanism establishes temporal dependencies between video frames along the sequence length dimension. A multi-head attention layer is a neural network layer containing multiple parallel attention heads. Each head independently computes attention and concatenates the outputs to model temporal dependencies between video frames along the temporal dimension. The output of the multi-head attention layer is regressed through a fully connected layer to obtain the cropping center point coordinates for each video frame (output dimension [B, 2], where the first column is the center point position and the second column is the adaptive mask required for subsequent loss calculation).

[0070] S130. Based on the cropping center point, crop the corresponding video frames from landscape to portrait mode to obtain a portrait video sequence.

[0071] Among them, vertical screen video sequence refers to a video sequence composed of vertical screen images with an aspect ratio of less than 1, which are cropped from the original horizontal screen video frames based on the cropping center point of each frame and arranged in chronological order.

[0072] Specifically, in this embodiment of the invention, for each video frame, the corresponding cropping center point is used as the horizontal center of the vertical screen image, and the height of the frame is used as the height of the vertical screen image. The width of the vertical screen image (i.e., height multiplied by the preset aspect ratio) is calculated according to a preset aspect ratio (e.g., 9:16). Then, the image within this width range is cropped from the original horizontal screen frame. If the cropping boundary exceeds the original frame range, boundary constraint processing is performed (e.g., forced boundary value). The vertical screen images obtained from cropping all frames are combined in the original time order to obtain the final vertical screen video sequence, thereby achieving adaptive horizontal-to-vertical conversion centered on the speaker without manual rule intervention.

[0073] This invention provides a method for converting video between landscape and portrait orientations. The method obtains image features of a video frame sequence and target object information for each video frame. Based on the target object information and image features, it performs cross-modal attention fusion to obtain multimodal fusion features. It establishes temporal dependencies among these multimodal fusion features along the time dimension and obtains the cropping center point for each video frame based on these temporal dependencies. Finally, it performs landscape-to-portrait cropping on the corresponding video frames based on the cropping center points to obtain a portrait video sequence. This invention achieves video landscape-to-portrait conversion without relying on complex logical rules by performing cross-modal attention fusion between target object information and image features to focus on the region where the target object is located and establishing temporal dependencies among the multimodal fusion features along the time dimension.

[0074] Optionally, in the above Figure 1 Based on one or more embodiments shown, in another optional embodiment provided by the present invention, step S100 may specifically include:

[0075] Extract the image features of each video frame in the video frame sequence.

[0076] Specifically, in this embodiment of the invention, for an input video frame sequence, each video frame is input to a feature extraction model to obtain the image features corresponding to that frame. This feature extraction model maps the original pixel space to a high-dimensional feature space, outputting a feature representation that can characterize the semantic information of the image.

[0077] As examples, feature extraction models can employ a backbone feature extraction network (such as Dinov2, ResNet, VGG, etc.) to resize each frame of the image to a uniform size (e.g., 644 pixels long and 364 pixels wide, both multiples of 14) before feeding it into the backbone network. This network outputs an image feature map with dimensions [B, 1196, 768], where B is the batch training size (i.e., the sequential modeling length, e.g., 16), 1196 is the number of spatial locations in the feature map, and 768 is the feature dimension at each location. During subsequent model training, the parameters of this backbone network can be frozen and used only as a feature extractor to improve training efficiency.

[0078] Obtain the face detection bounding box, face identifier, and speech confidence score for each video frame, and combine the bounding box information, face identifier, and speech confidence score into target object information.

[0079] Here, a face detection bounding box refers to a rectangular box generated by a face detection algorithm to locate the face in a video frame image. The box information refers to all the geometric parameters contained in the face detection bounding box, specifically including the x-coordinate and y-coordinate of the center point of the bounding box, the width w, and the height h of the bounding box, used to describe the position and size of the face in the image.

[0080] Among them, face identification refers to the unique personal identification number (ID) assigned to the face of the same person in the video through face tracking algorithm. It is used to associate multiple detection boxes belonging to the same person across different video frames to achieve identity consistency across frames.

[0081] Among them, the speaking confidence score refers to the degree of credibility that a face is speaking in the current video frame. The higher the score, the higher the probability that the person is speaking.

[0082] Specifically, in this embodiment of the invention, for each video frame, the face region is first detected and a face detection bounding box is generated. Each box contains information such as center point coordinates, width, and height. Then, a unique face identifier is assigned to each face to achieve cross-frame identity association. Simultaneously, each face region is analyzed to obtain the corresponding speech confidence score. Finally, the bounding box information, face identifier, and speech confidence score of the same face are combined into a single target object information.

[0083] As some examples, face detection boxes can be obtained through face detection algorithms (such as...) The face identifier can be obtained through a face tracking algorithm (such as ByteTrack). The speech confidence score can be obtained by analyzing the cropped face region image and corresponding audio information using an active speaker detection model (such as TalkNet-Asd). The original output value of this model is typically between -5 and 5; values ​​exceeding this range are truncated and then normalized to between -1 and 1. A higher normalized value indicates a higher probability of speech. Finally, the detection bounding box coordinates (x, y, w, h), face ID, and speech confidence score of the same face are combined into a single target object information string. The dimensions are [B, K, 6], where K is the maximum number of target objects to be retained in a frame (the default is no more than 10, and if it exceeds this, it will be truncated according to the speech confidence and the detection box size), and B is the batch size.

[0084] This invention extracts image features from video frame sequences and combines them with face detection boxes, face markers, and speech confidence to construct target object information. This enables subsequent cross-modal attention fusion to accurately focus on the area where the target object is located, guided by the target object information. This improves the accuracy of the association between the cropping center point and the target subject, and achieves the effect of the image adaptively following the target object in horizontal and vertical rotation.

[0085] Optionally, in the above Figure 1 Based on one or more embodiments shown, in another optional embodiment provided by the present invention, step S110 may specifically include:

[0086] The target object information is mapped into a query vector of the same dimension as the image features through a projection network.

[0087] The query vector refers to the vector generated by mapping the target object information through a projection network during the cross-modal attention fusion process. It is used to calculate the similarity with the key vector to guide the model to focus on the region where the target object is located.

[0088] Among them, the projection network refers to the feature transformation structure that maps the target object information from the original feature space to the image feature space, so that the target object information and the image features have the same feature dimension, thereby enabling subsequent attention calculation.

[0089] Specifically, in this embodiment of the invention, for each video frame, the obtained target object information is input into a projection network. This network maps the original dimension of the target object information to the same feature dimension as the image features and outputs a query vector, enabling the target object information to perform cross-modal attention calculation with subsequent image features.

[0090] As examples, projection networks can be implemented using multilayer perceptrons. A multilayer perceptron is a feedforward neural network structure consisting of an input layer, one or more hidden layers, and an output layer. Each layer contains several neurons and a non-linear activation function. The target object information T has dimensions [B, K, 6], where B is the batch size, K is the maximum number of target objects retained per frame, and 6 represents the six values ​​corresponding to each target object: the x-coordinate of the detection box center point, the y-coordinate of the center point, the width of the detection box, the height of the detection box, face identifier, and speech confidence. The projection network maps the original 6-dimensional features to the same feature dimension (768 dimensions) as the image features, outputting a query vector Q with the dimension transformed to [B, K, 768].

[0091] Image features are used as key and value vectors to determine the similarity between the query vector and the key vector. After normalizing the similarity, attention weights are obtained.

[0092] In this context, the key vector refers to a vector composed of image features during cross-modal attention fusion, used to calculate the similarity with the query vector to measure the degree of correlation between each region of the image and the target object information.

[0093] In this context, the value vector refers to the vector that is homologous with the key vector (both originating from image features) during the cross-modal attention fusion process. It is used to perform a weighted summation with the normalized attention weights to generate a multimodal fusion feature that incorporates the guidance information of the target object.

[0094] In cross-modal attention fusion, attention weight refers to the probability distribution value obtained by calculating the similarity (dot product) between the query vector and the key vector, and then normalizing it (e.g., using softmax). The magnitude of this weight reflects the degree of attention the query vector (target object information) pays to each position in the key vector (image features); a higher weight indicates that the position is more relevant to the target object information.

[0095] Specifically, in this embodiment of the invention, the extracted image features F (dimensions [B, 1196, 768]) can be used as the key vector K and the value vector V, respectively. For each target object query vector Q (dimensions [B, K, 768]) in each frame, the dot product similarity between Q and K is calculated to obtain a similarity matrix (dimensions [B, K, 1196]). This similarity is then divided by a scaling factor. ( The feature dimension is 768, and then the attention weight matrix is ​​obtained by normalizing the key vector spatial dimension (1196) using the softmax function. The dimension is [B, K, 1196], and each weight represents the degree of attention the corresponding speaker's query pays to each spatial location in the image. This indicates the matrix transpose.

[0096] The attention weights are summed with the value vectors to obtain multimodal fusion features.

[0097] Specifically, embodiments of the present invention can use the attention weight matrix A matrix multiplication is performed between the weight matrix (dimension [B, K, 1196]) and the value vector V (dimension [B, 1196, 768]), i.e., the weight matrix and the value vector are weighted and summed in spatial dimensions, outputting a multimodal fusion feature with dimension [B, K, 768]. ,Right now This multimodal fusion feature integrates target object information with image visual information, enabling the feature vector corresponding to each target object to be spatially focused on its face and surrounding area, providing a speaker-oriented visual representation for subsequent temporal modeling and cropping center point prediction.

[0098] This invention projects the target object information into a query vector of the same dimension as the image features, and uses the image features as the key vector and value vector. The similarity between the query vector and the key vector is calculated to obtain the attention weight. The attention weight and the value vector are then weighted and summed, enabling the model to dynamically focus on the target object region. This significantly improves the accuracy of the association between the cropping center point and the target object, and achieves the effect of the image adaptively following the target object in both horizontal and vertical orientations.

[0099] Optionally, in the above Figure 1Based on one or more embodiments shown, in another optional embodiment provided by the present invention, step S120 may specifically include:

[0100] The multimodal fusion features are sorted along the time dimension so that the time dimension becomes the main dimension for time series modeling.

[0101] In this context, the primary dimension of temporal modeling refers to the dimensional direction that serves as the core basis for establishing temporal dependencies in multidimensional feature data. In this embodiment of the invention, by adjusting the time dimension as the primary dimension of temporal modeling, subsequent temporal modeling can establish relationships between frames along the time axis, thereby ensuring the continuity and smoothness of the clipping trajectory over time.

[0102] Specifically, in this embodiment of the invention, the multimodal fusion features can be sorted along the time dimension, making the time dimension the main dimension of temporal modeling, thus preparing for the subsequent establishment of temporal dependencies between video frames along the time dimension.

[0103] As examples, the original dimensions of multimodal fusion features can be [B, K, 768], where B is the batch size (i.e., time sequence length) and K is the maximum number of target objects retained per frame. Dimension transformation adjusts dimension B to become the primary dimension for temporal modeling, i.e., rearranges it to [K, B, 768]. In this way, the temporal direction becomes the main axis for subsequent temporal modeling in the multidimensional features, facilitating attention calculations along the time axis to establish inter-frame associations.

[0104] Attention is calculated on the sorted multimodal fusion features along the time dimension to establish the temporal dependency relationship between each video frame.

[0105] Temporal dependencies refer to the contextual relationships or dynamic changes between frames at different time points in a video sequence. Attention calculations applied to multimodal fusion features along the temporal dimension can capture long-distance dependencies between video frames, enabling the model to learn how the cropping decision for the current frame references information from preceding and following frames, thus ensuring the temporal continuity and smoothness of the cropping trajectory.

[0106] Specifically, in this embodiment of the invention, attention calculation is performed on the sorted multimodal fusion features along the time dimension to establish the temporal dependency relationship between each video frame, so that the cropping decision of the current frame can refer to the information of the previous and next frames, ensuring the continuity and smoothness of the cropping trajectory.

[0107] As examples, embodiments of the present invention can simultaneously input the dimension-transformed feature R' (dimensions [K, B, 768]) as a query, key, and value into a multi-head attention layer. A multi-head attention layer is a neural network layer containing multiple parallel attention heads. Each attention head independently calculates the correlation between different subspaces of the input features, and then the outputs of each head are concatenated and linearly transformed. In this layer, multiple parallel attention heads each calculate the attention distribution, weighting and aggregating features along the sequence length dimension (i.e., the original B dimension, representing time). This allows the model to capture the contextual dependencies between different video frames in terms of speaker state and visual features, thereby enabling the cropping decision of the current frame to reference information from preceding and following frames, ensuring the continuity and smoothness of the cropping trajectory.

[0108] Regression prediction is performed based on temporal dependencies to obtain the cropping center point for each video frame.

[0109] Specifically, in this embodiment of the invention, based on the established temporal dependency relationship, the corresponding feature representation is used for regression prediction to obtain the coordinates of the cropping center point corresponding to each video frame. This utilizes temporal information to ensure that the cropping center point is continuous and smooth in time, avoiding abrupt changes between frames, and providing a basis for subsequent landscape-to-portrait cropping.

[0110] As examples, embodiments of the present invention can achieve regression prediction through fully connected layers. A fully connected layer is a neural network layer in which each input neuron has a trainable weight connection to each output neuron. First, the calculated results of the temporal dependencies (dimensions [K, B, 768]) are restored to [B, 768] (by merging the K dimensions or other methods), and then input into the fully connected layer. This fully connected layer maps the high-dimensional features to an output of [B, 2], where the first column corresponds to the cropping center point coordinates (horizontal position) of each video frame, and the second column outputs an adaptive mask. This yields the cropping center point for each video frame, used for subsequent landscape-to-portrait cropping operations.

[0111] This invention establishes temporal dependencies between video frames by sorting multimodal fusion features along the time dimension and performing attention calculations along the time dimension. Based on these temporal dependencies, regression prediction is performed to obtain the cropping center point, making the cropping trajectory continuous and smooth in time. This effectively avoids abrupt changes between frames and image jitter, improving the stability of horizontal to vertical transitions and the viewing experience.

[0112] Optional, based on Figure 1 The method shown is as follows: Figure 2 The diagram shows a flowchart of one embodiment of the model parameter training process provided in this invention. This model parameter training process may include:

[0113] S200: Obtain the labeled cropping center points of the training video frame sequence.

[0114] Specifically, in this embodiment of the invention, for each video in the training dataset, the ideal vertical screen cropping center point (usually based on the center of the speaker's face) is first labeled for each frame manually or semi-automatically, and all labeled values ​​are standardized to the range of 0 to 1 (relative to the image width). The labeled data, along with the corresponding video frame path and target object information, are stored together to form training samples (e.g., in JSON format, containing "path", "face", and "result" fields).

[0115] S210. Perform regression prediction on the calculation results of temporal dependencies to obtain the predicted cropping center point and adaptive mask for each video frame.

[0116] The adaptive mask refers to a learnable weight parameter generated for each video frame during training. It dynamically reduces the weight of acceleration sequence elements corresponding to shot transitions or speaker transitions in the acceleration smoothing loss, preventing the drastic changes in these special frames from affecting the overall smoothing constraint. This adaptive mask automatically adjusts with the network parameters during training, and its dimension is the same as the length of the crop center point sequence.

[0117] Specifically, in this embodiment of the invention, regression prediction is performed on the calculation results of temporal dependencies, and the predicted crop center point and adaptive mask for each video frame are obtained simultaneously. The predicted crop center point is used to calculate the basic position loss with the labeled crop center point, and the adaptive mask is used to calculate the acceleration smoothing loss in conjunction with the acceleration sequence.

[0118] As examples, embodiments of the present invention can achieve regression prediction through a fully connected layer. The calculation results of temporal dependencies are input into the fully connected layer, and the output dimension of the fully connected layer is [B, 2]. The first column serves as the coordinates P of the predicted pruning center point, and the second column serves as the adaptive mask M. The two outputs are optimized synchronously during training.

[0119] S220. Determine the basic position loss between the predicted clipping center point and the marked clipping center point.

[0120] Here, the base position loss refers to the loss function that measures the difference between the crop center point predicted by the model and the actual crop center point labeled by the human. In this embodiment of the invention, Smooth L1 Loss can be used. This loss guides the model to learn the accurate crop center position.

[0121] Specifically, in this embodiment of the invention, Smooth L1 Loss can be used as the basic position loss function. For each video frame, calculate the difference between the predicted crop center point P and the labeled crop center point y, using the following formula: ,in, It is usually set to 1.

[0122] S230. The predicted cropping center points of each video frame are arranged in chronological order to form a predicted center point sequence.

[0123] The predicted center point sequence refers to a one-dimensional sequence formed by arranging the predicted cropping center points output by the model on each video frame in chronological order. The sequence length is equal to the number of video frames used in training.

[0124] Specifically, in this embodiment of the invention, for B consecutive video frames in a batch, the predicted cropping center point P (a one-dimensional sequence of length B) is arranged in frame order to form a predicted center point sequence, where the i-th element in the sequence corresponds to the coordinates of the predicted cropping center point of the i-th frame.

[0125] S240, Obtain the acceleration sequence of the predicted center point sequence.

[0126] The acceleration sequence refers to the sequence obtained by performing first-order and second-order difference operations on the predicted center point sequence in sequence. Each element represents the acceleration change at the corresponding frame and is used to measure the smoothness of the movement of the cropping center point.

[0127] Specifically, in this embodiment of the invention, a predetermined number of elements are first added to the beginning of the predicted center point sequence to increase the sequence length. Then, the first-order difference is calculated on the expanded sequence to obtain the velocity sequence, and the first-order difference is calculated on the velocity sequence to obtain the acceleration sequence. Finally, the sequence length is restored to the original length, thus obtaining the acceleration sequence of the predicted center point sequence.

[0128] As examples, embodiments of the present invention can first supplement the beginning of the predicted center point sequence with two elements, each with a value equal to the first element of the original sequence, increasing the sequence length to B+2. Then, a first-order difference (subtraction of adjacent elements) is calculated on the expanded sequence to obtain a velocity sequence, and then another first-order difference is calculated on the velocity sequence to obtain an acceleration sequence. After these two differences, the sequence length is restored to B, thus obtaining the acceleration sequence, where each element represents the acceleration change at the corresponding frame. The length of this acceleration sequence is the same as the length of the adaptive mask, and it is used for subsequent element-wise multiplication with the adaptive mask to calculate the acceleration smoothing loss.

[0129] S250. Multiply the adaptive mask by the corresponding elements of the acceleration sequence to obtain the acceleration smoothing loss.

[0130] The acceleration smoothing loss refers to the loss term obtained by multiplying the acceleration sequence based on the predicted center point sequence element-wise with the adaptive mask. The acceleration sequence reflects the change in acceleration of the cropping center point. By minimizing this loss term and using the mask to ignore the influence of switching frames, the cropping center point can be allowed to move at a constant speed in time but abrupt acceleration changes can be suppressed, thereby effectively reducing image jitter.

[0131] Specifically, in this embodiment of the invention, an adaptive mask can be multiplied with the corresponding elements of the acceleration sequence to obtain the acceleration smoothing loss. This adaptive mask is automatically adjusted during training and can output a value close to 0 at camera transitions or target object transition frames, thereby reducing the contribution of these frames to the smoothing loss.

[0132] As some examples, embodiments of the present invention can use adaptive masks. (Length B) and second-order difference results (acceleration sequence) Multiply each element-wise, then sum all the products to obtain the acceleration smoothing loss. ,Right now: .

[0133] S260. The sum of the base position loss and the acceleration smoothing loss is used as the total loss to adjust the trainable parameters in the cross-modal attention fusion, temporal dependency establishment, and regression prediction processes.

[0134] Trainable parameters refer to the parameters that need to be updated through backpropagation and gradient descent during model training. The adjustment of these parameters aims to minimize the total loss function, enabling the model to accurately predict the cropping center point and generate a smooth cropping trajectory.

[0135] Specifically, in this embodiment of the invention, the basic position loss and the acceleration smoothing loss are added together to form the total loss. Then, the backpropagation algorithm is used to calculate the gradient of the total loss with respect to all trainable parameters in the process of cross-modal attention fusion, establishment of temporal dependencies and regression prediction. These parameters are then updated by the optimizer so that the model gradually converges to the optimal state that can accurately predict the pruning center point and ensure temporal smoothness.

[0136] As some examples, trainable parameters may include the projection matrix in a cross-attention mechanism, the weight matrix and biases in a multi-head attention layer, and the weight matrix and biases in a fully connected layer. The optimizer can be the Adam optimizer, and the learning rate can be set to 0.0001. Total loss. It can be represented as: ,in, Based on location loss, This is the acceleration smoothing loss. Added to the acceleration smoothing loss, it forms the gradient of the total loss with respect to each trainable parameter, calculated using the backpropagation algorithm. The optimizer then updates these parameters, gradually bringing the model to an optimal state that accurately predicts the pruning center point while ensuring temporal smoothness.

[0137] This invention provides a method to simultaneously obtain the predicted crop center point and an adaptive mask through regression prediction. It calculates the basic position loss to ensure the positional accuracy of the crop center point, and uses the adaptive mask multiplied with the acceleration sequence to obtain the acceleration smoothing loss. The sum of the two losses is used to adjust the trainable parameters, so that while maintaining the positional accuracy of the crop center point, it can automatically suppress the shaking effect at the camera or speaker frame switching, thereby achieving a more stable and smooth horizontal to vertical transition effect during the inference stage.

[0138] Optionally, in the above Figure 2 Based on one or more embodiments shown, in another optional embodiment provided by the present invention, step S240 may specifically include:

[0139] The velocity sequence is obtained by performing a first-order difference operation on the predicted center point sequence.

[0140] The velocity sequence refers to the sequence obtained by performing a first-order difference operation on the predicted center point sequence. Each element represents the difference between the predicted cropping center points of two adjacent frames, reflecting the movement speed of the cropping center point per unit time. The sequence length is the same as the original predicted center point sequence (after boundary padding).

[0141] Specifically, in this embodiment of the invention, the length of the prediction center point sequence P can be set to B (B is the batch size, i.e., the time-series modeling length). First, two elements are added to the beginning of sequence P, with each element's value equal to the value of the first element of P, making the length of the expanded sequence B+2. Then, the difference between adjacent elements is calculated sequentially for this expanded sequence; that is, for i from 1 to B+1, the difference is calculated... , thus obtaining the velocity sequence Due to the padding operation at the beginning of the original sequence, after truncating the effective portion, the length of the velocity sequence is the same as the length of the original predicted center point sequence (restored to B after boundary processing).

[0142] The velocity sequence is subjected to a first-order difference operation again to obtain the acceleration sequence, where the length of the acceleration sequence is the same as the length of the adaptive mask.

[0143] The acceleration sequence refers to the sequence obtained by performing a first-order difference operation on the velocity sequence again. Each element represents the difference between two adjacent velocity values, reflecting the acceleration change at the clipping center point. The sequence length is the same as the adaptive mask (restored to its original length after boundary padding) and is used to measure the smoothness of the clipping trajectory. The smaller the absolute value of the acceleration, the smoother the motion.

[0144] Specifically, in this embodiment of the invention, the velocity sequence obtained in the previous step can be used... (Length B+1, reduced by 1 after first-order differencing, but restored to B+1 through header padding) The difference between adjacent elements is calculated again; that is, for i from 1 to B, the difference is calculated... , thus obtaining the acceleration sequence Because of the head padding and truncation operations during the two differencing processes, the final length of the acceleration sequence is restored to the original sequence length B, which is the same as the length of the adaptive mask M. This acceleration sequence reflects the acceleration changes of the truncation center point and is used for subsequent element-wise multiplication with the adaptive mask to calculate the acceleration smoothing loss.

[0145] The embodiments of the present invention obtain an acceleration sequence by performing first-order and second-order differences on the predicted center point sequence, which provides an accurate quantitative basis for acceleration smoothing loss. This effectively suppresses abrupt acceleration changes at the cropping center point, and maintains a smooth transition of the image when switching between shots or speakers, significantly improving the viewing smoothness of horizontal to vertical videos.

[0146] Optionally, the adaptive mask provided in this embodiment of the invention is automatically adjusted during training to reduce the weight of the acceleration sequence corresponding to the shot switching frame or speaker switching frame in the acceleration smoothing loss.

[0147] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0148] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0149] Corresponding to the above method embodiments, this invention also provides a video landscape / portrait screen switching device, the structure of which is as follows: Figure 3 As shown, it may include: a video feature information acquisition unit 10, a multimodal fusion feature acquisition unit 20, a cropping center point acquisition unit 30, and a video cropping unit 40.

[0150] The video feature information acquisition unit 10 is used to acquire the image features of the video frame sequence and the target object information of each video frame.

[0151] The multimodal fusion feature acquisition unit 20 is used to perform cross-modal attention fusion based on target object information and image features to obtain multimodal fusion features.

[0152] The cropping center point acquisition unit 30 is used to establish the temporal dependency relationship of multimodal fusion features along the time dimension, and obtain the cropping center point corresponding to each video frame based on the temporal dependency relationship.

[0153] The video cropping unit 40 is used to crop the corresponding video frames from horizontal to vertical screen based on the cropping center point to obtain a vertical video sequence.

[0154] Optionally, the video feature information acquisition unit 10 can be used to extract the image features of each video frame in the video frame sequence;

[0155] Obtain the face detection bounding box, face identifier, and speech confidence score for each video frame, and combine the bounding box information, face identifier, and speech confidence score into target object information.

[0156] Optionally, the multimodal fusion feature acquisition unit 20 can be used to map the target object information into a query vector with the same dimension as the image features through a projection network; use the image features as key vectors and value vectors to determine the similarity between the query vector and the key vector, and after normalizing the similarity, obtain the attention weight; and sum the attention weight and the value vector by weight to obtain the multimodal fusion feature.

[0157] Optionally, the crop center point acquisition unit 30 can be used to sort the multimodal fusion features along the time dimension so that the time dimension becomes the main dimension of temporal modeling; perform attention calculation on the sorted multimodal fusion features along the time dimension to establish the temporal dependency relationship between each video frame; and perform regression prediction based on the temporal dependency relationship to obtain the crop center point corresponding to each video frame.

[0158] Optionally, the video landscape / portrait switching device may also include a model parameter training unit.

[0159] The model parameter training unit is used to obtain the labeled crop center points of the training video frame sequence; perform regression prediction on the calculation results of temporal dependencies to obtain the predicted crop center points and adaptive masks for each video frame; determine the basic position loss between the predicted crop center points and the labeled crop center points; construct a predicted center point sequence by arranging the predicted crop center points of each video frame in chronological order; obtain the acceleration sequence of the predicted center point sequence; multiply the corresponding elements of the adaptive mask and the acceleration sequence to obtain the acceleration smoothing loss; and use the sum of the basic position loss and the acceleration smoothing loss as the total loss to adjust the trainable parameters in the cross-modal attention fusion, temporal dependency establishment, and regression prediction processes.

[0160] Optionally, the model parameter training unit can be used to perform a first-order difference operation on the predicted center point sequence to obtain a velocity sequence; and to perform a first-order difference operation on the velocity sequence again to obtain an acceleration sequence, wherein the length of the acceleration sequence is the same as the length of the adaptive mask.

[0161] Optionally, an adaptive mask is automatically adjusted during training to reduce the weight of the acceleration sequences corresponding to shot transition frames or speaker transition frames in the acceleration smoothing loss.

[0162] This invention provides a video landscape / portrait conversion device that obtains image features of a video frame sequence and target object information for each video frame; performs cross-modal attention fusion based on the target object information and image features to obtain multimodal fusion features; establishes temporal dependencies of the multimodal fusion features along the time dimension, and obtains the cropping center point corresponding to each video frame based on the temporal dependencies; and performs landscape-to-portrait cropping on the corresponding video frames based on the cropping center points to obtain a portrait video sequence. This invention achieves video landscape / portrait conversion without relying on complex logical rules by performing cross-modal attention fusion of target object information and image features to focus on the region where the target object is located and establishing temporal dependencies of the multimodal fusion features along the time dimension.

[0163] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0164] The video landscape / portrait screen conversion device includes a processor and a memory. The video feature information acquisition unit 10, the multimodal fusion feature acquisition unit 20, the cropping center point acquisition unit 30, and the video cropping unit 40 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0165] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting the kernel parameters, cross-modal attention fusion of target object information and image features is performed to focus on the region where the target object is located. Temporal dependencies of multimodal fusion features are established along the time dimension, thereby adaptively obtaining a smooth cropping center point and realizing video landscape / portrait conversion without relying on complex logical rules.

[0166] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the video landscape / portrait switching method.

[0167] This invention provides a processor for running a program, wherein the program executes the video landscape / portrait switching method during runtime.

[0168] like Figure 4As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned video landscape / portrait screen switching method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0169] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program that initializes a method for switching between landscape and portrait modes for video.

[0170] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0172] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0173] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0175] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0176] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0177] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0178] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

1. A method for converting video between landscape and portrait orientations, characterized in that, include: Obtain image features of the video frame sequence and target object information for each video frame; Cross-modal attention fusion is performed based on the target object information and the image features to obtain multimodal fusion features; Establish the temporal dependency relationship of the multimodal fusion features along the time dimension, and obtain the cropping center point corresponding to each video frame based on the temporal dependency relationship; Based on the cropping center point, the corresponding video frames are cropped from landscape to portrait mode to obtain a portrait video sequence.

2. The method according to claim 1, characterized in that, The process of obtaining image features of the video frame sequence and target object information for each video frame includes: Extract the image features of each video frame in the video frame sequence; Obtain the face detection bounding box, face identifier, and speech confidence score for each video frame, and combine the bounding box information, face identifier, and speech confidence score into target object information.

3. The method according to claim 1, characterized in that, The step of performing cross-modal attention fusion based on the target object information and the image features to obtain multimodal fusion features includes: The target object information is mapped into a query vector of the same dimension as the image features through a projection network; The image features are used as key vectors and value vectors to determine the similarity between the query vector and the key vector. After normalizing the similarity, attention weights are obtained. The attention weights are summed with the value vector to obtain the multimodal fusion features.

4. The method according to claim 1, characterized in that, The process of establishing the temporal dependency relationship of the multimodal fusion features along the time dimension, and obtaining the cropping center point corresponding to each video frame based on the temporal dependency relationship, includes: The multimodal fusion features are sorted along the time dimension so that the time dimension becomes the main dimension for time series modeling; Attention is calculated on the sorted multimodal fusion features along the time dimension to establish the temporal dependency relationship between each video frame; Regression prediction is performed based on the temporal dependency to obtain the cropping center point corresponding to each video frame.

5. The method according to claim 4, characterized in that, Also includes: Obtain the labeled cropping center points of the training video frame sequence; Regression prediction is performed on the calculation results of the temporal dependency to obtain the predicted cropping center point and adaptive mask for each video frame; Determine the basic positional loss between the predicted clipping center point and the labeled clipping center point; The predicted cropping center points of each video frame are arranged in chronological order to form a sequence of predicted center points; Obtain the acceleration sequence of the predicted center point sequence; Multiply the adaptive mask by the corresponding element of the acceleration sequence to obtain the acceleration smoothing loss; The sum of the basic position loss and the acceleration smoothing loss is used as the total loss to adjust the trainable parameters in the cross-modal attention fusion, temporal dependency establishment, and regression prediction processes.

6. The method according to claim 5, characterized in that, The process of obtaining the acceleration sequence of the predicted center point sequence includes: Perform a first-order difference operation on the predicted center point sequence to obtain the velocity sequence; The velocity sequence is subjected to a first-order difference operation again to obtain an acceleration sequence, wherein the length of the acceleration sequence is the same as the length of the adaptive mask.

7. The method according to claim 6, characterized in that, The adaptive mask is automatically adjusted during training to reduce the weight of the acceleration sequence corresponding to the shot switching frame or speaker switching frame in the acceleration smoothing loss.

8. A video landscape / portrait screen switching device, characterized in that, include: The system includes a video feature information acquisition unit, a multimodal fusion feature acquisition unit, a cropping center point acquisition unit, and a video cropping unit. The video feature information acquisition unit is used to obtain the image features of the video frame sequence and the target object information of each video frame; The multimodal fusion feature acquisition unit is used to perform cross-modal attention fusion based on the target object information and the image features to obtain multimodal fusion features; The cropping center point acquisition unit is used to establish the temporal dependency relationship of the multimodal fusion features along the time dimension, and obtain the cropping center point corresponding to each video frame based on the temporal dependency relationship. The video cropping unit is used to crop the corresponding video frames from horizontal to vertical screen based on the cropping center point to obtain a vertical video sequence.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the video landscape / portrait switching method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the video landscape / portrait switching method as described in any one of claims 1 to 7.

Images