Lens segmentation, visual task processing method, device, electronic device and medium

Abstract

The present disclosure provides a shot segmentation and visual task processing method, apparatus, electronic device, and medium, relating to the fields of artificial intelligence technology, particularly video processing, image processing, computer vision, and deep learning technology. A specific implementation scheme comprises: performing two-dimensional feature extraction on at least one video frame in a video to obtain local visual features corresponding to the at least one video frame; determining global visual features corresponding to the at least one video frame based on the local visual features corresponding to the at least one video frame; and determining shot segmentation information for the video based on the global visual features corresponding to the at least one video frame.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, particularly to video processing, image processing, computer vision, and deep learning technology. Specifically, it relates to a shot segmentation and visual task processing method, device, electronic device, and medium. Background Art

[0002] With the development of computer technology, artificial intelligence has also advanced. For example, AI can be used to achieve video understanding. Shot segmentation is a task in video retrieval and provides key information for video understanding. Shot segmentation refers to the technology used to divide a complete video into shot segments based on shot boundaries. Summary of the Invention

[0003] The present disclosure provides a method, device, electronic device, and medium for shot segmentation and visual task processing.

[0004] According to one aspect of the present disclosure, a shot segmentation method is provided, comprising: performing two-dimensional feature extraction on at least one video frame in a video to obtain local visual features corresponding to the at least one video frame; determining global visual features corresponding to the at least one video frame based on the local visual features corresponding to the at least one video frame; and determining shot segmentation information of the video based on the global visual features corresponding to the at least one video frame.

[0005] According to another aspect of the present disclosure, a visual task processing method is provided, comprising: determining shot segmentation information of a video to be processed, wherein the shot segmentation information of the video to be processed is determined using a shot segmentation method; and processing a visual task according to the shot segmentation information of the video to be processed.

[0006] According to another aspect of the present disclosure, a shot segmentation device is provided, comprising: a two-dimensional feature extraction module for performing two-dimensional feature extraction on at least one video frame in a video to obtain local visual features corresponding to the at least one video frame; a first determination module for determining global visual features corresponding to the at least one video frame based on the local visual features corresponding to the at least one video frame; and a second determination module for determining shot segmentation information of the video based on the global visual features corresponding to the at least one video frame.

[0007] According to another aspect of the present disclosure, a visual task processing device is provided, including: a third determination module for determining the shot segmentation information of a video to be processed, wherein the shot segmentation information of the video to be processed is determined using a shot segmentation method; and a processing module for processing a visual task according to the shot segmentation information of the video to be processed.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above in the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the method described above.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 Schematically illustrates an exemplary system architecture to which the shot segmentation method, visual task processing method, and apparatus according to an embodiment of the present disclosure can be applied;

[0014] Figure 2 The flowchart of the shot segmentation method according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3 An example schematic diagram of a shot segmentation method according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 4A Schematically illustrates an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level under the condition of 1<m≤M according to an embodiment of the present disclosure;

[0017] Figure 4B Schematically illustrating an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level when 1<m≤M according to another embodiment of the present disclosure;

[0018] Figure 4CSchematically illustrating an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level when 1<m≤M according to another embodiment of the present disclosure;

[0019] Figure 4D Schematically illustrating an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level when 1<m≤M according to another embodiment of the present disclosure;

[0020] Figure 5A Schematically illustrates an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at a second level when m=1 according to an embodiment of the present disclosure;

[0021] Figure 5B Schematically illustrates an example schematic diagram of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at a second level when m=1 according to another embodiment of the present disclosure;

[0022] Figure 5C Schematically illustrates an example schematic diagram of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at a second level when m=1 according to another embodiment of the present disclosure;

[0023] Figure 5D Schematically illustrates an example schematic diagram of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at a second level when m=1 according to another embodiment of the present disclosure;

[0024] Figure 6 Schematically shows a flow chart of a visual task processing method according to an embodiment of the present disclosure;

[0025] Figure 7 An example schematic diagram of a visual task processing method according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 8 The following schematically shows a block diagram of a shot splitting device according to an embodiment of the present disclosure;

[0027] Figure 9 A block diagram schematically shows a visual task processing apparatus according to an embodiment of the present disclosure; and

[0028] Figure 10 The block diagram of an electronic device suitable for implementing the shot segmentation method and the visual task processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] Video is a media format for storing dynamic information and communicating it. Video can be categorized into video clips, video scenes, video shots (i.e., shots), and video frames, from coarse to fine granularity. A video can include at least one video clip. A video clip can include at least one video scene. A video scene can include at least one shot.

[0031] A shot can consist of several temporally consecutive frames captured by a set of image acquisition devices. Shot segmentation refers to the technology used to divide a complete video into shot segments based on shot boundaries. The effectiveness of shot segmentation can easily affect subsequent video structuring and video retrieval.

[0032] The shot segmentation method may include one of the following methods: a shot segmentation method based on feature differences between adjacent images and a shot segmentation method based on a sliding window.

[0033] However, since the shot segmentation method based on the difference in adjacent image features does not consider more contextual information, and the various strategies in this method need to rely on thresholds for adjustment, it is difficult to determine the thresholds that are suitable for various scenarios, which leads to the poor adaptability and flexibility of the shot segmentation method based on the difference in adjacent image features.

[0034] Furthermore, since sliding window-based shot segmentation methods typically utilize 3D Convolutional Neural Networks (3D CNNs) for shot segmentation, the computational complexity of 3D CNNs is high, resulting in significant consumption of computational and storage resources. Furthermore, since 3D CNNs typically only consider local information, the shot segmentation results are poor.

[0035] To this end, embodiments of the present disclosure propose a shot segmentation solution. For example, two-dimensional feature extraction is performed on at least one video frame in a video to obtain local visual features corresponding to the at least one video frame. Based on the local visual features corresponding to the at least one video frame, global visual features corresponding to the at least one video frame are determined. Based on the global visual features corresponding to the at least one video frame, shot segmentation information for the video is determined.

[0036] According to the embodiments of the present disclosure, since the local visual features of video frames are obtained by extracting two-dimensional features from the video frames rather than using three-dimensional features, resource consumption is reduced and processing efficiency is improved. In addition, since the shot segmentation information is determined based on the global visual features of the video frames, and the global visual features of the video frames are obtained based on the local visual features of the video frames, the global visual features can represent the temporal information between video frames, thereby improving the accuracy of shot segmentation.

[0037] Figure 1 An exemplary system architecture to which the shot segmentation method, visual task processing method, and apparatus according to an embodiment of the present disclosure can be applied is schematically illustrated.

[0038] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure. This does not mean that the embodiments of the present disclosure cannot be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the shot segmentation method, visual task processing method, and apparatus may be applied may include a terminal device, but the terminal device may implement the shot segmentation method, visual task processing method, and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0039] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as at least one of a wired and wireless communication link. The terminal device may include at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0040] A user can use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. At least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 can be installed with various communication client applications, such as at least one of a knowledge reading application, a web browser application, a search application, an instant messaging tool, an email client, and a social platform software.

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing. For example, the electronic device can include at least one of a smartphone, a tablet computer, a laptop computer, and a desktop computer.

[0042] Server 105 can be a server that provides various services. For example, server 105 can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the problems of traditional physical hosts and VPS services (Virtual Private Servers), such as difficult management and poor business scalability.

[0043] It should be noted that the shot segmentation method and visual task processing method provided in the embodiments of the present disclosure can generally be executed by one of the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the shot segmentation apparatus and visual task processing apparatus provided in the embodiments of the present disclosure can also be set in one of the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0044] Alternatively, the shot segmentation method and the visual task processing method provided by the embodiments of the present disclosure may also be generally executed by the server 105. Accordingly, the shot segmentation device and the visual task processing device provided by the embodiments of the present disclosure may generally be set in the server 105. The shot segmentation method and the visual task processing method provided by the embodiments of the present disclosure may also be executed by a server or a server cluster that is different from the server 105 and can communicate with at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. Accordingly, the shot segmentation device and the visual task processing device provided by the embodiments of the present disclosure may also be set in a server or a server cluster that is different from the server 105 and can communicate with at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105.

[0045] It should be understood that Figure 1 The number of the first terminal device, the second terminal device, the third terminal device, the network and the server is only illustrative. According to the implementation requirements, there can be any number of the first terminal device, the second terminal device, the third terminal device, the network and the server.

[0046] It should be noted that the sequence numbers of the operations in the following method are only used to indicate the operation for the purpose of description, and should not be regarded as indicating the order in which the operations should be performed. Unless explicitly stated, the method does not need to be performed in the order shown.

[0047] Figure 2 The flowchart of the shot segmentation method according to an embodiment of the present disclosure is schematically shown.

[0048] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0049] In operation S210 , two-dimensional feature extraction is performed on at least one video frame in a video to obtain a local visual feature corresponding to the at least one video frame.

[0050] In operation S220 , a global visual feature corresponding to the at least one video frame is determined based on the local visual feature corresponding to the at least one video frame.

[0051] In operation S230, shot segmentation information of the video is determined based on the global visual features corresponding to at least one video frame.

[0052] According to embodiments of the present disclosure, videos can be divided into video clips, video scenes, shots, and video frames, from coarse to fine granularity. A video can include at least one video clip. A video clip can include at least one video scene. A video scene can include at least one shot. The shots within a video scene can have similar semantic information. A shot can refer to video frames captured continuously by an image acquisition device. Video frames within the same shot have similar semantic information. A shot can refer to a logical segment.

[0053] According to an embodiment of the present disclosure, visual features may include at least one of shallow visual features and deep visual features. Shallow visual features may refer to fine-grained visual features used to characterize video frames. Fine-grained visual features may include at least one of the following: color features, texture features, edge features, and angular features. Deep visual features may be used to characterize coarse-grained visual features of video frames. Coarse-grained visual features may refer to abstract visual features. Abstract visual features may refer to visual features that can express semantic information. Semantic information may include at least one of local semantic information and global semantic information. Local semantic information may be used to characterize the semantic features of the video frame itself. Global semantic information may be used to characterize the semantic information of the video frame itself and the associated semantic information between the video frame and other video frames. The receptive field of global semantic information is larger than that of local semantic information.

[0054] According to an embodiment of the present disclosure, deep visual features may include at least one of the following: local deep visual features and global deep visual features. Shallow visual features may include at least one of the following: local shallow visual features and global shallow visual features. Global visual features can represent the temporal information between video frames. Global visual features may include at least one of global deep visual features and global shallow visual features. Local visual features may include at least one of local deep visual features and local shallow visual features. Global visual features may include at least one of global shallow visual features and global deep visual features. Local visual features may include at least one of local shallow visual features and local deep visual features. In addition, global visual features may include at least one of global explicit visual features and global implicit visual features. Local shallow visual features may include at least one of the following: corner-type local visual features and region-type local visual features. Corner-type local visual features may include at least one of the following: corner features, line features, and curve features. Region-type local visual features may include at least one of the following: edge features and other attribute region features.

[0055] According to an embodiment of the present disclosure, shot segmentation information can be used to characterize the relationship between video frames and shot cut video frames. A shot cut video frame may refer to a video frame in which a shot cut occurs in a video. The shot segmentation information may include at least one of shot segmentation information characterizing that the video frame is a shot cut video frame and shot segmentation information characterizing that the video frame is a non-shot cut video frame. A non-shot cut video frame may refer to a video frame in which no shot cut occurs in a video. The shot segmentation information may include a shot segmentation position. The shot segmentation position may refer to the position of the video frame in which a shot cut occurs in a video. The shot segmentation position can be used for video editing.

[0056] For example, a first predetermined identifier can be used to indicate the shot segmentation information of a video frame that is a shot cut video frame. A second predetermined identifier can be used to indicate the shot segmentation information of a video frame that is not a shot cut video frame. The first predetermined identifier and the second predetermined identifier can be configured based on actual business needs and are not limited here. For example, the first predetermined identifier can be "1." The second predetermined identifier can be "0." Alternatively, the first predetermined identifier can be "0." The second predetermined identifier can be "1."

[0057] According to an embodiment of the present disclosure, in response to detecting a shot splitting instruction, a video can be obtained from a data source. The data source can include at least one of the following: a local database, a cloud database, and a network resource. The video can refer to a data stream encoding video frames in chronological order. For example, a data interface can be invoked. The data interface is used to obtain the video from the data source.

[0058] According to embodiments of the present disclosure, two-dimensional feature extraction can be performed on video frames based on at least one of a traditional two-dimensional feature extraction method and a two-dimensional feature extraction method based on a deep learning model to obtain local visual features of the video frames. For example, traditional two-dimensional feature extraction methods may include at least one of the following: interest point detection methods and dense extraction methods. Interest points include at least one of the following: Harris corner detector, Laplacian of Gaussian (LoG) operator, Difference of Gaussian (DoG) operator, etc. Dense extraction methods may include at least one of the following: Scale-invariant feature transform (SIFT), Histogram of Oriented Gradient (HOG), and Local Binary Pattern (LBP). For example, the first deep learning model may include at least one of the following: a deep learning model based on a first visual encoder and a deep learning model based on a first convolutional neural network. The first visual encoder may include an encoder of a visual transformer (i.e., a transformer). The visual transformer may include at least one of the following: Vision Transformer and Swin Transformer. The deep learning model based on the first convolutional neural network may include a deep learning model based on two-dimensional convolution. The deep learning model based on two-dimensional convolution may include at least one of the following: AlexNet, LeNet, VGG (Visual Geometry Group), D2-Net, R2D2, etc.

[0059] According to an embodiment of the present disclosure, local visual features corresponding to at least one video frame can be processed based on a global feature extraction strategy to obtain global visual features corresponding to the at least one video frame. For example, local visual features corresponding to each of the at least one video frame can be processed based on the global feature extraction strategy to obtain global visual features corresponding to each of the at least one video frame. The global feature extraction strategy may include at least one of the following: an attention strategy and a similarity strategy. The attention strategy may include at least one of the following: a self-attention strategy and a mutual attention strategy. In addition, the attention strategy may include a multi-head attention strategy. The self-attention strategy may include a multi-head self-attention strategy. The mutual attention strategy may include a multi-head mutual attention strategy. The global visual features may include at least one of the following: a global implicit visual feature and a global explicit visual feature. The global implicit visual feature may be used to implicitly characterize the global visual feature. The global implicit visual feature may include at least one of the following: a self-attention feature. The global explicit visual feature may be used to explicitly characterize the global visual feature. The global explicit visual feature may include a similarity feature.

[0060] According to an embodiment of the present disclosure, after determining the global visual features corresponding to the video frames, shot segmentation (i.e., classification) can be performed on the global visual features corresponding to at least one video frame to obtain shot segmentation information for the video. For example, shot segmentation can be performed on the global visual features corresponding to each of the at least one video frames to obtain shot segmentation information corresponding to each of the at least one video frames. Shot segmentation information for the video is obtained based on the shot segmentation information corresponding to each of the at least one video frames.

[0061] According to an embodiment of the present disclosure, performing shot segmentation on the global visual features corresponding to each of at least one video frame to obtain shot segmentation information corresponding to each of at least one video frame may include: when the global visual features include global implicit visual features, performing shot segmentation on the global implicit visual features corresponding to each of at least one shot to obtain shot segmentation information corresponding to each of at least one shot. When the global visual features include global explicit visual features, performing shot segmentation on the global explicit visual features corresponding to each of at least one shot to obtain shot segmentation information corresponding to each of at least one shot. When the global visual features include global explicit visual features and global implicit visual features, performing scene segmentation on the fusion features obtained from the global explicit visual features and the global implicit visual features corresponding to each of at least one shot to obtain shot segmentation information corresponding to each of at least one video frame.

[0062] According to an embodiment of the present disclosure, determining shot segmentation information of a video based on global visual features corresponding to at least one video frame may include: obtaining a temporal feature corresponding to the at least one video frame based on the global visual features corresponding to the at least one video frame. Determining video segmentation information of the video based on the temporal feature corresponding to the at least one video frame. The temporal feature may represent a temporal relationship between at least one video frame.

[0063] According to an embodiment of the present disclosure, determining the temporal features corresponding to at least one video frame based on the global visual features corresponding to at least one video frame may include: processing the global visual features corresponding to at least one video frame using a temporal model to determine the temporal features corresponding to the at least one video frame. The temporal model may include at least one of the following: a recurrent neural network (RNN) and a bidirectional recurrent neural network (Bi-RNN). The recurrent neural network may include a long short-term memory network (LSTM), etc. The bidirectional recurrent convolutional neural network model may include a bidirectional long short-term memory network (Bi-LSTM), etc. In addition, the long short-term memory network may include at least one of a basic long short-term memory network and an improved network for the basic long short-term memory network. The bidirectional long short-term memory network may include at least one of a basic bidirectional long short-term memory network and a deformed network for the basic bidirectional long short-term memory network.

[0064] According to the embodiments of the present disclosure, since the shots in the embodiments of the present disclosure are temporally ordered continuous semantic shots, the global visual features corresponding to at least one video frame are determined based on the global visual features corresponding to at least one video frame, and then the shot segmentation information of the video is determined based on the global visual features corresponding to at least one video frame. This enables the timing information to be combined in the process of determining the shot segmentation information, thereby improving the accuracy of the shot segmentation information and further improving the accuracy of the shot segmentation.

[0065] According to an embodiment of the present disclosure, obtaining temporal features corresponding to at least one video frame based on global visual features corresponding to at least one video frame may include: processing a global visual feature set corresponding to at least one video frame set based on a long-term dependency information learning strategy to obtain temporal features corresponding to the at least one video frame. The long-term dependency information learning strategy may include one of a unidirectional long-term dependency information learning strategy and a bidirectional long-term dependency information learning strategy. The video frame set is obtained by dividing at least one video frame. The video frames in two adjacent video frame sets are at least partially different.

[0066] According to an embodiment of the present disclosure, a long-term dependency information learning strategy can be used to determine the temporal features of a video frame. In the process of determining the temporal features of a video frame, the temporal features can reflect the information carried by long-term memory. A unidirectional long-term dependency learning strategy can be used to determine the temporal features of a video frame by forward processing the global visual features corresponding to at least one video frame. A bidirectional long-term dependency learning strategy can be used to determine the temporal features of a video frame by forward and reverse processing the global visual features corresponding to at least one video frame.

[0067] According to an embodiment of the present disclosure, by utilizing a long-term dependency information learning strategy to process a global visual feature set corresponding to at least one video frame set to obtain temporal features corresponding to the video frames, the long-term dependency information learning strategy can achieve long-term memory, thereby improving the accuracy of the temporal features.

[0068] According to an embodiment of the present disclosure, when the long-term dependency information learning strategy includes a unidirectional long-term dependency information learning strategy, processing a global visual feature set corresponding to at least one video frame set based on the long-term dependency information learning strategy to obtain a temporal feature corresponding to at least one video frame may include: performing forward long-term dependency information learning on the global visual feature set corresponding to at least one video frame set to obtain a temporal feature corresponding to at least one video frame.

[0069] According to an embodiment of the present disclosure, when the long-term dependency information learning strategy may include a bidirectional long-term dependency information learning strategy, processing a global visual feature set corresponding to at least one video frame set based on the long-term dependency information learning strategy to obtain a temporal feature corresponding to at least one video frame may include: performing forward long-term dependency information learning and reverse long-term dependency information learning on the global visual feature set corresponding to at least one video frame set to obtain a temporal feature corresponding to at least one video frame.

[0070] According to an embodiment of the present disclosure, forward long-term dependency information learning may refer to long-term dependency information learning using input information at the current moment and hidden state information at the previous moment. Reverse long-term dependency information learning may refer to long-term dependency information learning using hidden state information at the next moment and input information at the current moment.

[0071] According to an embodiment of the present disclosure, after obtaining shot segmentation information for a video, the video can be segmented based on the shot segmentation information to obtain video frames belonging to the same shot in the video. Shot segmentation may refer to an operation for determining shot segmentation locations in a video. At least one shot is obtained based on the video frames belonging to the same shot in the video. A shot may include at least one video frame.

[0072] According to the embodiments of the present disclosure, since the local visual features of video frames are obtained by extracting two-dimensional features from the video frames rather than using three-dimensional features, resource consumption is reduced and processing efficiency is improved. In addition, since the shot segmentation information is determined based on the global visual features of the video frames, and the global visual features of the video frames are obtained based on the local visual features of the video frames, the global visual features can represent the temporal information between video frames, thereby improving the accuracy of shot segmentation.

[0073] Reference below Figure 3 、 Figure 4A 、 Figure 4B 、 Figure 4C 、 Figure 4D 、 Figure 5A 、 Figure 5B 、 Figure 5C and Figure 5D , the shot segmentation method according to the embodiment of the present disclosure is further explained in conjunction with specific embodiments.

[0074] According to an embodiment of the present disclosure, operation S210 may include the following operations.

[0075] Performing two-dimensional convolution processing on at least one video frame in a video to obtain a local visual feature corresponding to the at least one video frame.

[0076] According to an embodiment of the present disclosure, after obtaining at least one video frame in a video, two-dimensional convolution processing can be performed on each video frame in the at least one video frame to obtain local visual features corresponding to each video frame in the at least one video frame. For example, a first deep learning model can be used to perform two-dimensional convolution processing on at least one video frame in a video to obtain local visual features corresponding to the at least one video frame. The model structure of the first deep learning model can be configured according to actual business needs and is not limited here. For example, the second deep learning model may include at least one model structure. The model structure may include at least one model substructure and a connection relationship between each model substructure. The model structure may be a structure obtained by connecting at least one model substructure based on the connection relationship between the model substructures. At least one model substructure included in the model structure may be a structure from at least one operation layer. For example, the model structure may be a structure obtained by connecting at least one model substructure from at least one operation layer based on the connection relationship between the model substructures. For example, at least one operation layer may include at least one of the following: input layer, convolution layer, hidden layer, transcription layer, pooling layer, depooling layer, deconvolution layer, feedforward neural network layer, attention layer, residual layer, fully connected layer, batch normalization layer, linear embedding (i.e., Linear Embedding) layer and nonlinear layer, etc.

[0077] For example, the second deep learning model may include at least one of the following: a deep learning model based on a second visual encoder and a deep learning model based on a second convolutional neural network. The second visual encoder may include an encoder of a visual transformer. The visual transformer may include at least one of the following: a vision transformer and a Swin transformer. The deep learning model based on the second convolutional neural network may include a deep learning model based on two-dimensional convolution. The deep learning model based on two-dimensional convolution may include at least one of the following: AlexNet, LeNet, VGG, D2-Net, R2D2, etc.

[0078] According to an embodiment of the present disclosure, since the local visual features corresponding to at least one video frame are obtained by performing two-dimensional convolution processing on at least one video frame in the video, the computational complexity is reduced, thereby reducing the consumption of computing resources and storage resources.

[0079] According to an embodiment of the present disclosure, operation S220 may include the following operations.

[0080] At least one video frame is divided into at least one video frame set. A global visual feature corresponding to the at least one video frame is determined based on a local visual feature set corresponding to the at least one video frame set.

[0081] According to an embodiment of the present disclosure, feature extraction can be performed on at least one video frame in a video to obtain feature information corresponding to the at least one video frame. Dividing the at least one video frame into at least one video frame set, and determining global visual features corresponding to the at least one video frame based on a local visual feature set corresponding to the at least one video frame set can include: performing feature extraction on the at least one video frame included in the video based on a sliding window to obtain feature information corresponding to the at least one video frame.

[0082] According to an embodiment of the present disclosure, at least one video frame corresponding to a sliding window at the qth moment can be determined from at least one video frame included in a video. Based on the at least one video frame corresponding to the sliding window at the qth moment, the at least one video frame can be divided to obtain at least one video frame set corresponding to the sliding window at the qth moment. Feature extraction is performed on the at least one video frame set corresponding to the sliding window at the qth moment to obtain feature information corresponding to the at least one video frame set corresponding to the sliding window at the qth moment. q can be an integer greater than or equal to 1 and less than or equal to Q. Q can be an integer greater than or equal to 1. Q can represent the number of moments. The video frames within the sliding window are sorted in chronological order. The number of video frames within the sliding window can be configured based on business needs and is not limited here. For example, the number of video frames within the sliding window can be less than or equal to a predetermined number. The predetermined number can be configured based on actual business needs and is not limited here. In addition, two adjacent sliding windows at least partially do not overlap.

[0083] For example, a video may include U video frames. For example, the 1st video frame, the 2nd video frame, ..., the nth video frame, ..., the U-1th video frame, and the Uth video frame. U may be an integer greater than or equal to 1. u may be an integer greater than or equal to 1 and less than or equal to U. The number of video frames within the sliding window is 3.

[0084] In the case where two adjacent sliding windows partially overlap, the video frames within the sliding window at the first moment may be the first to third video frames. In this case, the video frame set corresponding to the first moment may include the first to third video frames. The video frames within the sliding window at the second moment may be the second to fourth video frames. In this case, the video frame set corresponding to the second moment may include the second to fourth video frames.

[0085] In the case where two adjacent sliding windows do not overlap at all, the video frames within the sliding window at the first moment may be the first to third video frames, in which case the video frame set corresponding to the first moment may include the first to third video frames. The video frames within the sliding window at the second moment may be the fourth to sixth video frames, in which case the video frame set corresponding to the second moment may include the fourth to sixth video frames.

[0086] According to an embodiment of the present disclosure, since the global visual features corresponding to at least one video frame are determined based on the local visual feature set corresponding to at least one video frame set, and at least one video frame set is obtained by dividing at least one video frame, the shot segmentation information of the video is determined by using the global visual features corresponding to at least one video frame, which improves the accuracy of the shot segmentation information, thereby improving the accuracy of the shot segmentation.

[0087] According to an embodiment of the present disclosure, operation S230 may include the following operations.

[0088] According to the global visual feature set corresponding to the at least one video frame set, shot segmentation information corresponding to the at least one video frame set is determined. According to the shot segmentation information corresponding to the at least one video frame set, shot segmentation information of the video is determined.

[0089] According to an embodiment of the present disclosure, at least one video frame corresponding to a sliding window at time y can be determined from at least one video frame included in a video, and the at least one video frame can be divided based on the at least one video frame corresponding to the sliding window at time y to obtain at least one video frame set corresponding to the sliding window at time y. Shot segmentation information corresponding to the sliding window at time y can be determined based on the global visual feature sets of the at least one video frame set corresponding to the sliding window at time y.

[0090] According to an embodiment of the present disclosure, after obtaining the shot segmentation information of at least one video frame set corresponding to the sliding window at the yth moment, the shot segmentation information of at least one video frame set corresponding to the sliding window at the yth moment can be processed according to the sliding window to obtain the shot segmentation information of the video. According to the shot segmentation information, the shot segmentation position of the video is determined. According to the shot segmentation position, at least one shot is determined from the video. y can be an integer greater than or equal to 1 and less than or equal to Y. Y can be an integer greater than or equal to 1. Y can represent the number of moments. The video frames in the sliding window are sorted in chronological order. The number of video frames in the sliding window can be configured according to business needs and is not limited here. For example, the number of video frames in the sliding window can be less than or equal to a predetermined number. The predetermined number can be configured according to actual business needs and is not limited here. In addition, there is at least partial non-overlap between two adjacent sliding windows.

[0091] For example, a video may include X video frames. For example, the 1st video frame, the 2nd video frame, ..., the xth video frame, ..., the X-1th video frame, and the Xth video frame. X may be an integer greater than or equal to 1. x may be an integer greater than or equal to 1 and less than or equal to X. The number of video frames within the sliding window is 5.

[0092] When two adjacent sliding windows partially overlap, the video frames within the sliding window at the first moment may be the first to fifth video frames, the corresponding video frame set at the first moment may include the first to fifth video frames, and the shot segmentation information for the video frame set corresponding to the first moment may be before the third video frame. The video frames within the sliding window at the second moment may be the third to seventh video frames, the corresponding video frame set at the second moment may include the third to seventh video frames, and the shot segmentation information for the video frame set corresponding to the second moment may be before the fifth video frame. The video frames within the sliding window at the third moment may be the fifth to ninth video frames, the corresponding video frame set at the third moment may include the fifth to ninth video frames, and the shot segmentation information for the video frame set corresponding to the third moment may be before the seventh video frame. In this case, the shot segmentation information for the video may include before the fifth video frame.

[0093] If two adjacent sliding windows do not overlap at all, the video frames within the sliding window at the first moment may be the first to fifth video frames, the corresponding video frame set at the first moment may include the first to fifth video frames, and the shot segmentation information for the video frame set corresponding to the first moment may be before the third video frame. The video frames within the sliding window at the second moment may be the sixth to tenth video frames, the corresponding video frame set at the second moment may include the sixth to tenth video frames, and the shot segmentation information for the video frame set corresponding to the second moment may be before the eighth video frame. The video frames within the sliding window at the third moment may be the eleventh to fifteenth video frames, the corresponding video frame set at the third moment may include the eleventh to fifteenth video frames, and the shot segmentation information for the video frame set corresponding to the third moment may be before the thirteenth video frame. In this case, the shot segmentation information for the video may include before the third video frame, before the eighth video frame, and before the thirteenth video frame.

[0094] Figure 3 An example schematic diagram of a shot segmentation method according to an embodiment of the present disclosure is schematically shown.

[0095] like Figure 3 As shown in 300, video 301 may include 12 video frames, for example, video frame 301_1, video frame 301_2, video frame 301_3, video frame 301_4, video frame 301_5, video frame 301_6, video frame 301_7, video frame 301_8, video frame 301_9, video frame 301_10, video frame 301_11 and video frame 301_12.

[0096] Two-dimensional feature extraction may be performed on each of the 12 video frames in the video 301 to obtain local visual features 302 corresponding to each of the 12 video frames. For example, the local visual features 302 may include the local visual features 302_1 of the video frame 301_1, the local visual features 302_2 of the video frame 301_2, the local visual features 302_3 of the video frame 301_3, the local visual features 302_4 of the video frame 301_4, the local visual features 302_5 of the video frame 301_5, the local visual features 302_6 of the video frame 301_6, the local visual features 302_7 of the video frame 301_7, the local visual features 302_8 of the video frame 301_8, the local visual features 302_9 of the video frame 301_9, the local visual features 302_10 of the video frame 301_10, the local visual features 302_11 of the video frame 301_11, and the local visual features 302_12 of the video frame 301_12.

[0097] After obtaining the local visual features 302 corresponding to the 12 video frames, the global visual features 303 corresponding to the 12 video frames can be determined based on the local visual features corresponding to the 12 video frames. For example, the global visual features 303 may include the global visual features 303_1 of the video frame 301_1, the global visual features 303_2 of the video frame 301_2, the global visual features 303_3 of the video frame 301_3, the global visual features 303_4 of the video frame 301_4, the global visual features 303_5 of the video frame 301_5, the global visual features 303_6 of the video frame 301_6, the global visual features 303_7 of the video frame 301_7, the global visual features 303_8 of the video frame 301_8, the global visual features 303_9 of the video frame 301_9, the global visual features 303_10 of the video frame 301_10, the global visual features 303_11 of the video frame 301_11, and the global visual features 303_12 of the video frame 301_12.

[0098] After obtaining the global visual features 303 corresponding to each of the 12 video frames, the shot segmentation information 304 of the video 301 can be determined based on the global visual features 303 corresponding to each of the 12 video frames. The shot segmentation information 304 may include the shot segmentation information 304_1 of the video frame 301_1 being "0", the shot segmentation information 304_2 of the video frame 301_2 being "0", the shot segmentation information 304_3 of the video frame 301_3 being "0", the shot segmentation information 304_4 of the video frame 301_4 being "1", the shot segmentation information 304_5 of the video frame 301_5 being "0", and the shot segmentation information 304_6 of the video frame 301_6 being "1". "0", the lens segmentation information 304_7 of video frame 301_7 is "1", the lens segmentation information 304_8 of video frame 301_8 is "0", the lens segmentation information 304_9 of video frame 301_9 is "0", the lens segmentation information 304_10 of video frame 301_10 is "0", the lens segmentation information 304_11 of video frame 301_11 is "1" and the lens segmentation information 304_12 of video frame 301_12 is "1".

[0099] According to an embodiment of the present disclosure, determining a global visual feature corresponding to at least one video frame based on a local visual feature set corresponding to at least one video frame set may include the following operations.

[0100] Based on the global feature extraction strategy, the local visual feature set corresponding to the at least one video frame set is processed in M levels to obtain the global visual feature corresponding to the at least one video frame.

[0101] According to an embodiment of the present disclosure, the global feature extraction strategy may include at least one of a self-attention strategy and a similarity strategy. M may be an integer greater than or equal to 1. M may be configured according to actual business needs and is not limited here. For example, M = 4.

[0102] According to an embodiment of the present disclosure, a global feature extraction strategy may refer to a strategy for determining global visual features. The global feature extraction strategy may include at least one of the following: a self-attention strategy and a similarity strategy. The self-attention strategy may refer to a strategy for determining self-attention features. The self-attention strategy may include a multi-head attention strategy. The multi-head attention strategy may refer to a strategy for determining multi-head attention features. The similarity strategy may refer to a strategy for determining similarity features. The global visual features may include at least one of the following: a global implicit visual feature and a global explicit visual feature. The global implicit visual feature may be used to implicitly characterize the global visual feature. The global implicit visual feature may include a self-attention feature. The global explicit visual feature may be used to explicitly characterize the global visual feature. The global explicit visual feature may include a similarity feature.

[0103] According to embodiments of the present disclosure, the self-attention strategy can be used to focus on important information with high weights and ignore unimportant information with low weights. It can also exchange important information with other information by sharing it, thereby achieving the transmission of important information. In embodiments of the present disclosure, the self-attention strategy can extract information between at least one video frame in a video to better complete the video shot segmentation.

[0104] According to an embodiment of the present disclosure, performing M-level processing on a local visual feature set corresponding to at least one video frame set based on a global feature extraction strategy to obtain a global visual feature corresponding to the at least one video frame may include: performing M-level processing on a local visual feature set corresponding to the at least one video frame set based on a self-attention strategy to obtain a self-attention feature corresponding to the at least one video frame. Alternatively, performing M-level processing on a local visual feature set corresponding to the at least one video frame set based on a similarity strategy to obtain a similarity feature corresponding to the at least one video frame.

[0105] According to an embodiment of the present disclosure, a third deep learning model can be used to perform M-level processing on a local visual feature set corresponding to at least one video frame set to obtain a global visual feature corresponding to at least one video frame. The third deep learning model may include at least one of the following: a deep learning model based on a third visual encoder and a deep learning model based on a third convolutional neural network, etc. The third visual encoder may include an encoder of a visual transformer. The visual transformer may include at least one of the following: Vision Transformer and Swin Transformer, etc. The deep learning model based on the third convolutional neural network may include a deep learning model based on two-dimensional convolution. The deep learning model based on two-dimensional convolution may include at least one of the following: AlexNet, LeNet and VGG, D2-Net and R2D2, etc.

[0106] According to an embodiment of the present disclosure, since the global visual features corresponding to at least one video frame are obtained by processing the local visual feature set corresponding to at least one video frame set based on a global feature extraction strategy, and the global feature extraction strategy includes at least one of a self-attention strategy and a similarity strategy, the global visual features can represent deep visual features and shallow visual features, thereby improving the accuracy of shot segmentation information.

[0107] According to an embodiment of the present disclosure, when M is greater than 1, the local visual feature set corresponding to at least one video frame set is processed at M levels based on the global feature extraction strategy to obtain the global visual feature corresponding to at least one video frame, which may include the following operations.

[0108] When 1<m≤M, based on the first intermediate global visual feature set corresponding to at least one video frame set at the m-1th level, a second intermediate global visual feature set corresponding to at least one video frame set at the m-th level is obtained. Based on the second intermediate global visual feature set corresponding to at least one video frame set at the m-th level and the first intermediate global visual feature set corresponding to at least one video frame set at the m-1th level, a first intermediate global visual feature set corresponding to at least one video frame set at the m-th level is obtained. Based on the first intermediate global visual feature set corresponding to at least one video frame set at the R-th level, a global visual feature corresponding to at least one video frame is obtained.

[0109] According to an embodiment of the present disclosure, m may be an integer greater than or equal to 1 and less than or equal to M. R may be an integer greater than or equal to 1 and less than or equal to M. The second intermediate global visual feature set may include at least one second intermediate global visual feature. The second intermediate global visual feature may include at least one of an intermediate global implicit visual feature and an intermediate global explicit visual feature. The global feature extraction strategy may include at least one of a self-attention strategy and a similarity strategy.

[0110] For example, when the global feature extraction strategy includes a self-attention strategy, the first intermediate global visual feature set corresponding to at least one video frame set at the m-1th level can be processed based on the self-attention strategy to obtain an intermediate global implicit visual feature set corresponding to at least one video frame set at the mth level.

[0111] For example, when the global feature extraction strategy includes a similarity strategy, the first intermediate global visual feature set corresponding to at least one video frame set at the m-1th level can be processed based on the similarity strategy to obtain an intermediate global explicit visual feature set corresponding to at least one video frame set at the mth level.

[0112] For example, when the global feature extraction strategy includes a self-attention strategy and a similarity strategy, the intermediate global implicit visual feature set and the intermediate global explicit visual feature set corresponding to at least one video frame set at the mth level can be fused to obtain a first intermediate global visual feature set corresponding to at least one video frame set at the mth level.

[0113] According to the embodiments of the present disclosure, in the case of 1<m≤M, since the intermediate global explicit visual features can explicitly represent the global visual features of the nth video frame set in the N video frame sets, the intermediate global explicit visual features are used to determine the shot segmentation information of the video, thereby improving the accuracy of the shot segmentation information, thereby improving the accuracy of the shot segmentation. In addition, since the intermediate global implicit visual features can implicitly represent the nth video frame set in the nth video frame set, the intermediate global implicit visual features can implicitly represent the nth video frame set in the nth video frame set. tTherefore, the intermediate global implicit visual features are used to determine the shot segmentation information of the video, which improves the accuracy of the shot segmentation information and thus improves the accuracy of the shot segmentation.

[0114] Figure 4A The following schematically illustrates an example process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level when 1<m≤M according to an embodiment of the present disclosure.

[0115] like Figure 4A As shown, in 400A, a second intermediate global visual feature set 402 corresponding to at least one video frame set at the m-1th level can be obtained based on a first intermediate global visual feature set 401 corresponding to at least one video frame set at the m-1th level.

[0116] According to the second intermediate global visual feature set 402 of the mth level corresponding to at least one video frame set and the first intermediate global visual feature set 401 of the m-1th level corresponding to at least one video frame set, the first intermediate global visual feature set 403 of the mth level corresponding to at least one video frame set is obtained.

[0117] According to an embodiment of the present disclosure, at least one video frame set includes N, and the nth video frame set includes n T video frames, N is an integer greater than or equal to 1, and T is an integer greater than or equal to 1.

[0118] According to an embodiment of the present disclosure, when the second intermediate global visual features include intermediate global implicit visual features, obtaining the second intermediate global visual feature set of the mth level corresponding to at least one video frame set based on the first intermediate global visual feature set of the m-1th level corresponding to at least one video frame set may include the following operations.

[0119] For the nth video frame set in the N video frame sets, at least one first matrix set corresponding to the nth video frame set at the m-1th level is determined based on the first intermediate global visual feature set corresponding to the nth video frame set. t video frames, according to the mth level and the nth level t The first query matrix corresponding to the video frame and the first key matrix corresponding to the n-th video frame set at the m-th level are obtained. t The first attention matrix corresponding to the video frame. According to the m-th level and the n-th level t The first attention matrix corresponding to the video frame and the mth level and the nth level t The first value matrix corresponding to the video frame is obtained at the mth level and the nth level. tThe third intermediate global visual feature corresponding to the video frame. t The third intermediate global visual feature corresponding to the video frame is obtained at the mth level and the nth level. t The intermediate global implicit visual features corresponding to each video frame.

[0120] According to an embodiment of the present disclosure, the first matrix set may include a first query matrix, a first key matrix, and a first value matrix. t is an integer greater than or equal to 1 and less than or equal to T.

[0121] According to an embodiment of the present disclosure, the first intermediate global visual feature set can be used to determine the first key matrix, the first value matrix, and the first query matrix. For example, when the global feature extraction strategy is a self-attention strategy, the first intermediate global visual feature can be used as the first key matrix, the first value matrix, and the first query matrix. The key (i.e., Key) matrix, the value (i.e., Value), and the query (i.e., Query) matrix can be matrices in the attention mechanism.

[0122] According to an embodiment of the present disclosure, when the global feature extraction strategy is a self-attention strategy, the first intermediate global visual features corresponding to at least one video frame set and used as the first key matrix, the first value matrix, and the first query matrix can be processed based on the self-attention strategy to obtain intermediate global implicit visual features corresponding to each of the at least one video frame set. For example, an attention unit can be determined based on the self-attention strategy. The first intermediate global visual features corresponding to the at least one video frame set and used as the first key matrix, the first value matrix, and the first query matrix are processed using the attention unit to obtain intermediate global implicit visual features corresponding to each of the at least one video frame set.

[0123] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global implicit visual feature and 1<m≤M, for the nth video frame set among the N video frame sets, the first intermediate global visual feature set corresponding to the nth video frame set at the m-1th level can be processed based on the self-attention strategy to obtain at least one first matrix set corresponding to the nth video frame set at the mth level. The first matrix set includes a first query matrix, a first key matrix, and a first value matrix.

[0124] According to an embodiment of the present disclosure, after obtaining at least one first matrix set corresponding to the nth video frame set at the mth level, for the nth video frame set t video frames, the mth and nth levels can be processed based on the self-attention strategy. t The first query matrix corresponding to the nth video frame and the first key matrix corresponding to the nth video frame set are obtained. tThe first attention matrix corresponding to the video frame. According to the m-th level and the n-th level t The first attention matrix and the first value matrix corresponding to the video frame are obtained at the mth level and the nth level. t The intermediate global implicit visual features corresponding to each video frame.

[0125] According to an embodiment of the present disclosure, when 1<m≤M, since the intermediate global implicit visual features are determined based on the first query matrix, the first key matrix and the first value matrix, the intermediate global implicit visual features can implicitly represent the global visual features between at least one video frame in the video frame set. Therefore, the intermediate global implicit visual features are used to determine the shot segmentation information, thereby improving the accuracy of the shot segmentation information.

[0126] Figure 4B The following schematically illustrates an example process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level when 1<m≤M according to another embodiment of the present disclosure.

[0127] like Figure 4B As shown, in 400B, at least one first matrix set 402 corresponding to the nth video frame set at the mth level can be determined based on the first intermediate global visual feature set 404 corresponding to the nth video frame set at the m-1th level.

[0128] For the nth video frame set t video frames, the first matrix set 405 includes the mth level and the nth level t The first query matrix 405_1 corresponding to the video frame, the m-th level and the n-th level t The first key matrix 405_2 corresponding to the video frame and the m-th level and the n-th level t The first value matrix 405_3 corresponding to the video frames.

[0129] According to the mth level and the nth t The first query matrix 405_1 corresponding to the nth video frame and the first key matrix 405_2 corresponding to the nth video frame set are obtained. t The first attention matrix 406 corresponding to the video frames.

[0130] According to the mth level and the nth t The first attention matrix 406 and the first value matrix 405_3 corresponding to the video frame are obtained at the mth level and the nth level. t The third intermediate global visual feature 407 corresponding to the video frame.

[0131] According to the mth level and the nth tThe third intermediate global visual feature 407 corresponding to the video frame is obtained at the mth level and the nth level. t The intermediate global implicit visual features 408 corresponding to the video frames.

[0132] According to the first intermediate global visual feature set 404 of the m-1th level corresponding to the nth video frame set and the mth level corresponding to the nth t The intermediate global implicit visual features 408 corresponding to the video frames are obtained to obtain a first intermediate global visual feature set 409 corresponding to at least one video frame set at the mth level.

[0133] According to an embodiment of the present disclosure, when the second intermediate global visual features include intermediate global explicit visual features, obtaining the second intermediate global visual feature set of the mth level corresponding to at least one video frame set based on the first intermediate global visual feature set of the m-1th level corresponding to at least one video frame set may include the following operations.

[0134] For the n-th video frame set among the N video frame sets, the similarity between the video frames in the n-th video frame set at the m-1-th level is determined based on the first intermediate global visual feature set corresponding to the n-th video frame set. Based on the similarity between the video frames in the n-th video frame set at the m-th level, the intermediate global explicit visual feature set corresponding to the n-th video frame set at the m-th level is obtained.

[0135] According to an embodiment of the present disclosure, when the global feature extraction strategy is a similarity strategy, the first intermediate global visual feature set corresponding to at least one video frame set can be processed based on the similarity strategy to obtain an intermediate global explicit visual feature set corresponding to each of the at least one video frame set.

[0136] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global explicit visual feature and 1<m≤M, for the nth video frame set among N video frame sets, the first intermediate global visual feature set corresponding to the nth video frame set at the m-1th level can be processed based on the similarity strategy to obtain the similarity between the video frames in the nth video frame set at the mth level. After obtaining the similarity between the video frames in the nth video frame set at the mth level, the intermediate global explicit visual feature set corresponding to the nth video frame set at the mth level can be obtained based on the similarity between the video frames in the nth video frame set at the mth level.

[0137] According to an embodiment of the present disclosure, similarity can be used to characterize the degree of similarity between two objects to be matched. The relationship between the numerical value of similarity and the degree of similarity can be configured according to actual business needs and is not limited here. For example, the larger the numerical value of similarity, the greater the degree of similarity between the two objects to be matched. Conversely, the smaller the degree of similarity between the two objects to be matched. Alternatively, the smaller the numerical value of similarity, the greater the degree of similarity between the two objects to be matched. Conversely, the smaller the degree of similarity between the two objects to be matched. Similarity can be configured according to actual business needs and is not limited here. For example, similarity can include at least one of the following: a method based on literal similarity, a method based on text similarity, and a method based on entity similarity, etc. The method based on literal similarity can include at least one of the following: edit distance, Dice coefficient, and Jaccard similarity, etc. The method based on text similarity can include at least one of the following: cosine similarity, relative entropy, KL (Kullback-Leibler, KL) divergence, and probability model similarity, etc.

[0138] According to an embodiment of the present disclosure, the two objects to be matched may refer to any two video frames in at least one video frame included in the video frame set. For a video frame in at least one video frame included in the video frame set, the first similarity between the video frame and other video frames may be determined to obtain at least one first similarity corresponding to the video frame. Based on the at least one first similarity corresponding to the video frame, an intermediate global explicit visual feature set corresponding to the video frame set is obtained. For example, a weighted summation may be performed on the at least one first similarity corresponding to the video frame to obtain weighted information corresponding to the video frame. The weighted information corresponding to the video frame is determined as the intermediate global explicit visual feature corresponding to the video frame. The other video frames in the video frame set may refer to at least one video frame other than the video frame in at least one video frame.

[0139] According to an embodiment of the present disclosure, when 1<m≤M, since the intermediate global explicit visual feature is determined based on the similarity between the global visual features of the nth video frame set in N video frame sets, the intermediate global explicit visual feature can explicitly characterize the global visual features between at least one video frame in the video frame set. Therefore, the intermediate global explicit visual feature is used to determine the shot segmentation information of the video, thereby improving the accuracy of the shot segmentation information.

[0140] Figure 4C The following schematically illustrates an example process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level when 1<m≤M according to another embodiment of the present disclosure.

[0141] like Figure 4CAs shown, in 400C, the similarity 411 between the video frames in the nth video frame set at the mth level is determined based on the first intermediate global visual feature set 410 at the m-1th level corresponding to the nth video frame set.

[0142] According to the similarity 411 between the video frames in the n-th video frame set of the m-th level, an intermediate global explicit visual feature set 412 corresponding to the n-th video frame set of the m-th level is obtained.

[0143] According to the first intermediate global visual feature set 410 of the m-1th level corresponding to the nth video frame set and the intermediate global explicit visual feature set 412 of the mth level corresponding to the nth video frame set, a first intermediate global visual feature set 413 of the mth level corresponding to at least one video frame set is determined.

[0144] According to an embodiment of the present disclosure, when the second intermediate global visual features include intermediate global implicit visual features and intermediate global explicit visual features, the following operations may be included based on the second intermediate global visual feature set of the mth level corresponding to at least one video frame set and the first intermediate global visual feature set of the m-1th level corresponding to at least one video frame set.

[0145] The intermediate global implicit visual feature set and the intermediate global explicit visual feature set of the mth level corresponding to the at least one video frame set are fused to obtain a fourth intermediate global visual feature set of the mth level corresponding to the at least one video frame set. The first intermediate global visual feature set of the mth level corresponding to the at least one video frame set is obtained based on the fourth intermediate global visual feature set of the mth level corresponding to the at least one video frame set and the first intermediate global visual feature set of the m-1th level corresponding to the at least one video frame set.

[0146] According to an embodiment of the present disclosure, when the global feature extraction strategy is a self-attention strategy and a similarity strategy, the first intermediate global visual features corresponding to at least one video frame set and used as a first key matrix, a first value matrix, and a first query matrix can be processed based on the self-attention strategy to obtain intermediate global implicit visual features corresponding to each of the at least one video frame set. The first intermediate global visual feature set corresponding to at least one video frame set can be processed based on the similarity strategy to obtain an intermediate global explicit visual feature set corresponding to each of the at least one video frame set.

[0147] According to an embodiment of the present disclosure, when the second intermediate global visual features include intermediate global implicit visual features and intermediate global explicit visual features and 1<m≤M, the intermediate global implicit visual feature set and the intermediate global explicit visual feature set corresponding to at least one video frame set at the mth level can be fused to obtain a fourth intermediate global visual feature set corresponding to at least one video frame set at the mth level. The fusion can include one of the following: addition and concatenation.

[0148] According to an embodiment of the present disclosure, after obtaining the fourth intermediate global visual feature set corresponding to at least one video frame set at the mth level, the fourth intermediate global visual feature set corresponding to at least one video frame set at the mth level and the first intermediate global visual feature set corresponding to at least one video frame set at the m-1th level can be processed using a fourth deep learning model to obtain the first intermediate global visual feature set corresponding to at least one video frame set at the mth level. The fourth deep learning model may include at least one of the following: a deep learning model based on a fourth visual encoder and a deep learning model based on a fourth convolutional neural network. The fourth visual encoder may include an encoder of a visual transformer. The visual transformer may include at least one of the following: Vision Transformer and Swin Transformer. The deep learning model based on the fourth convolutional neural network may include a deep learning model based on two-dimensional convolution. The deep learning model based on two-dimensional convolution may include at least one of the following: AlexNet, LeNet and VGG, D2-Net and R2D2, etc. According to an embodiment of the present disclosure, since the shot segmentation information is determined based on the fourth intermediate global visual feature set, and the fourth intermediate global visual feature set is determined based on the intermediate global implicit visual feature set and the intermediate global explicit visual feature set, the information carried by the fourth intermediate global visual feature set is more comprehensive, thereby improving the accuracy of the shot segmentation information, and further improving the accuracy of the shot segmentation.

[0149] Figure 4D The following schematically illustrates an example process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the mth level when 1<m≤M according to another embodiment of the present disclosure.

[0150] like Figure 4DAs shown in 400D, a second intermediate global visual feature set 415 corresponding to at least one video frame set at the m-1th level can be obtained based on the first intermediate global visual feature set 414 corresponding to at least one video frame set at the m-1th level. The second intermediate global visual feature set 415 corresponding to at least one video frame set at the mth level can include an intermediate global implicit visual feature set 4151 corresponding to at least one video frame set at the mth level and an intermediate global explicit visual feature set 415_2 corresponding to at least one video frame set at the mth level.

[0151] The intermediate global implicit visual feature set 415_1 and the intermediate global explicit visual feature set 415_2 of the mth level corresponding to at least one video frame set are fused to obtain a fourth intermediate global visual feature set 416 of the mth level corresponding to at least one video frame set.

[0152] Based on the first intermediate global visual feature set 414 of the m-1th level corresponding to at least one video frame set and the fourth intermediate global visual feature set 416 of the mth level corresponding to at least one video frame set, the first intermediate global visual feature set 417 of the mth level corresponding to at least one video frame set is determined.

[0153] According to an embodiment of the present disclosure, performing M-level processing on a local visual feature set corresponding to at least one video frame set based on a global feature extraction strategy to obtain a global visual feature corresponding to at least one video frame may also include the following operations.

[0154] In the case of m=1, a second intermediate global visual feature set corresponding to at least one video frame set at the second level is obtained based on the local visual feature set corresponding to at least one video frame set at the first level. A first intermediate global visual feature set corresponding to at least one video frame set at the second level is obtained based on the second intermediate global visual feature set corresponding to at least one video frame set at the second level and the local visual feature set corresponding to at least one video frame set at the first level.

[0155] According to an embodiment of the present disclosure, the second intermediate global visual feature set includes at least one second intermediate global visual feature. The second intermediate global visual feature includes at least one of an intermediate global implicit visual feature and an intermediate global explicit visual feature. The global feature extraction strategy includes at least one of a self-attention strategy and a similarity strategy.

[0156] For example, when m=1 and the global feature extraction strategy includes a self-attention strategy, the local visual feature set of the first level corresponding to the at least one video frame set can be processed based on the self-attention strategy to obtain an intermediate global implicit visual feature set of the second level corresponding to the at least one video frame set.

[0157] For example, when m=1 and the global feature extraction strategy includes a similarity strategy, the local visual feature set of the first level corresponding to the at least one video frame set can be processed based on the similarity strategy to obtain an intermediate global explicit visual feature set of the second level corresponding to the at least one video frame set.

[0158] For example, when the m=1 global feature extraction strategy includes a self-attention strategy and a similarity strategy, the local visual feature set corresponding to at least one video frame set of the first level can be processed based on the self-attention strategy to obtain an intermediate global implicit visual feature set corresponding to at least one video frame set of the second level. The local visual feature set corresponding to at least one video frame set of the first level is processed based on the similarity strategy to obtain an intermediate global explicit visual feature set corresponding to at least one video frame set of the second level. The intermediate global implicit visual feature set corresponding to at least one video frame set of the second level and the intermediate global explicit visual feature set are fused to obtain a first intermediate global visual feature set corresponding to at least one video frame set of the second level.

[0159] According to an embodiment of the present disclosure, when m=1, since the intermediate global explicit visual features can explicitly represent the global visual features of at least one video frame set at the second level, the intermediate global explicit visual features are used to determine the shot segmentation information of the video, thereby improving the accuracy of the shot segmentation information, thereby improving the accuracy of the shot segmentation. In addition, since the intermediate global implicit visual features can implicitly represent the global visual features of at least one video frame set at the second level, the intermediate global implicit visual features are used to determine the shot segmentation information of the video, thereby improving the accuracy of the shot segmentation information, thereby improving the accuracy of the shot segmentation.

[0160] According to an embodiment of the present disclosure, when M=1, performing M-level processing on the local visual feature set corresponding to at least one video frame set based on the global feature extraction strategy to obtain the global visual features corresponding to at least one video frame may include: obtaining the second intermediate global visual feature set corresponding to at least one video frame set at the second level according to the local visual feature set corresponding to at least one video frame set at the first level. Obtaining the first intermediate global visual feature set corresponding to at least one video frame set at the second level according to the second intermediate global visual feature set corresponding to at least one video frame set at the second level and the local visual feature set corresponding to at least one video frame set at the first level. Obtaining the global visual features corresponding to at least one video frame according to the first intermediate global visual feature set corresponding to at least one video frame set at the second level.

[0161] Figure 5AThe figure schematically shows an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the second level when m=1 according to an embodiment of the present disclosure.

[0162] like Figure 5A As shown, in 500A, a second intermediate global visual feature set 502 corresponding to the at least one video frame set at the second level can be obtained based on the local visual feature set 501 corresponding to the at least one video frame set at the first level.

[0163] According to the second intermediate global visual feature set 502 of the second level corresponding to the at least one video frame set and the local visual feature set 501 of the first level corresponding to the at least one video frame set, a first intermediate global visual feature set 503 of the second level corresponding to the at least one video frame set is obtained.

[0164] According to an embodiment of the present disclosure, at least one video frame set may include N. The nth video frame set may include n T video frames. N may be an integer greater than or equal to 1. T may be an integer greater than or equal to 1.

[0165] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global implicit visual feature, obtaining a second intermediate global visual feature set corresponding to at least one video frame set at the second level based on a local visual feature set corresponding to at least one video frame set at the first level may include the following operations.

[0166] For the nth video frame set in the N video frame sets, at least one second matrix set corresponding to the nth video frame set at the second level is determined based on the local visual feature set corresponding to the nth video frame set at the first level. t video frames, according to the second level and the nth t The second query matrix corresponding to the nth video frame and the second key matrix corresponding to the nth video frame set at the second level are obtained. t The second attention matrix corresponding to the video frame. According to the second level and the nth t The second attention matrix corresponding to the video frame and the second level and the nth level t The second value matrix corresponding to the video frame is obtained at the second level and the nth level. t The third intermediate global visual feature corresponding to the nth video frame. t The third intermediate global visual feature corresponding to the video frame is obtained at the second level and the nth level. t The intermediate global implicit visual features corresponding to each video frame.

[0167] According to an embodiment of the present disclosure, t may be an integer greater than or equal to 1 and less than or equal to T. The second matrix set may include a second query matrix, a second key matrix, and a second value matrix.

[0168] According to an embodiment of the present disclosure, the local visual feature set can be used to determine a second matrix set. The second matrix set may include a second key matrix, a second value matrix, and a second query matrix. For example, when the global feature extraction strategy is a self-attention strategy, the local visual feature set can be used as the second key matrix, the second value matrix, and the second query matrix.

[0169] According to an embodiment of the present disclosure, when the global feature extraction strategy is a self-attention strategy, local visual features corresponding to at least one video frame set and used as a second key matrix, a second value matrix, and a second query matrix can be processed based on the self-attention strategy to obtain intermediate global implicit visual features corresponding to each of the at least one video frame set. For example, an attention unit can be determined based on the self-attention strategy. The attention unit is used to process local visual features corresponding to the at least one video frame set and used as a second key matrix, a second value matrix, and a second query matrix to obtain intermediate global implicit visual features corresponding to each of the at least one video frame set.

[0170] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global implicit visual feature and m=1, for the nth video frame set among the N video frame sets, a first-level local visual feature set corresponding to the nth video frame set can be processed based on a self-attention strategy to obtain at least one second matrix set corresponding to the nth video frame set at the second level. The two matrix sets include a second query matrix, a second key matrix, and a second value matrix.

[0171] According to an embodiment of the present disclosure, after obtaining at least one second matrix set corresponding to the nth video frame set at the second level, for the nth video frame set, t video frames, the second layer and the nth layer can be processed based on the self-attention strategy t The second query matrix corresponding to the nth video frame and the second key matrix corresponding to the nth video frame set are obtained. t The second attention matrix corresponding to the video frame. According to the second level and the nth t The second attention matrix and the second value matrix corresponding to the video frame are obtained at the second level and the nth level. t The intermediate global implicit visual features corresponding to each video frame.

[0172] According to an embodiment of the present disclosure, when m=1, since the intermediate global implicit visual features are determined based on the second query matrix, the second key matrix and the second value matrix, the intermediate global implicit visual features can implicitly characterize the global visual features between at least one video frame in the video frame set. Thus, the intermediate global implicit visual features are used to determine the shot segmentation information, thereby improving the accuracy of the shot segmentation information.

[0173] Figure 5B The figure schematically shows an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the second level when m=1 according to another embodiment of the present disclosure.

[0174] like Figure 5B As shown, in 500B, at least one second matrix set 505 corresponding to the nth video frame set at the second level can be determined based on the local visual feature set 504 corresponding to the nth video frame set at the first level.

[0175] For the nth video frame set t video frames, the second matrix set 505 includes at least one second query matrix 505_1 of the second level corresponding to the n-th video frame set, at least one second key matrix 505_2 of the second level corresponding to the n-th video frame set, and at least one second value matrix 505_3 of the second level corresponding to the n-th video frame set.

[0176] According to at least one second query matrix 502_1 of the second level corresponding to the n-th video frame set and at least one second key matrix 502_2 of the second level corresponding to the n-th video frame set, the second level corresponding to the n-th video frame set is obtained. t The second attention matrix 506 corresponding to the video frames.

[0177] According to the second level and the nth t The second attention matrix 506 and the first value matrix 505_3 corresponding to the video frame are obtained at the second level and the nth level. t The third intermediate global visual feature 507 corresponding to the video frame.

[0178] According to the second level and the nth t The third intermediate global visual feature 507 corresponding to the video frame is obtained at the second level and the nth level. t The intermediate global implicit visual features 508 corresponding to the video frames.

[0179] According to the local visual feature set 504 corresponding to the n-th video frame set at the first level and the local visual feature set 504 corresponding to the n-th video frame set at the second level tThe intermediate global implicit visual features 508 corresponding to the video frames are used to determine a first intermediate global visual feature set 509 corresponding to the at least one video frame set at the second level.

[0180] According to an embodiment of the present disclosure, when the second intermediate global visual features include intermediate global explicit visual features, obtaining a second intermediate global visual feature set corresponding to at least one video frame set at the second level based on a local visual feature set corresponding to at least one video frame set at the first level may include the following operations.

[0181] For the nth video frame set among the N video frame sets, the similarity between the video frames in the nth video frame set at the second level is determined based on the local visual feature set corresponding to the nth video frame set at the first level. Based on the similarity between the video frames in the nth video frame set at the second level, the intermediate global explicit visual feature set corresponding to the nth video frame set at the mth level is obtained.

[0182] According to an embodiment of the present disclosure, when the global feature extraction strategy is a similarity strategy, the local visual feature set corresponding to at least one video frame set can be processed based on the similarity strategy to obtain an intermediate global explicit visual feature set corresponding to each of the at least one video frame sets.

[0183] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global explicit visual feature and m=1, for the nth video frame set among N video frame sets, the local visual feature set corresponding to the nth video frame set at the first level can be processed based on the similarity strategy to obtain the similarity between the video frames in the nth video frame set at the second level. After obtaining the similarity between the video frames in the nth video frame set at the second level, the intermediate global explicit visual feature set corresponding to the nth video frame set at the second level can be obtained based on the similarity between the video frames in the nth video frame set at the second level.

[0184] According to an embodiment of the present disclosure, similarity can be used to characterize the degree of similarity between two objects to be matched. The relationship between the numerical value of the similarity and the degree of similarity can be configured according to actual business needs and is not limited here. The two objects to be matched can refer to any two video frames in at least one video frame included in the video frame set. For a video frame in at least one video frame included in the video frame set, a second similarity between the video frame and other video frames can be determined to obtain at least one second similarity corresponding to the video frame. Based on the at least one second similarity corresponding to the video frame, an intermediate global explicit visual feature set corresponding to the video frame set is obtained. For example, a weighted summation can be performed on the at least one second similarity corresponding to the video frame to obtain weighted information corresponding to the video frame. The weighted information corresponding to the video frame is determined as the intermediate global explicit visual feature corresponding to the video frame. Other video frames in the video frame set can refer to at least one video frame other than the video frame in at least one video frame.

[0185] According to an embodiment of the present disclosure, when m=1, since the intermediate global explicit visual feature is determined based on the similarity between the local visual features of at least one video frame set of the second level, the intermediate global explicit visual feature can explicitly characterize the global visual features between at least one video frame in the video frame set. Thus, the intermediate global explicit visual feature is used to determine the shot segmentation information of the video, thereby improving the accuracy of the shot segmentation information.

[0186] Figure 5C The figure schematically shows an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the second level when m=1 according to another embodiment of the present disclosure.

[0187] like Figure 5C As shown, in 500C, the similarity 511 between the video frames in the n-th video frame set of the second level is determined based on the local visual feature set 510 corresponding to the n-th video frame set of the first level.

[0188] According to the similarity 511 between the video frames in the n-th video frame set of the second level, an intermediate global explicit visual feature set 512 corresponding to the n-th video frame set of the second level is obtained.

[0189] According to the local visual feature set 510 corresponding to the nth video frame set of the first level and the intermediate global explicit visual feature set 512 corresponding to the nth video frame set of the second level, a first intermediate global visual feature set 513 corresponding to at least one video frame set of the second level is determined.

[0190] According to an embodiment of the present disclosure, when the second intermediate global visual features include intermediate global implicit visual features and intermediate global explicit visual features, obtaining the first intermediate global visual feature set of the second level corresponding to at least one video frame set based on the second intermediate global visual feature set of the second level corresponding to at least one video frame set and the local visual feature set of the first level corresponding to at least one video frame set may include the following operations.

[0191] The intermediate global implicit visual feature set and the intermediate global explicit visual feature set of the second level corresponding to the at least one video frame set are fused to obtain a fourth intermediate global visual feature set of the second level corresponding to the at least one video frame set. A first intermediate global visual feature set of the mth level corresponding to the at least one video frame set is obtained based on the fourth intermediate global visual feature set of the second level corresponding to the at least one video frame set and the local visual feature set of the first level corresponding to the at least one video frame set.

[0192] According to an embodiment of the present disclosure, when the global feature extraction strategy is a self-attention strategy and a similarity strategy, local visual features corresponding to at least one video frame set and used as a second key matrix, a second value matrix, and a second query matrix can be processed based on the self-attention strategy to obtain intermediate global implicit visual features corresponding to each of the at least one video frame set. Local visual features corresponding to at least one video frame set can be processed based on the similarity strategy to obtain intermediate global explicit visual feature sets corresponding to each of the at least one video frame set.

[0193] According to an embodiment of the present disclosure, when the second intermediate global visual features include intermediate global implicit visual features and intermediate global explicit visual features and m=1, the intermediate global implicit visual feature set and the intermediate global explicit visual feature set corresponding to at least one video frame set at the second level can be fused to obtain a fourth intermediate global visual feature set corresponding to the at least one video frame set at the second level. The fusion can include one of the following: addition and splicing.

[0194] According to an embodiment of the present disclosure, since the shot segmentation information is determined based on the fourth intermediate global visual feature set, and the fourth intermediate global visual feature set is determined based on the intermediate global implicit visual feature set and the intermediate global explicit visual feature set, the information carried by the fourth intermediate global visual feature set is more comprehensive, thereby improving the accuracy of the shot segmentation information, and further improving the accuracy of the shot segmentation.

[0195] Figure 5D The figure schematically shows an example of a process of obtaining a first intermediate global visual feature set corresponding to at least one video frame set at the second level when m=1 according to another embodiment of the present disclosure.

[0196] like Figure 5D As shown in 500D, a second intermediate global visual feature set 515 corresponding to at least one video frame set at a second level can be obtained based on a first intermediate global visual feature set 514 corresponding to at least one video frame set at a first level. The second intermediate global visual feature set 515 corresponding to at least one video frame set at a second level can include an intermediate global implicit visual feature set 515_1 corresponding to at least one video frame set at a second level and an intermediate global explicit visual feature set 515_2 corresponding to at least one video frame set at a second level.

[0197] The intermediate global implicit visual feature set 515_1 and the intermediate global explicit visual feature set 5152 of the second level corresponding to at least one video frame set are fused to obtain a fourth intermediate global visual feature set 516 of the second level corresponding to at least one video frame set.

[0198] According to the first intermediate global visual feature set 515 corresponding to at least one video frame set at the first level and the fourth intermediate global visual feature set 516 corresponding to at least one video frame set at the second level, the first intermediate global visual feature set 513 corresponding to at least one video frame set at the second level is determined.

[0199] The above are merely exemplary embodiments, but are not limited thereto. Other shot segmentation methods known in the art may also be included as long as the accuracy of shot segmentation can be improved.

[0200] Figure 6 The flowchart of the visual task processing method according to an embodiment of the present disclosure is schematically shown.

[0201] like Figure 6 As shown, the method 600 includes operations S610 to S620.

[0202] In operation S610 , shot segmentation information of a video to be processed is determined.

[0203] In operation S620 , a visual task is processed according to shot segmentation information of the video to be processed.

[0204] According to an embodiment of the present disclosure, the shot segmentation information of the video to be processed may be determined using the shot segmentation method according to an embodiment of the present disclosure.

[0205] According to embodiments of the present disclosure, a shot segmentation method can be used to determine shot segmentation information for a video to be processed. Based on the shot segmentation information, the video to be processed is shot segmented to obtain at least one target shot. A visual task is then processed based on the at least one target shot to obtain visual task processing information.

[0206] According to the embodiments of the present disclosure, since the shot segmentation information is obtained by processing the video to be processed according to the shot segmentation method described in the embodiments of the present disclosure, and the shot segmentation method of the embodiments of the present disclosure can improve the accuracy of shot segmentation, therefore, processing the visual task according to the shot segmentation information of the video to be processed can improve the accuracy of the visual task processing information of the visual task.

[0207] Reference below Figure 7 , the shot segmentation method according to the embodiment of the present disclosure is further explained in conjunction with specific embodiments.

[0208] According to an embodiment of the present disclosure, the visual task includes at least one of the following: a video scene recognition task, a video frame deduplication task, and a shot extraction task.

[0209] According to an embodiment of the present disclosure, for a video scene recognition task, the video to be processed can be segmented based on the shot segmentation information of the video to be processed to obtain at least one first target shot. Based on the at least one first target shot, scene segmentation information corresponding to each of the at least one first target shot can be determined. Based on the scene segmentation information corresponding to each of the at least one first target shot, a target video scene can be determined.

[0210] According to an embodiment of the present disclosure, for a video frame deduplication task, the video to be processed can be segmented based on the shot segmentation information of the video to be processed to obtain at least one second target shot. Based on the at least one second target shot, video frame deduplication is performed on a set of video frames corresponding to each of the at least one second target shot to obtain deduplication information of the video frames corresponding to each of the at least one second target shot.

[0211] According to an embodiment of the present disclosure, for a shot extraction task, the video to be processed can be shot segmented based on the shot segmentation information of the video to be processed to obtain at least one third target shot. Based on the at least one third target shot, shot extraction can be performed on the at least one third target shot to obtain shot extraction information.

[0212] Figure 7 An example schematic diagram of a visual task processing method according to an embodiment of the present disclosure is schematically shown.

[0213] like Figure 7 As shown in 700, taking the video scene recognition task as an example, the video to be processed 701 may include 10 video frames, for example, video frame 7011, video frame 7012, video frame 7013, video frame 7014, video frame 7015, video frame 7016, video frame 7017, video frame 7018, video frame 7019, and video frame 70110.

[0214] Two-dimensional feature extraction can be performed on each of the 10 video frames in the processed video 301 to obtain local visual features corresponding to each of the 10 video frames. Based on the local visual features corresponding to each of the 10 video frames, global visual features corresponding to each of the 10 video frames are determined. Based on the global visual features corresponding to each of the 10 video frames, shot segmentation information 702 of video 701 is determined. Shot segmentation information 702 may include shot segmentation information 7021 of video frame 7011, shot segmentation information 7022 of video frame 7012, shot segmentation information 7023 of video frame 7013, shot segmentation information 7024 of video frame 7014, shot segmentation information 7025 of video frame 7015, shot segmentation information 7026 of video frame 7016, shot segmentation information 7027 of video frame 7017, shot segmentation information 7028 of video frame 7018, shot segmentation information 7029 of video frame 7019, and shot segmentation information 70210 of video frame 70110.

[0215] Based on the shot segmentation information 702 of the video to be processed, the video to be processed 301 can be segmented to obtain a first target shot 7031 and a first target shot 7032. Based on the first target shot 7031 and the first target shot 7032, the scene segmentation information 704 corresponding to each of the first target shot 7031 and the first target shot 7032 can be determined. For example, the scene segmentation information 7041 corresponding to the first target shot 7031 and the scene segmentation information 7042 corresponding to the first target shot 7032 can be determined. Based on the scene segmentation information 7041 and the scene segmentation information 7042, the target video scene can be determined.

[0216] The above are merely exemplary embodiments, but are not limited thereto. Other visual task processing methods known in the art may also be included, as long as the accuracy of visual task processing information of the visual task can be improved.

[0217] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0218] Figure 8 The block diagram of the shot splitting device according to an embodiment of the present disclosure is schematically shown.

[0219] like Figure 8 As shown, the shot segmentation device 800 may include a two-dimensional feature extraction module 810 , a first determination module 820 and a second determination module 830 .

[0220] The two-dimensional feature extraction module 810 is configured to perform two-dimensional feature extraction on at least one video frame in a video to obtain a local visual feature corresponding to the at least one video frame.

[0221] The first determining module 820 is configured to determine a global visual feature corresponding to at least one video frame based on a local visual feature corresponding to at least one video frame.

[0222] The second determining module 830 is configured to determine shot segmentation information of the video based on global visual features corresponding to at least one video frame.

[0223] According to an embodiment of the present disclosure, the two-dimensional feature extraction module 810 may include a two-dimensional convolution processing submodule.

[0224] The two-dimensional convolution processing submodule is used to perform two-dimensional convolution processing on at least one video frame in the video to obtain local visual features corresponding to the at least one video frame.

[0225] According to an embodiment of the present disclosure, the first determining module may include a dividing submodule and a first determining submodule.

[0226] The dividing submodule is configured to divide at least one video frame into at least one video frame set. The first determining submodule is configured to determine a global visual feature corresponding to at least one video frame based on a local visual feature set corresponding to at least one video frame set.

[0227] According to an embodiment of the present disclosure, the first determining submodule may include a first processing unit.

[0228] The first processing unit performs M-level processing on the local visual feature set corresponding to the at least one video frame set based on a global feature extraction strategy to obtain a global visual feature corresponding to the at least one video frame. The global feature extraction strategy includes at least one of a self-attention strategy and a similarity strategy. M is an integer greater than or equal to 1.

[0229] According to an embodiment of the present disclosure, when M is greater than 1, the first processing unit may include a first obtaining subunit, a second obtaining subunit, and a third obtaining subunit.

[0230] The first obtaining subunit is configured to obtain, when 1 < m ≤ M, a second intermediate global visual feature set corresponding to at least one video frame set at the m-1th level based on the first intermediate global visual feature set corresponding to at least one video frame set. The second intermediate global visual feature set includes at least one second intermediate global visual feature. The second intermediate global visual feature includes at least one of an intermediate global implicit visual feature and an intermediate global explicit visual feature.

[0231] The second obtaining subunit is used to obtain the first intermediate global visual feature set corresponding to at least one video frame set at the mth level based on the second intermediate global visual feature set corresponding to at least one video frame set at the mth level and the first intermediate global visual feature set corresponding to at least one video frame set at the m-1th level.

[0232] The third obtaining subunit is configured to obtain a global visual feature corresponding to at least one video frame according to a first intermediate global visual feature set corresponding to at least one video frame set at the T-th level.

[0233] According to an embodiment of the present disclosure, m is an integer greater than or equal to 1 and less than or equal to M. T is an integer greater than or equal to 1 and less than or equal to M.

[0234] According to an embodiment of the present disclosure, at least one video frame set includes N. The nth video frame set includes n T video frames, N is an integer greater than or equal to 1. T is an integer greater than or equal to 1.

[0235] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global implicit visual feature, the first obtaining subunit may be configured to:

[0236] For the nth video frame set in the N video frame sets, at least one first matrix set corresponding to the nth video frame set at the m-1th level is determined based on the first intermediate global visual feature set corresponding to the nth video frame set. The first matrix set includes a first query matrix, a first key matrix, and a first value matrix. For the nth video frame set in the nth video frame set, t video frames, according to the mth level and the nth level t The first query matrix corresponding to the video frame and the first key matrix corresponding to the n-th video frame set at the m-th level are obtained. t The first attention matrix corresponding to the video frame. According to the m-th level and the n-th level t The first attention matrix corresponding to the video frame and the mth level and the nth level t The first value matrix corresponding to the video frame is obtained at the mth level and the nth level. t The third intermediate global visual feature corresponding to the video frame. t The third intermediate global visual feature corresponding to the video frame is obtained at the mth level and the nth level. t The intermediate global implicit visual features corresponding to each video frame.

[0237] According to an embodiment of the present disclosure, t is an integer greater than or equal to 1 and less than or equal to T.

[0238] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global explicit visual feature, the first obtaining subunit may be configured to:

[0239] For the n-th video frame set among the N video frame sets, the similarity between the video frames in the n-th video frame set at the m-1-th level is determined based on the first intermediate global visual feature set corresponding to the n-th video frame set. Based on the similarity between the video frames in the n-th video frame set at the m-th level, the intermediate global explicit visual feature set corresponding to the n-th video frame set at the m-th level is obtained.

[0240] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global implicit visual feature and an intermediate global explicit visual feature, the second obtaining subunit may be configured to:

[0241] The intermediate global implicit visual feature set and the intermediate global explicit visual feature set of the mth level corresponding to the at least one video frame set are fused to obtain a fourth intermediate global visual feature set of the mth level corresponding to the at least one video frame set. The first intermediate global visual feature set of the mth level corresponding to the at least one video frame set is obtained based on the fourth intermediate global visual feature set of the mth level corresponding to the at least one video frame set and the first intermediate global visual feature set of the m-1th level corresponding to the at least one video frame set.

[0242] According to an embodiment of the present disclosure, when m=1, the first processing unit may further include a fourth obtaining subunit and a fifth obtaining subunit.

[0243] The fourth obtaining subunit is configured to obtain a second intermediate global visual feature set of a second level corresponding to at least one video frame set based on the local visual feature set of the first level corresponding to at least one video frame set.

[0244] The fifth obtaining subunit is used to obtain a first intermediate global visual feature set of the second level corresponding to at least one video frame set based on the second intermediate global visual feature set of the second level corresponding to at least one video frame set and the local visual feature set of the first level corresponding to at least one video frame set.

[0245] According to an embodiment of the present disclosure, at least one video frame set includes N, and the nth video frame set includes n T video frames. N is an integer greater than or equal to 1. T is an integer greater than or equal to 1.

[0246] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global implicit visual feature, the fourth obtaining subunit may be configured to:

[0247] For the nth video frame set in the N video frame sets, at least one second matrix set corresponding to the nth video frame set at the second level is determined based on the local visual feature set corresponding to the nth video frame set at the first level. The second matrix set includes a second query matrix, a second key matrix and a second value matrix. t video frames, according to the second level and the nth t The second query matrix corresponding to the nth video frame and the second key matrix corresponding to the nth video frame set at the second level are obtained. t The second attention matrix corresponding to the video frame. According to the second level and the nth t The second attention matrix corresponding to the video frame and the second level and the nth level t The second value matrix corresponding to the video frame is obtained at the second level and the nth level. t The third intermediate global visual feature corresponding to the nth video frame. t The third intermediate global visual feature corresponding to the video frame is obtained at the second level and the nth level. t The intermediate global implicit visual features corresponding to each video frame.

[0248] According to an embodiment of the present disclosure, t is an integer greater than or equal to 1 and less than or equal to T.

[0249] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global explicit visual feature, the fourth obtaining subunit may be configured to:

[0250] For the nth video frame set among the N video frame sets, the similarity between the video frames in the nth video frame set at the second level is determined based on the local visual feature set corresponding to the nth video frame set at the first level. Based on the similarity between the video frames in the nth video frame set at the second level, the intermediate global explicit visual feature set corresponding to the nth video frame set at the mth level is obtained.

[0251] According to an embodiment of the present disclosure, when the second intermediate global visual feature includes an intermediate global implicit visual feature and an intermediate global explicit visual feature, the fifth obtaining subunit may be configured to:

[0252] The intermediate global implicit visual feature set and the intermediate global explicit visual feature set of the second level corresponding to the at least one video frame set are fused to obtain a fourth intermediate global visual feature set of the second level corresponding to the at least one video frame set. A first intermediate global visual feature set of the mth level corresponding to the at least one video frame set is obtained based on the fourth intermediate global visual feature set of the second level corresponding to the at least one video frame set and the local visual feature set of the first level corresponding to the at least one video frame set.

[0253] According to an embodiment of the present disclosure, the second determining module may include a second determining submodule and a third determining submodule.

[0254] The second determining submodule is configured to determine shot segmentation information corresponding to the at least one video frame set based on a global visual feature set corresponding to the at least one video frame set.

[0255] The third determining submodule is configured to determine the shot segmentation information of the video according to the shot segmentation information corresponding to at least one video frame set.

[0256] Figure 9 The block diagram of the visual task processing device according to an embodiment of the present disclosure is schematically shown.

[0257] like Figure 9 As shown, the visual task processing apparatus 900 may include a third determining module 910 and a processing module 920 .

[0258] The third determining module 910 is configured to determine shot segmentation information of the video to be processed.

[0259] The processing module 920 is used to process visual tasks according to the shot segmentation information of the video to be processed.

[0260] According to an embodiment of the present disclosure, the shot segmentation information of the video to be processed may be determined using the shot segmentation apparatus according to an embodiment of the present disclosure.

[0261] According to an embodiment of the present disclosure, the visual task includes at least one of the following: a video scene recognition task, a video frame deduplication task, and a shot extraction task.

[0262] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0263] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the present disclosure.

[0264] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in the present disclosure.

[0265] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described in the present disclosure.

[0266] Figure 10A block diagram of an electronic device suitable for implementing the shot segmentation method and the visual task processing method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0267] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the electronic device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0268] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0269] The computing unit 1001 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the shot segmentation method and the visual task processing method. For example, in some embodiments, the shot segmentation method and the visual task processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 1000 via the ROM 802 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the shot segmentation method and the visual task processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the shot segmentation method and the visual task processing method in any other appropriate manner (for example, by means of firmware).

[0270] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0271] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0272] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0273] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0274] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0275] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0276] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0277] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A shot segmentation method, comprising: Performing two-dimensional feature extraction on at least one video frame in the video to obtain a local visual feature corresponding to the at least one video frame; dividing the at least one video frame into at least one video frame set; Performing M-level processing on the local visual feature set corresponding to the at least one video frame set based on the global feature extraction strategy to obtain global visual features corresponding to the at least one video frame, including: when M>1 and 1<m≤M, obtaining a second intermediate global visual feature set corresponding to the at least one video frame set at the m-1th level based on the first intermediate global visual feature set corresponding to the at least one video frame set, wherein the second intermediate global visual feature set includes at least one second intermediate global visual feature; obtaining a first intermediate global visual feature set corresponding to the at least one video frame set at the m-1th level based on the second intermediate global visual feature set corresponding to the at least one video frame set at the m-1th level; and obtaining global visual features corresponding to the at least one video frame based on the first intermediate global visual feature set corresponding to the at least one video frame set at the Rth level; wherein R is an integer greater than or equal to 1 and less than or equal to M; and Shot segmentation information of the video is determined based on the global visual features corresponding to the at least one video frame.

2. The method according to claim 1, wherein The performing two-dimensional feature extraction on at least one video frame in the video to obtain a local visual feature corresponding to the at least one video frame includes: Performing two-dimensional convolution processing on at least one video frame in the video to obtain local visual features corresponding to the at least one video frame.

3. The method according to claim 1, wherein The global feature extraction strategy includes at least one of a self-attention strategy and a similarity strategy.

4. The method according to claim 3, wherein: The second intermediate global visual feature includes at least one of an intermediate global implicit visual feature and an intermediate global explicit visual feature.

5. The method according to claim 4, wherein The at least one video frame set includes N, and the nth video frame set includes n T the video frames, N is an integer greater than or equal to 1, and T is an integer greater than or equal to 1; Wherein, in a case where the second intermediate global visual feature includes the intermediate global implicit visual feature, obtaining the second intermediate global visual feature set of the mth level corresponding to the at least one video frame set according to the first intermediate global visual feature set of the m-1th level corresponding to the at least one video frame set includes: For the nth video frame set among the N video frame sets, Determining, based on the first intermediate global visual feature set of the m-1th level corresponding to the nth video frame set, at least one first matrix set of the mth level corresponding to the nth video frame set, wherein the first matrix set includes a first query matrix, a first key matrix, and a first value matrix; and For the nth video frame set t the video frames, According to the mth level and the nth level t The first query matrix corresponding to the nth video frame and the first key matrix corresponding to the nth video frame set at the mth level are obtained. t A first attention matrix corresponding to the video frames; According to the mth level and the nth level t The first attention matrix corresponding to the video frame and the m-th level and the n-th level t The first value matrix corresponding to the video frame is obtained by t a third intermediate global visual feature corresponding to each of the video frames; According to the mth level and the nth level t The third intermediate global visual feature corresponding to the video frame is obtained by t intermediate global implicit visual features corresponding to the video frames; Here, t is an integer greater than or equal to 1 and less than or equal to T.

6. The method according to claim 5, wherein: In a case where the second intermediate global visual features include the intermediate global explicit visual features, obtaining, based on the first intermediate global visual feature set at the m-1th level corresponding to the at least one video frame set, a second intermediate global visual feature set at the m-1th level corresponding to the at least one video frame set, includes: For the nth video frame set among the N video frame sets, determining, based on the first intermediate global visual feature set at the m-1th level corresponding to the nth video frame set, similarities between video frames in the nth video frame set at the mth level; and According to the similarity between the video frames in the n-th video frame set of the m-th level, an intermediate global explicit visual feature set corresponding to the n-th video frame set of the m-th level is obtained.

7. The method according to any one of claims 4 to 6, wherein In a case where the second intermediate global visual features include the intermediate global implicit visual features and the intermediate global explicit visual features, obtaining, based on the second intermediate global visual feature set of the m-th level corresponding to the at least one video frame set and the first intermediate global visual feature set of the m-th level corresponding to the at least one video frame set, the first intermediate global visual feature set of the m-th level corresponding to the at least one video frame set, comprises: fusing the intermediate global implicit visual feature set and the intermediate global explicit visual feature set of the m-th level corresponding to the at least one video frame set to obtain a fourth intermediate global visual feature set of the m-th level corresponding to the at least one video frame set; and According to the fourth intermediate global visual feature set of the mth level corresponding to the at least one video frame set and the first intermediate global visual feature set of the m-1th level corresponding to the at least one video frame set, the first intermediate global visual feature set of the mth level corresponding to the at least one video frame set is obtained.

8. The method according to any one of claims 4 to 6, further comprising: In the case of m=1, Obtaining a second intermediate global visual feature set at a second level corresponding to the at least one video frame set according to the local visual feature set at the first level corresponding to the at least one video frame set; as well as According to the second intermediate global visual feature set of the second level corresponding to the at least one video frame set and the local visual feature set of the first level corresponding to the at least one video frame set, a first intermediate global visual feature set of the second level corresponding to the at least one video frame set is obtained.

9. The method according to claim 8, wherein The at least one video frame set includes N, and the nth video frame set includes n T the video frames, N is an integer greater than or equal to 1, and T is an integer greater than or equal to 1; Wherein, when the second intermediate global visual feature includes the intermediate global implicit visual feature, obtaining the second intermediate global visual feature set corresponding to the at least one video frame set at the second level according to the local visual feature set corresponding to the at least one video frame set at the first level includes: For the nth video frame set among the N video frame sets, Determining at least one second matrix set corresponding to the nth video frame set at the second level based on the local visual feature set corresponding to the nth video frame set at the first level, wherein the second matrix set includes a second query matrix, a second key matrix, and a second value matrix; and For the nth video frame set t the video frames, According to the second level and the nth t The second query matrix corresponding to the nth video frame and the second key matrix corresponding to the nth video frame set at the second level are obtained. t A second attention matrix corresponding to the video frames; According to the second level and the nth t The second attention matrix corresponding to the video frame and the second level and the nth t The second value matrix corresponding to the nth video frame is obtained. t a third intermediate global visual feature corresponding to each of the video frames; According to the second level and the nth t The third intermediate global visual feature corresponding to the nth video frame is obtained by t intermediate global implicit visual features corresponding to the video frames; Here, t is an integer greater than or equal to 1 and less than or equal to T.

10. The method according to claim 9, wherein: In a case where the second intermediate global visual feature includes the intermediate global explicit visual feature, obtaining a second intermediate global visual feature set corresponding to the at least one video frame set at a second level according to the local visual feature set corresponding to the at least one video frame set at the first level includes: For the nth video frame set among the N video frame sets, Determining the similarity between the video frames in the nth video frame set of the second level according to the local visual feature set corresponding to the nth video frame set of the first level; and According to the similarity between the video frames in the n-th video frame set of the second level, an intermediate global explicit visual feature set corresponding to the n-th video frame set of the second level is obtained.

11. The method according to claim 8, wherein In a case where the second intermediate global visual features include the intermediate global implicit visual features and the intermediate global explicit visual features, obtaining, based on the second intermediate global visual feature set of the second level corresponding to the at least one video frame set and the local visual feature set of the first level corresponding to the at least one video frame set, a first intermediate global visual feature set of the second level corresponding to the at least one video frame set, comprises: fusing the intermediate global implicit visual feature set and the intermediate global explicit visual feature set of the second level corresponding to the at least one video frame set to obtain a fourth intermediate global visual feature set of the second level corresponding to the at least one video frame set; According to the fourth intermediate global visual feature set of the second level corresponding to the at least one video frame set and the local visual feature set of the first level corresponding to the at least one video frame set, the first intermediate global visual feature set of the second level corresponding to the at least one video frame set is obtained.

12. The method according to any one of claims 1 to 6, wherein The determining, based on the global visual features corresponding to the at least one video frame, shot segmentation information of the video includes: Determining shot segmentation information corresponding to the at least one video frame set based on a global visual feature set corresponding to the at least one video frame set; and The shot segmentation information of the video is determined according to the shot segmentation information corresponding to the at least one video frame set.

13. A method for processing a visual task, comprising: Determining shot segmentation information of a video to be processed, wherein the shot segmentation information of the video to be processed is determined using the method according to any one of claims 1 to 12; and Processing a visual task according to the shot segmentation information of the video to be processed.

14. The method according to claim 13, wherein The visual task includes at least one of the following: a video scene recognition task, a video frame deduplication task, and a shot extraction task.

15. A shot splitting device, comprising: a two-dimensional feature extraction module, configured to perform two-dimensional feature extraction on at least one video frame in a video to obtain a local visual feature corresponding to the at least one video frame; a division submodule, configured to divide the at least one video frame into at least one video frame set; a first processing unit, configured to perform M-level processing on the local visual feature set corresponding to the at least one video frame set based on a global feature extraction strategy to obtain a global visual feature corresponding to the at least one video frame; as well as The second determining module is configured to determine shot segmentation information of the video based on global visual features corresponding to the at least one video frame, When M is greater than 1, the first processing unit includes a first obtaining subunit, a second obtaining subunit and a third obtaining subunit. The first obtaining subunit is configured to obtain, based on the first intermediate global visual feature set corresponding to the at least one video frame set at the m-1th level, a second intermediate global visual feature set corresponding to the at least one video frame set, when 1<m≤M, wherein the second intermediate global visual feature set includes at least one second intermediate global visual feature; a second obtaining subunit, configured to obtain a first intermediate global visual feature set at the mth level corresponding to the at least one video frame set based on the second intermediate global visual feature set at the mth level corresponding to the at least one video frame set and the first intermediate global visual feature set at the m-1th level corresponding to the at least one video frame set; The third obtaining subunit is used to obtain the global visual features corresponding to the at least one video frame based on the first intermediate global visual feature set corresponding to the at least one video frame set at the R-th level, where R is an integer greater than or equal to 1 and less than or equal to M.

16. A visual task processing device, comprising: a third determining module, configured to determine shot segmentation information of a video to be processed, wherein the shot segmentation information of the video to be processed is determined using the method according to any one of claims 1 to 12; and A processing module is used to process visual tasks according to the shot segmentation information of the video to be processed.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 14.

Images