Highlight video recognition method and device, electronic device and storage medium

Abstract

The present disclosure relates to a method and apparatus for highlight video recognition, an electronic device, and a storage medium. The method is applied to a recognition network and includes: extracting video features of a video to be recognized, wherein the video to be recognized includes multiple video clips arranged in time sequence, and each of the video clips includes at least one video sub-feature; recognizing highlight videos based on the video sub-features of the multiple video clips to obtain a recognition result; wherein the recognition network performs parameter update based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in a feature space, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update. The embodiments of the present disclosure can improve the accuracy of highlight video recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for recognizing highlight videos, an electronic device, and a storage medium. Background Art

[0002] In recent years, with the rapid rise of short video media, many wonderful videos have been posted on the Internet. For long videos, in order to save time, people prefer to watch the wonderful parts (i.e., the highlights). However, manually cropping and editing videos to select the highlights is relatively labor-intensive and time-consuming. Therefore, the video highlight detection task was born based on this practical application.

[0003] The main goal of the video highlight detection task is to locate highlight segments in uncropped videos. This task is closely related to practical applications and can be well applied to various downstream tasks such as video action detection and video summarization.

[0004] However, traditional methods cannot effectively distinguish highlight segments from non-highlight segments, and the probability of confusion between video segments is high, resulting in low accuracy of highlight video detection. Summary of the Invention

[0005] This disclosure proposes a technical solution for highlight video recognition.

[0006] According to one aspect of the present disclosure, a method for identifying highlight videos is provided, comprising:

[0007] Extracting video features of a video to be identified, wherein the video to be identified includes a plurality of video clips arranged in time sequence, and each of the video clips includes at least one video sub-feature;

[0008] Identifying highlight videos based on the video sub-features of the plurality of video clips to obtain an identification result;

[0009] The recognition network performs parameter updates based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in the feature space, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update.

[0010] In one possible implementation, performing parameter updating based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in a feature space includes: determining a first loss of the recognition network based on the first distance and the second distance, wherein the first loss is positively correlated with the first distance and negatively correlated with the second distance;

[0011] Based on the first loss, parameters of the recognition network are updated to reduce the first loss.

[0012] In a possible implementation, the method further includes:

[0013] determining, in the video clip, a junction between a highlight video clip and a non-highlight video clip;

[0014] Based on a third distance between the video sub-features of the highlight video clip and the non-highlight video clip before and after the adjacent portion, the parameters of the recognition network are updated so that the distance between the video sub-features of the highlight video clip and the non-highlight video clip output by the recognition network is greater than the distance before the update.

[0015] In a possible implementation, the video to be identified is a sample video, and the video segment is annotated with a label for indicating whether the video segment is a highlight;

[0016] The determining of the adjacent portion between the highlight video segment and the non-highlight video segment in the video segment includes:

[0017] Determining, based on the labels of the video segments, the adjacency between the highlight video segment and the non-highlight video segment;

[0018] Based on the highlight video segments and the non-highlight video segments before and after the adjacent portion, positive and negative sample pairs are constructed.

[0019] In a possible implementation, updating the parameters of the recognition network based on the third distance between the video sub-features of the highlight video segments and the non-highlight video segments before and after the adjacent portion includes:

[0020] constructing a plurality of positive and negative sample pairs based on a plurality of highlight video segments and a plurality of non-highlight video segments before and after the adjacent portion, wherein a single positive and negative sample pair includes one highlight video segment and one non-highlight video segment;

[0021] determining a second loss of the recognition network according to a third distance between the video sub-features of the highlight video segment and the non-highlight video segment in each positive and negative sample pair, where the second loss is negatively correlated with the third distance;

[0022] Based on the second loss, the parameters of the recognition network are updated to reduce the second loss.

[0023] In a possible implementation, the video to be identified is a sample video, the video segment is annotated with a label indicating whether the video segment is a highlight, and the method further includes:

[0024] determining a third loss based on a difference between the recognition result and the label of the video clip;

[0025] Based on the third loss, the parameters of the recognition network are updated so that the recognition result output by the recognition network is consistent with the label.

[0026] In a possible implementation manner, updating the parameters of the identification network includes:

[0027] Performing a weighted summation on at least two of the first loss, the second loss, and the third loss to obtain a target loss;

[0028] Based on the target loss, the recognition network is updated to reduce the target loss.

[0029] In one possible implementation, extracting video features of a video to be identified includes:

[0030] Obtaining visual features, audio features, and a splicing feature of the video to be identified after encoding the visual features and the audio features; wherein each video segment corresponds to at least one visual sub-feature, audio sub-feature, and splicing sub-feature;

[0031] The visual features, audio features and splicing features are fused to obtain video features, wherein at least one visual sub-feature, audio sub-feature and splicing sub-feature corresponding to each video clip is fused to obtain the video sub-feature.

[0032] According to one aspect of the present disclosure, a highlight video recognition device is provided, comprising:

[0033] An extraction module, configured to extract video features of a video to be identified, wherein the video to be identified comprises a plurality of video clips arranged in time sequence, and each of the video clips comprises at least one video sub-feature;

[0034] an identification module, configured to identify highlight videos based on the video sub-features of the plurality of video clips and obtain an identification result;

[0035] The recognition network performs parameter updates based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in the feature space, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update.

[0036] In a possible implementation, the apparatus further includes:

[0037] A first parameter updating module is configured to determine a first loss of the recognition network based on the first distance and the second distance, wherein the first loss is positively correlated with the first distance and negatively correlated with the second distance; and based on the first loss, update the parameters of the recognition network to reduce the first loss.

[0038] In a possible implementation, the apparatus further includes:

[0039] The second parameter updating module is used to determine the adjacent part of the highlight video segment and the non-highlight video segment in the video segment; based on the third distance between the video sub-features of the highlight video segment and the non-highlight video segment before and after the adjacent part, update the parameters of the recognition network so that the distance between the video sub-features of the highlight video segment and the non-highlight video segment output by the recognition network is greater than the distance before the update.

[0040] In a possible implementation, the video to be identified is a sample video, and the video segment is annotated with a label for indicating whether the video segment is a highlight;

[0041] The second parameter updating module is used to determine the adjacent part of the highlight video segment and the non-highlight video segment based on the label of the video segment; and construct positive and negative sample pairs based on the highlight video segment and the non-highlight video segment before and after the adjacent part.

[0042] In a possible implementation, the second parameter updating module is used to construct a plurality of positive and negative sample pairs based on a plurality of highlight video clips and a plurality of non-highlight video clips before and after the adjacent portion, wherein a single positive and negative sample pair includes one highlight video clip and one non-highlight video clip; determine a second loss of the recognition network based on a third distance between video sub-features of the highlight video clip and the non-highlight video clip in each positive and negative sample pair, wherein the second loss is negatively correlated with the third distance; and update the parameters of the recognition network based on the second loss to reduce the second loss.

[0043] In a possible implementation, the video to be identified is a sample video, the video segment is annotated with a label indicating whether the video segment is a highlight, and the apparatus further includes:

[0044] A third parameter updating module is used to determine a third loss based on the difference between the recognition result and the label of the video clip; based on the third loss, update the parameters of the recognition network so that the recognition result output by the recognition network is consistent with the label.

[0045] In one possible implementation, the device further includes a fourth parameter updating module, configured to perform a weighted summation on at least two of the first loss, the second loss, and the third loss to obtain a target loss; and update the recognition network based on the target loss to reduce the target loss.

[0046] In one possible implementation, an extraction module is configured to obtain visual features, audio features, and a splicing feature of the video to be identified after encoding the visual features and the audio features; wherein each of the video clips corresponds to at least one visual sub-feature, audio sub-feature, and splicing sub-feature; and the visual features, audio features, and splicing features are fused to obtain a video feature, wherein at least one visual sub-feature, audio sub-feature, and splicing sub-feature corresponding to each of the video clips are fused to obtain the video sub-feature.

[0047] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0048] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0049] In an embodiment of the present disclosure, a recognition network extracts video features of a video to be recognized, and the video to be recognized includes multiple video clips arranged in time sequence, each of the video clips includes at least one video sub-feature, and then recognizes highlight videos based on the video sub-features of the multiple video clips to obtain a recognition result, wherein the recognition network updates parameters based on a first distance between video sub-features of video clips with the same label in the feature space and a second distance between video sub-features of video clips with different labels, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update. Compared with unsupervised learning for the entire video, the recognition network provided by the present disclosure divides an entire video into multiple video clips during training, and the video clips have labels, thereby enabling supervised intensive learning in the dimension of the video clips, that is, recognition of highlight parts in the entire video is achieved through learning in the dimension of the video clips, which can significantly improve the accuracy of highlight video recognition. After the recognition network updates its parameters, it can shorten the distance between the video sub-features of video clips with the same label and increase the distance between the video sub-features of video clips with different labels. This makes the feature space more structured, enhances the compactness of similar samples and the discreteness of different samples, and enables more accurate distinction between highlight video clips and non-highlight video clips, thereby improving the accuracy of highlight video recognition.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0052] Figure 1 A flowchart of a highlight video recognition method according to an embodiment of the present disclosure is shown.

[0053] Figure 2 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown.

[0054] Figure 3 A schematic diagram of another application scenario according to an embodiment of the present disclosure is shown.

[0055] Figure 4 A block diagram of a highlight video recognition device according to an embodiment of the present disclosure is shown.

[0056] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0057] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0059] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0060] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0061] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0062] In an embodiment of the present disclosure, a method for highlight video recognition is proposed to address the problems that traditional methods cannot effectively distinguish and differentiate between the features of highlight segments and non-highlight segments, and the high probability of confusion of video segments. For the video sub-features of multiple video segments of the video to be recognized output by the recognition network, parameters are updated based on a first distance between the video sub-features of video segments with the same label in the feature space and a second distance between the video sub-features of video segments with different labels, so that the distance between the video sub-features of video segments with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between the video sub-features of video segments with different labels is larger than the distance before the update. Compared with unsupervised learning for the entire video, the recognition network provided by the present disclosure divides an entire video into multiple video segments during training, and the video segments have labels. Therefore, intensive learning can be performed in a supervised manner on the dimension of the video segments, that is, recognition of the highlight part of the entire video is achieved through learning on the dimension of the video segments, which can significantly improve the accuracy of highlight video recognition. After the recognition network updates its parameters, it can shorten the distance between the video sub-features of video clips with the same label and increase the distance between the video sub-features of video clips with different labels. This makes the feature space more structured, enhances the compactness of similar samples and the discreteness of different samples, and enables more accurate distinction between highlight video clips and non-highlight video clips, thereby improving the accuracy of highlight video recognition.

[0063] In one possible implementation, the highlight video recognition method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0064] Figure 1 A flow chart of a method for identifying highlight video according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the highlight video recognition method is applied to the recognition network, including:

[0065] In step S11, video features of the video to be identified are extracted.

[0066] The video to be identified includes a plurality of video clips arranged in time sequence, and each of the video clips includes at least one video sub-feature.

[0067] The video to be identified can be any video for which highlight clips are to be identified. During the identification process, the video to be identified can be divided into multiple video clips, and then the video clips are identified as highlight clips. Highlight clips can be wonderful or interesting clips in the video.

[0068] The lengths of the multiple video segments divided here can be the same, for example, the lengths of the video segments are all 50 frames, or the lengths of the video segments are all 100 frames; the lengths of the multiple video segments can also be different, which is not limited in this disclosure.

[0069] The video to be identified can be a video file stored in a local storage space, and the video to be identified can be read from the local storage space of the terminal and segmented. For example, it can be a video file of a locally stored sports event, or a video frame of a locally stored shopping mall management video.

[0070] Alternatively, the video to be identified may be a video captured in real time by an image capture device, for example, a live video of a sports event, or a video captured in real time by an image capture device located at the entrance of a shopping mall.

[0071] For the segmented video clips, the visual features F of the video can be extracted through the visual feature extraction network (Inflated 3D ConvNet, I3D) v , using the trained audio network (Pretrained Audio Neural-Networks, PANN) to extract the audio features F in the video a For the visual feature F v Specifically, all video frames in a single video clip can be extracted and fused to obtain the visual feature F v Alternatively, some video frames (e.g., 16 frames randomly selected) from a single video clip can be input into the I3D network, and the final output feature vector is used as the visual feature F v . Visual features F v The visual sub-features of each video clip arranged in time sequence are included in F. a contains the audio sub-features of each audio clip arranged in time sequence.

[0072] For the extracted visual features F v and audio feature F a , the linear layer can be used to unify the feature dimensions of the two and unify them to the same feature space dimension so that operations can be performed on the same feature space dimension, and then the visual feature F v and audio feature F aBy performing operations such as encoding and weighted fusion, the video features of the video to be identified can be obtained. The video features include video sub-features of each video clip arranged in time sequence. The specific process can be found in the feature extraction method provided in the present disclosure, which will not be described here.

[0073] In step S12, highlight video is identified based on the video sub-features of the plurality of video features to obtain an identification result.

[0074] The recognition network performs parameter updates based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in the feature space, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update.

[0075] Each video clip corresponds to a video sub-feature. The recognition network can classify each video sub-feature using a classifier. This classifier can be a binary classifier, that is, based on the video sub-feature, output video clips as highlight clips or non-highlight clips. In this way, highlight video clips can be obtained, realizing the recognition of highlight video clips in the video.

[0076] The recognition network can be an end-to-end network, that is, it inputs a video / video clip and outputs a recognition result of whether the video clip is a highlight video. During the training process of the recognition network, the training process can be based on the labels of the video clip to adjust the parameters in the recognition network.

[0077] During the training process of the recognition network, video clips are pre-labeled into two categories: highlights and non-highlights. The recognition network then performs supervised learning based on these labels. In the feature space, network parameters are updated to bring samples of the same class closer together and samples of different classes farther apart. This makes the sample space more specific and structured, enhancing intra-class cohesion and inter-class dispersion.

[0078] The recognition network parameters can be updated based on the first distance between video sub-features with the same label in the feature space and the second distance between video sub-features with different labels, so that the distance between video sub-features with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features with different labels is larger than the distance before the update.

[0079] The first distance and the second distance here may be the Euclidean distance between the two video sub-features. The specific method for determining the Euclidean distance is not described here.

[0080] Since the recognition network after parameter update can shorten the distance between video sub-features with the same label and increase the distance between video sub-features with different labels, when identifying highlight videos based on video sub-features, the recognition network can better distinguish highlight videos from non-highlight videos, thereby improving the accuracy of highlight video recognition.

[0081] In one possible implementation, the method further includes: determining a first distance between video sub-features with the same label in the feature space, and a second distance between video sub-features with different labels, wherein the label is used to mark whether the video clip belongs to a highlight video clip; updating the parameters of the recognition network based on the first distance and the second distance, so that the distance between video sub-features with the same label output by the recognition network is smaller than the distance before the update, and the distance between video sub-features with different labels is larger than the distance before the update.

[0082] In this implementation, the parameters of the recognition network can be updated by taking video clips labeled as highlight videos as positive samples and video clips labeled as non-highlight videos as negative samples, and then learning by comparing the positive and negative samples, thereby shortening the distance between positive samples and increasing the distance between positive and negative samples.

[0083] For any video sub-feature labeled y, positive samples are all video sub-features labeled y other than that sub-feature, while negative samples are those not labeled y. Using that sub-feature as an anchor point, the first distance between it and other sub-features labeled y is determined. Then, the second distance between it and other sub-features not labeled y is determined. Based on the first and second distances, the parameters of the recognition network are updated, reducing the first distance output by the recognition network and increasing the second distance. This approach shortens the distance between samples of the same type and increases the distance between samples of different types.

[0084] The updating of the recognition network parameters can be achieved by calculating the network loss through a loss function. In one possible implementation method, the parameters are updated based on the first distance between the video sub-features of the video clips with the same label in the feature space, and the second distance between the video sub-features of the video clips with different labels, including: determining the first loss of the recognition network based on the first distance and the second distance, wherein the first loss is positively correlated with the first distance and the first loss is negatively correlated with the second distance; based on the first loss, updating the parameters of the recognition network to reduce the first loss.

[0085] The larger the loss calculated by the loss function, the greater the error of the recognition network, and the need to further optimize the parameters of the recognition network. The smaller the loss calculated by the loss function, the smaller the error of the recognition network and the more accurate the network's prediction results.

[0086] Then, the first loss is positively correlated with the first distance, which means that the larger the first distance, the greater the loss, that is, the parameters of the recognition network need to be updated to make the first distance smaller; and when the first distance becomes smaller, the first loss will also become smaller, and the recognition network will be more accurate.

[0087] The first loss is negatively correlated with the second distance, which means that the smaller the first distance, the greater the loss, that is, the parameters of the recognition network need to be updated to increase the second distance; and when the second distance increases, the first loss will also decrease, and the recognition network will be more accurate.

[0088] Please refer to formula (1), which is a method for determining the first loss provided by the embodiment of the present disclosure. The loss function of .

[0089]

[0090] Among them, T represents the length of the video segment, P represents the set of positive sample segments in all video segments, represents a set of video sub-features of a video clip, q is a video sub-feature of any positive sample clip in the positive sample clip set, k + represents the video sub-features of other positive sample segments except q, N represents the set of negative samples in all video segments, k - represents the video sub-features of all negative sample segments, and τ represents the temperature coefficient.

[0091] Obviously, in formula (1), q·k + That is the first distance between positive samples, q·k - That is the second distance between the positive sample and the negative sample, which can be determined by formula (1). The first loss The value of is proportional to the first distance and inversely proportional to the second distance. The smaller the distance between the positive sample q and other positive samples and the larger the distance between the positive sample q and other negative samples, the smaller the first loss.

[0092] Moreover, in the above formula (1), the first distance and the second distance between each video sub-feature q in the positive sample and other video clips are considered to obtain the first loss, that is, all video clips in the video are densely feature-structured, thereby improving the accuracy of highlight detection.

[0093] In one possible implementation, the method further includes: determining the adjoining point between the highlight video segment and the non-highlight video segment in the video segment; and updating the parameters of the recognition network based on a third distance between the video sub-features of the highlight video segment and the non-highlight video segment before and after the adjoining point, so that the distance between the video sub-features of the highlight video segment and the non-highlight video segment output by the recognition network is greater than the distance before the update.

[0094] The adjacent portion between the highlight video segment and the non-highlight video segment may be a position where the highlight video segment and the non-highlight video segment are adjacent.

[0095] In one possible implementation, the video to be identified is a sample video, and the video clip is marked with a label for characterizing whether the video clip belongs to a highlight; determining the adjacency of the highlight video clip and the non-highlight video clip in the video clip includes: determining the adjacency of the highlight video clip and the non-highlight video clip based on the label of the video clip; and constructing positive and negative sample pairs based on the highlight video clip and the non-highlight video clip before and after the adjacency.

[0096] Since the video clips are pre-labeled, the junctions are also fixed for a series of time-sequential video clips. For example, the number 1 represents a highlight video clip, and the number 0 represents a non-highlight video clip. For example, the sequence of video clips in a video is represented as 0000000001111111000000011000. The locations where 0 and 1 change are the junctions between the highlight video clip and the non-highlight video clip. These junctions are also called watersheds.

[0097] The highlight video clips and non-highlight video clips before and after the adjacent points often have a high degree of similarity, making it difficult for the recognition network to distinguish them. Therefore, the highlight video clips and non-highlight video clips before and after the adjacent points can be selected as difficult samples for targeted training of the recognition network. Positive and negative sample pairs can be constructed based on the highlight video clips and non-highlight video clips before and after the adjacent points. Then, based on the third distance between the video sub-features of the highlight video clips and non-highlight video clips before and after the adjacent points, the parameters of the recognition network can be updated so that the distance between the video sub-features of the highlight video clips and non-highlight video clips output by the recognition network is greater than the distance before the update.

[0098] The third distance here may be the Euclidean distance between the video sub-features, and the specific process of determining the Euclidean distance is not described in detail here.

[0099] In the disclosed embodiment, the adjacent part of the highlight video segment and the non-highlight video segment in the video segment is determined; based on the third distance between the video sub-features of the highlight video segment and the non-highlight video segment before and after the adjacent part, the parameters of the recognition network are updated so that the distance between the video sub-features of the highlight video segment and the non-highlight video segment output by the recognition network is greater than the distance before the update. Since the highlight segment and the adjacent non-highlight segment in a video have extremely high visual similarity, and the relationship between these segments can be regarded as a relationship mining problem of difficult sample pairs, then, by using the third distance between the video sub-features of the highlight video segment and the non-highlight video segment before and after the adjacent part, the parameters of the recognition network are updated so that the recognition network can distinguish the highlight video segment and the non-highlight video segment before and after the adjacent part as much as possible, thereby improving the ability to distinguish these similar video segments and improving the accuracy of highlight video recognition.

[0100] The updating of the recognition network parameters can be achieved by calculating the network loss through a loss function. In a possible implementation method, the parameters of the recognition network are updated based on the third distance between the video sub-features of the highlight video clips and the non-highlight video clips before and after the adjacent point, including: constructing a plurality of positive and negative sample pairs based on multiple highlight video clips and multiple non-highlight video clips before and after the adjacent point, wherein a single positive and negative sample pair contains one highlight video clip and one non-highlight video clip; determining the second loss of the recognition network according to the third distance between the video sub-features of the highlight video clip and the non-highlight video clip in each positive and negative sample pair, wherein the second loss is negatively correlated with the third distance; and updating the parameters of the recognition network based on the second loss to reduce the second loss.

[0101] For multiple highlight video clips and non-highlight video clips before and after the adjacent point, multiple sample pairs can be constructed. Each sample pair contains a positive sample and a negative sample. For example, assuming that the adjacent point is between samples 5 and 6, then samples 5 and 6 can constitute a sample pair, samples 4 and 7 can constitute a sample pair, and samples 3 and 8 can constitute a sample pair. In this way, three sample pairs are obtained. Similarly, for other adjacent points, multiple sample pairs can be constructed using this method. In addition, the adjacent point can be represented by any video clip in the adjacent positive and negative samples. In this case, the samples before and after the video clip can be selected to form a sample pair. For example, assuming that the adjacent point is between samples 5 and 6, we select sample 5 to represent the adjacent point. Then, samples 4 and 6 can constitute a sample pair, samples 3 and 7 can constitute a sample pair, and samples 2 and 8 can constitute a sample pair. In this way, three sample pairs are obtained.

[0102] For each sample pair, the third distance between the positive and negative samples in the sample pair can be calculated. The smaller the third distance, the better. Therefore, we can construct a loss function based on the third distance to determine the second loss of the recognition network. The second loss is negatively correlated with the third distance, that is, the larger the third distance, the smaller the second loss. Then, based on the second loss, after updating the parameters of the recognition network, the second loss can be made smaller, thereby improving the ability to distinguish between highlight video clips and non-highlight video clips in adjacent areas, and improving the accuracy of highlight video recognition.

[0103] Please refer to formula (2), which is a method for determining the second loss provided by the embodiment of the present disclosure. The loss function of .

[0104]

[0105] Where Y is the set of all sample pairs, p is a single sample, and d(p) represents the Euclidean distance between the positive and negative samples in sample pair p. Margin is a hyperparameter. In one example, margin = 0.7 can be set. Margin can be considered the target distance between the positive and negative samples in a sample pair. When this target distance is reached, network updates can be stopped. Of course, the larger the target distance, the better. For the regularized Euclidean distance, the maximum value is 1, so the maximum distance for margin can be 1.

[0106] Obviously, in formula (2), the second loss The value of is inversely proportional to the third distance, that is, the greater the distance between positive and negative samples, the smaller the third loss. Therefore, updating the recognition network based on the loss obtained by formula (2) can improve the recognition network's ability to distinguish between adjacent highlight video segments and non-highlight video segments, and improve the detection accuracy of highlight video segments at highlight boundaries.

[0107] In one possible implementation, the method further includes: when the video segment in the video to be identified has a label, determining a third loss based on the difference between the recognition result and the label of the video segment; based on the third loss, updating the parameters of the recognition network so that the recognition result output by the recognition network is consistent with the label.

[0108] After the recognition network performs highlight recognition on the video sub-features, it will obtain a recognition result, which can be the result of whether each video clip belongs to a highlight video. The label of the video clip is used to characterize whether the video clip is actually a highlight video. Then, based on the difference between the recognition result and the label of the video clip, the third loss determined can characterize the gap between the prediction result of the recognition network and the expected result. When the recognition network is accurate, the prediction result should be consistent with the expected result. Then, based on the third loss, the parameters of the recognition network are updated, which can make the recognition result output by the recognition network consistent with the label, thereby improving the accuracy of the output result of the recognition network.

[0109] In step S11, the video features of the video to be identified are extracted, and the video features include video sub-features of multiple video clips arranged in time sequence. In one example, the video sub-features can be synthesized by three features, which are the encoded splicing features The visual fusion feature and audio fusion features The specific process of obtaining the three features can be found in the feature extraction method provided in this disclosure, which will not be described here in detail.

[0110] For the above three features, the recognition network can predict the highlight video separately and obtain three recognition results. Here, the encoded splicing features are The recognition result is represented as Visual fusion features The recognition result is represented as Audio fusion features The recognition result is represented as Then, based on the losses between these three recognition results and the label of the video clip, a third loss is determined.

[0111] Please refer to formula (3), which is a method for determining the third loss provided by the embodiment of the present disclosure. The loss function of .

[0112]

[0113] Where y is the label of the video clip.

[0114] In one possible implementation, updating the parameters of the recognition network includes: performing a weighted summation on at least two of the first loss, the second loss, and the third loss to obtain a target loss; and updating the recognition network based on the target loss to reduce the target loss.

[0115] In this implementation, the identification network may be updated by more than one of the first loss, the second loss, and the third loss. In one example, the identification network may be updated by the first loss and the third loss; in another example, the identification network may be updated by the second loss and the third loss; in another example, the identification network may be updated by the first loss, the second loss, and the third loss.

[0116] The target loss can be obtained by performing a weighted summation on the losses. The weights of the losses can be the same, for example, each weight is 1 / 3, or the weights can be different, which is not limited here.

[0117] Please refer to formula (4), which is a method for determining target loss provided by the embodiment of the present disclosure. The loss function of .

[0118]

[0119] Among them, λ1, λ2 and λ3 are weights.

[0120] In the disclosed embodiment, a target loss is obtained by weighted summing at least two of the first, second, and third losses. Based on the target loss, the recognition network is updated to minimize the target loss. Thus, updating the recognition network using multiple losses can improve the accuracy of highlight detection.

[0121] See also Figure 2 , is a schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as Figure 2 The video features contain multiple sub-features. For each sub-feature, a first loss is calculated to bring samples with the same label closer together and samples with different labels farther apart. A second loss is calculated by selecting highlight and non-highlight video clips before and after the adjacent area as difficult samples. A third loss is calculated using the highlight estimation results and labels based on the video clip features. The recognition network is updated based on the first and second losses and the third impairment.

[0122] As a method for model training, the disclosed embodiment can be more flexibly embedded into various existing highlight detection networks and can be used as an insertion module to improve the accuracy of the recognition network in distinguishing highlight video clips from non-highlight video clips.

[0123] In one possible implementation, extracting video features of a video to be identified includes: obtaining visual features, audio features, and a splicing feature of the video to be identified after encoding the visual features and the audio features; wherein each of the video clips corresponds to at least one visual sub-feature, audio sub-feature, and splicing sub-feature; and fusing the visual features, audio features, and splicing features to obtain video features, wherein at least one visual sub-feature, audio sub-feature, and splicing sub-feature corresponding to each of the video clips are fused to obtain the video sub-feature.

[0124] Visual features are extracted from the visual characteristics of the video to be identified, which can represent the video to be identified. Audio features are extracted from the auditory characteristics of the video to be identified. And splicing features are derived by splicing and encoding the visual and auditory features. By fusing these three features, the resulting video features can better represent the video to be identified from multiple levels, greatly enhancing the feature representation ability and improving the accuracy of identifying highlight video clips.

[0125] It should be noted that the visual features here can be visual fusion features obtained by performing attention extraction and fusion on the extracted initial visual features. The specific process of determining the visual fusion feature can refer to the possible implementation methods provided in this disclosure; the audio feature here can be the audio fusion feature obtained by performing attention extraction and fusion on the extracted initial audio features. The specific process of determining the audio fusion feature can refer to the possible implementation methods provided in this disclosure; the splicing feature here can be obtained by splicing the visual features and audio features head to tail, then encoding based on self-attention, splitting at the splicing point, and then splicing the sub-features corresponding to the same segment. Specifically determine the splicing features The process can be referred to the possible implementation methods provided in this disclosure and will not be described in detail here.

[0126] The visual features, audio features, and splicing features all include sub-features of multiple video clips arranged in time sequence. Therefore, when fusing these visual, audio, and splicing features, the sub-features corresponding to the same video clip within these three features can be fused to obtain video sub-features. The sub-features of the multiple video clips can then constitute the video features of the video to be identified. As an example, the sub-feature fusion can be performed by performing an addition and summation operation on the sub-features.

[0127] The following describes in detail the method for extracting video features of the video to be identified.

[0128] In one possible implementation, extracting video features of a video to be identified includes:

[0129] In step S21, visual features and audio features of a video to be identified are extracted, wherein the video to be identified is divided into multiple video segments, the visual features include visual sub-features of the multiple video segments arranged in time sequence, and the audio features include audio sub-features of the multiple video segments arranged in time sequence;

[0130] In step S22, the visual features and the audio features are respectively encoded to obtain visual coding features and audio coding features;

[0131] In step S23, the visual coding features and the audio coding features are concatenated head-to-tail to obtain a concatenated feature;

[0132] In step S24, feature encoding is performed on the splicing features based on a self-attention mechanism to obtain encoded splicing features.

[0133] In one possible implementation, the extracting of visual features and audio features of a video to be identified includes: segmenting the video to be identified to obtain multiple video segments; extracting image features of each video frame in the multiple video segments; superimposing the image features of each video frame in a single video segment to obtain video sub-features of the single video segment; and arranging the video sub-features corresponding to each video segment in chronological order to obtain visual features.

[0134] The video to be identified can be expressed as V = {v t} T t＝1 , where v t For the continuous video clips sampled in the video to be identified, the visual features F of the video can be extracted through the trained visual feature extraction network (Inflated 3D ConvNet, I3D) v , using the trained audio network (Pretrained Audio Neural-Networks, PANN) to extract the audio features F in the video a .

[0135] The visual sub-features and audio sub-features of each video clip are flattened into a feature vector, and then transformed into the same embedding space using the linear layer of the neural network. Therefore, the visual features and audio features of the entire video are represented as F v ={f1 v ,...,f T v}∈R T×d and F a={f1 a ,...,f T a}∈R T×d , where the two dimensions of the embedding space R are the temporal dimension T and the feature dimension d. The specific value of the temporal dimension T is the number of video segments obtained by segmentation. The feature dimension can be 256, for example. T v is the visual sub-feature, f T a is the audio sub-feature.

[0136] In this implementation, the image features of each video frame in the multiple video clips are extracted, and the image features of each video frame in a single video clip are superimposed to obtain video sub-features of the single video clip. Thus, the obtained video sub-features can more accurately represent all video frames in the video clip. Then, the video sub-features corresponding to each video clip are arranged in time sequence to obtain visual features, which facilitates the subsequent encoding of visual features through the attention mechanism based on the context of the video sub-features, so as to accurately extract key information in the visual features, thereby improving the accuracy of highlight video clip recognition.

[0137] In step S22, the visual features and audio features are encoded to obtain visual coding features and audio coding features. That is, after obtaining the visual features and audio features, they can be encoded to further extract key information from the visual features and audio features. For example, global contextual features can be extracted from the visual features, and global contextual features can be extracted from the audio features. In addition, other features besides global contextual features can also be extracted, which is not limited in this disclosure.

[0138] In one possible implementation, the respectively encoding of the visual features and the audio features to obtain visual coding features and audio coding features includes: extracting a first global context feature of each visual sub-feature in the visual features; fusing each of the first global context features with the corresponding visual sub-feature to obtain multiple first visual sub-features as the visual coding features; extracting a second global context feature of each audio sub-feature in the audio features; fusing each of the second global context features with the corresponding audio sub-feature to obtain multiple first audio sub-features as the audio coding features.

[0139] The first global context feature here is used to characterize the contextual association information between the visual sub-features in the visual feature. Fusion of the first global context feature with the visual sub-feature can enhance the feature representation of the visual sub-feature, thereby obtaining the visual coding feature, which can improve the accuracy of the final highlight video clip recognition.

[0140] The second global context feature here is used to characterize the contextual association information between the audio sub-features in the audio feature. Fusion of the second global context feature with the audio sub-feature can enhance the feature representation of the audio sub-feature, thereby obtaining the audio coding feature, which can improve the accuracy of the final highlight video clip recognition.

[0141] In one possible implementation, multiple visual sub-features in a visual feature can be encoded based on a self-attention mechanism to extract contextual association information between multiple visual sub-features in the visual feature, and obtain multiple first visual sub-features as visual coding features; similarly, multiple audio sub-features in an audio feature can be encoded based on a self-attention mechanism to extract contextual association information between multiple audio sub-features in the audio feature, and obtain multiple first audio sub-features as audio coding features.

[0142] Specifically, for the input of the encoder layer i First, the input features are converted into three parts, namely the query Q v , key K v , value S v , the specific conversion process can be found in formula (5). Then, the first global context feature is captured through the multi-head dot product attention mechanism, and the first global context feature is fused with the visual sub-feature. For details, see formula (6). Then, a feedforward network (FFN) is used for transformation and nonlinear processing. For details, see formula (7).

[0143]

[0144]

[0145]

[0146] in, is the visual encoding feature output of the encoder layer i, W i q 、W i k and W i s are trainable network parameters, d k Q v feature dimension.

[0147] Similarly, for audio coding features The encoding can also be performed through the above process, which will not be described here.

[0148] In the above step S23, the visual coding features and the audio coding features can be spliced end to end to obtain a spliced feature; since the visual features include visual sub-features of multiple video clips arranged in time sequence, and the audio features include audio sub-features of multiple video clips arranged in time sequence, after the visual features and audio features are encoded respectively, the obtained visual coding features can also be multiple first visual sub-features arranged in time sequence, and the audio coding features can also be multiple first audio sub-features arranged in time sequence.

[0149] The visual coding features and audio coding features are spliced end to end, and the splicing position can be the last feature in the visual coding feature time sequence and the first feature in the audio coding feature time sequence, or the first feature in the visual coding feature time sequence and the last feature in the audio coding feature time sequence.

[0150] For example, the visual encoding feature can be expressed as F n v ={f1 v ′,...,fT v ′}, where f T v ′ is the Tth first visual sub-feature, and the audio coding feature can be expressed as F n a ={f1 a′ ,...,f T a′}, where f T a′ is the Tth first audio sub-feature.

[0151] Then, the visual coding features and audio coding features are concatenated head to tail, and the concatenated features can be expressed as F av ={f1 v ′,...,f T v ′,f1 a′ ,...,f T a′}.

[0152] Obviously, after the visual coding features and audio coding features with a time dimension length of T are concatenated head to tail, the length of the concatenated features in the time dimension is 2T.

[0153] In the above step S24, the splicing feature can be feature encoded based on the self-attention mechanism to obtain the encoded splicing feature; when the splicing feature is feature encoded based on the self-attention mechanism, the splicing feature is taken as a whole, and the sub-features in the whole are encoded. Through this process, not only the context features between the first visual sub-features and the context features between the first audio sub-features are extracted, but also the splicing feature is taken as a whole, and the global context features between the first visual sub-features and the first audio sub-features are extracted from the overall perspective.

[0154] Specifically, the self-attention mechanism encodes the entire sequence consisting of the first visual sub-feature and the first audio sub-feature to produce an encoded concatenated feature. For ease of description, the feature updated from the first visual sub-feature in the encoded concatenated feature is referred to as the second visual sub-feature, and the feature updated from the second visual sub-feature is referred to as the second audio sub-feature.

[0155] In this way, in the encoded splicing features, the second visual sub-feature will include context features with the second audio sub-feature; and the second audio sub-feature will also include context features with the second visual sub-feature.

[0156] In one possible implementation, the feature encoding of the splicing feature based on the self-attention mechanism to obtain the encoded splicing feature includes: extracting a third global context feature of each splicing sub-feature in the splicing feature, wherein the splicing sub-feature is the first visual sub-feature or the first audio sub-feature; and fusing each of the third global context features with the corresponding splicing sub-feature to obtain the encoded splicing feature.

[0157] For the convenience of description, the sub-features in the splicing feature are called splicing sub-features here. Since the splicing feature is obtained by splicing the visual coding feature and the audio coding feature, and the visual coding feature contains the first visual sub-feature, and the audio coding feature contains the first audio sub-feature, the splicing sub-feature is the first visual sub-feature or the first audio sub-feature.

[0158] The third global context feature here is used to characterize the contextual association information between each splicing sub-feature in the splicing feature. Fusion of the third global context feature with the corresponding splicing sub-feature can enhance the feature representation of each splicing sub-feature, thereby improving the accuracy of the final highlight video clip recognition.

[0159] Since audio features and visual features belong to two different domains, the domain gap between the two domains is very large, and the features of the two domains are expressed differently. At a certain moment in the video, the visual features may be highlight content, but the music may be calm, that is, the audio features are not highlight content, because the audio may be background music, which may have been processed and may be unrelated to the video's visual effects. For example, the video screen may be jittery for some reason, and the video may not contain highlight content, but the audio may indicate that something interesting happened at that moment, that is, there was highlight content at that moment. In other words, the visual features and audio features may not be aligned in terms of highlight time.

[0160] Therefore, directly superimposing the visual features and audio features at the same moment to estimate the highlight video clip may not be effective.

[0161] In this implementation, by concatenating the visual and audio coding features end-to-end and calculating the global contextual features between the sub-features in the concatenated whole, we can capture some of the relevant connections between the visual and audio coding features, allowing the visual coding features to capture the audio coding features, and vice versa. This reduces the domain gap between visual and audio, and improves the accuracy of highlight segment recognition.

[0162] Accordingly, in step S12, highlight video is identified based on the video sub-features of the plurality of video clips to obtain an identification result. After obtaining the encoded splicing features, highlight video clips in the plurality of video clips can be identified based on the encoded splicing features. For example, highlight video clips can be identified using a trained network. The specific identification process can be referred to the possible implementation methods provided in this disclosure and will not be described in detail here.

[0163] In an embodiment of the present disclosure, by extracting visual features and audio features of a video to be identified, the video to be identified is divided into multiple video clips, the visual features include visual sub-features of multiple video clips arranged in time sequence, and the audio features include audio sub-features of multiple video clips arranged in time sequence; the visual features and audio features are encoded respectively to obtain visual encoding features and audio encoding features; the visual encoding features and audio encoding features are spliced end to end to obtain splicing features; feature encoding is performed on the splicing features based on a self-attention mechanism to obtain encoded splicing features; highlight videos are identified based on the video sub-features of multiple video clips to obtain recognition results. By encoding the visual features and audio features of the video to be identified, visual coding features and audio coding features are obtained, and then the visual coding features (time-sequentially arranged visual sub-features) and audio coding features (time-sequentially arranged audio sub-features) are spliced end to end to obtain spliced features. The spliced features are feature encoded through the self-attention mechanism to obtain the encoded spliced features. This can effectively map the audio and video modal features to the same feature space, and the features of the two different modalities gradually establish correlation on the same distribution, effectively alleviating the problem of domain misalignment between audio and visual features, and creating favorable conditions for the fusion of different modal information. Then, based on the video sub-features of multiple video clips, highlight videos are identified to obtain recognition results, which can improve the accuracy of the identified highlight video clips.

[0164] In one possible implementation, the highlight video is identified based on the video sub-features of the multiple video clips to obtain an identification result, including: splitting the encoded splicing features at the splicing position to obtain a second visual sub-feature and a second audio sub-feature; fusing the second visual sub-feature and the second audio sub-feature corresponding to the same video clip to obtain multiple fused sub-features; and determining, based on the fused sub-features, whether the clip corresponding to the fused sub-feature is a highlight video clip.

[0165] In the encoded splicing feature, the second visual sub-feature and the second audio sub-feature capture some related information of each other. When identifying highlight video clips, the encoded splicing feature can be disassembled. The position of the disassembly is the position of the splicing in step S23. The two features obtained after the disassembly are respectively composed of the second visual sub-feature and the second audio sub-feature. These two features have the same length in the time dimension, both T.

[0166] The two separated features can be fused together to obtain a fused sub-feature. This fused sub-feature includes both the visual feature (the second visual sub-feature) and the audio feature (the second audio sub-feature) of the video clip. The fusion can be performed by adding the second visual sub-feature and the second audio sub-feature of the same video clip.

[0167] When determining whether the segment corresponding to the fused sub-feature is a highlight video segment based on the fused sub-feature, the fused sub-feature can be input into the trained network classification layer to determine whether the segment is a highlight video segment. In essence, it can be a binary classification network to determine whether the input feature is a highlight video segment, or it can output a score between 0 and 1. The higher the score, the greater the possibility that the video segment is a highlight video segment. For the sake of ease of description, the recognition result based on the encoded splicing feature output is referred to as the first recognition result.

[0168] In one possible implementation, after encoding the visual features and audio features respectively to obtain visual coding features and audio coding features, the method further includes: fusing the global highlight features extracted from the visual features and audio features with the corresponding visual coding features and audio coding features respectively to obtain visual fusion features and audio fusion features; identifying highlight videos based on the video sub-features of the multiple video clips to obtain recognition results, including: identifying highlight video clips among the multiple video clips based on the encoded splicing features, the visual fusion features and the audio fusion features.

[0169] In one possible implementation, the method for extracting global highlight features from the visual features and audio features includes: based on a cross-attention mechanism, utilizing global highlight embedding to respectively extract the global highlight features from the visual coding features and the audio coding features, wherein the global highlight embedding is a vector obtained through training that globally abstractly summarizes the highlight features.

[0170] The global highlight embedding is a trained vector that abstractly summarizes highlight features globally. The specific training process can be found in the possible implementations provided in this disclosure and will not be described in detail here. The global highlight embedding can be considered as the common features of highlight features in various videos obtained by abstracting and summarizing the highlight features in various videos.

[0171] Taking visual coding features as an example, in the process of extracting global highlight features of visual coding features by using global highlight embedding, the visual coding features can be And the global highlight embedding G is used as the input of the cross attention mechanism. Specifically, G can be used as the query query and the visual encoding feature Treated as value, query and aggregate visual encoding features The visual context information in the image is used to obtain the global highlight feature Gv. Then, the visual encoding feature With the global highlight feature G v The summation is the visual fusion feature The process can be expressed as

[0172] Taking audio coding features as an example, in the process of extracting global highlight features of audio coding features by using global highlight embedding, the audio coding features can be And the global highlight embedding G is used as the input of the cross attention mechanism. Specifically, G can be used as the query query and the audio encoding feature Treated as value, query and aggregate audio encoding features Visual context information in the image is used to obtain the global highlight feature G a Then, the audio encoding features With the global highlight feature G a The sum is the audio fusion feature The process can be expressed as

[0173] In one possible implementation, the global highlight embedding G is obtained through training. G is initially initialized to a value. During training, the initialized G is After the cross-attention mechanism operation, G' is obtained, and then G is updated based on the gradient of G', and the next cross-attention operation is continued to continuously iterate and update G.

[0174] In this implementation, global highlight features are extracted from the visual coding features and audio coding features respectively using global highlight embedding based on a cross-attention mechanism, and the global highlight features are fused with the corresponding visual coding features and audio coding features respectively to obtain visual fusion features and audio fusion features. Since the global highlight embedding is a trained vector that globally abstracts and summarizes the highlight features, the cross-attention mechanism is used to abstract and summarize the unimodal sequence into a global feature. This global feature can be considered as the highlight feature in the visual coding features and audio coding features. The resulting visual fusion features and audio fusion features can greatly enhance the feature representation capability and improve the accuracy of the identified highlight video clips.

[0175] Since the visual fusion features and the audio fusion features represent the features of the visual and audio modalities respectively, and the encoded splicing features fully exploit the contextual information between the audio and video modalities, identifying highlight video clips in multiple video clips based on the encoded splicing features, the visual fusion features and the audio fusion features can improve the accuracy of the identified highlight video clips.

[0176] In a possible implementation, the identifying of highlight video segments among the multiple video segments based on the encoded splicing features, the visual fusion features, and the audio fusion features includes: obtaining a first recognition result based on the encoded splicing features; obtaining a second recognition result based on the visual fusion features; obtaining a third recognition result based on the audio fusion features; and performing weighted fusion on the first recognition result, the second recognition result, and the third recognition result to obtain a recognition result of the highlight segment.

[0177] Based on the encoded splicing features, the first recognition result is obtained The process can be found in the previous description and will not be described here.

[0178] Accordingly, the visual fusion features and audio fusion features Also input into the trained network classification layer to obtain the second recognition result and the third recognition result In one example, and It can be a score between 0 and 1. The higher the score, the more likely the video clip is a highlight video clip. and The three scores are weighted and summed to obtain a weighted sum score, and then based on the weighted sum score, whether the video clip is a highlight video clip is determined. The weights of the three scores can be the same or different, and this disclosure does not limit this.

[0179] In another example, and It can also be used to determine whether the video clip is a highlight video clip. In the case of accuracy priority, when and When both indicate that the video clip is a highlight clip, it can be determined that the video clip is a highlight video clip.

[0180] In addition, the encoded splicing features, the visual fusion features, and the audio fusion features may be weightedly fused first, and then the weighted fusion features may be identified to obtain highlight video segments from multiple video segments.

[0181] See also Figure 3 , is a schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as Figure 3 As shown, through the visual feature extraction network E v Extract the visual features of the video clip to obtain the visual features F v , through the audio feature extraction network E a Extract the audio features of the video clip to obtain the audio features F a , Figure 3 In the visual feature F v Each small grid represents the visual sub-feature of a video clip, and the audio feature F a Each small grid in represents the audio sub-feature of a video clip.

[0182] Then, the self-attention mechanism is used to encode the visual features F v and audio feature F a Encode and obtain visual encoding features and audio coding features Then, based on the cross attention mechanism, the global highlight embedding G is used to extract the machine vision encoding features. and audio coding features The global highlight feature in the image is fused with the corresponding visual coding feature and audio coding feature to obtain the visual fusion feature. and audio fusion features

[0183] Then, through the co-occurrence encoding module, and Perform head-to-tail splicing and then encode based on self-attention to obtain Then at the joint After splitting and then splicing, we get

[0184] Then, yes and The video feature is constructed by concatenating the subfeatures corresponding to the same video clip. The video feature contains multiple subfeatures. For each subfeature, a first loss is calculated to bring samples with the same label closer together and samples with different labels farther apart. A second loss is calculated by selecting highlight and non-highlight video clips before and after the adjacent region as difficult samples. A third loss is calculated using the highlight estimation results and labels based on the video clip features. The recognition network is then updated based on the first and second losses and the third impairment.

[0185] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0186] In addition, the present disclosure also provides a highlight video recognition device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any highlight video recognition method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0187] Figure 4 A block diagram of a highlight video recognition device according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the device 30 includes:

[0188] An extraction module 31 is configured to extract video features of a video to be identified, wherein the video to be identified includes a plurality of video segments arranged in time sequence, and each of the video segments includes at least one video sub-feature;

[0189] an identification module 32 for identifying highlight videos based on the video sub-features of the plurality of video clips to obtain an identification result;

[0190] The recognition network performs parameter updates based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in the feature space, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update.

[0191] In a possible implementation, the apparatus further includes:

[0192] A first parameter updating module is configured to determine a first loss of the recognition network based on the first distance and the second distance, wherein the first loss is positively correlated with the first distance and negatively correlated with the second distance; and based on the first loss, update the parameters of the recognition network to reduce the first loss.

[0193] In a possible implementation, the apparatus further includes:

[0194] The second parameter updating module is used to determine the adjacent part of the highlight video segment and the non-highlight video segment in the video segment; based on the third distance between the video sub-features of the highlight video segment and the non-highlight video segment before and after the adjacent part, update the parameters of the recognition network so that the distance between the video sub-features of the highlight video segment and the non-highlight video segment output by the recognition network is greater than the distance before the update.

[0195] In a possible implementation, the video to be identified is a sample video, and the video segment is annotated with a label for indicating whether the video segment is a highlight;

[0196] The second parameter updating module is used to determine the adjacent part of the highlight video segment and the non-highlight video segment based on the label of the video segment; and construct positive and negative sample pairs based on the highlight video segment and the non-highlight video segment before and after the adjacent part.

[0197] In a possible implementation, the second parameter updating module is used to construct a plurality of positive and negative sample pairs based on a plurality of highlight video clips and a plurality of non-highlight video clips before and after the adjacent portion, wherein a single positive and negative sample pair includes one highlight video clip and one non-highlight video clip; determine a second loss of the recognition network based on a third distance between video sub-features of the highlight video clip and the non-highlight video clip in each positive and negative sample pair, wherein the second loss is negatively correlated with the third distance; and update the parameters of the recognition network based on the second loss to reduce the second loss.

[0198] In a possible implementation, the video to be identified is a sample video, the video segment is annotated with a label indicating whether the video segment is a highlight, and the apparatus further includes:

[0199] A third parameter updating module is used to determine a third loss based on the difference between the recognition result and the label of the video clip; based on the third loss, update the parameters of the recognition network so that the recognition result output by the recognition network is consistent with the label.

[0200] In one possible implementation, the device further includes a fourth parameter updating module, configured to perform a weighted summation on at least two of the first loss, the second loss, and the third loss to obtain a target loss; and update the recognition network based on the target loss to reduce the target loss.

[0201] In one possible implementation, the extraction module is used to extract the visual features, audio features and splicing features of the video to be identified after encoding the visual features and audio features; wherein each of the video clips corresponds to at least one visual sub-feature, audio sub-feature and splicing sub-feature; the visual features, audio features and splicing features are fused to obtain video features, wherein at least one visual sub-feature, audio sub-feature and splicing sub-feature corresponding to each of the video clips are fused to obtain the video sub-feature.

[0202] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0203] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0204] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0205] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0206] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0207] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0208] Figure 5 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other terminal device.

[0209] Reference Figure 5 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0210] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0211] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0212] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0213] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0214] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0215] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0216] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0217] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as wireless network (Wi-Fi), second generation mobile communication technology (2G), third generation mobile communication technology (3G), fourth generation mobile communication technology (4G), long term evolution (LTE) of universal mobile communication technology, fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0218] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0219] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions. The computer program instructions can be executed by the processor 820 of the electronic device 800 to perform the above method.

[0220] The present disclosure relates to the field of augmented reality. By acquiring image information of a target object in a real-world environment, the relevant features, states, and attributes of the target object are detected or identified using various vision-related algorithms, thereby achieving an AR effect that combines virtual and real life and matches the specific application. For example, the target object may be a face, limbs, gestures, movements, etc. related to the human body, or an identifier or marker related to an object, or a sandbox, display area, or display items related to a venue or location. Vision-related algorithms may involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and object pose or depth detection. Specific applications can involve not only interactive scenarios such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effects display, and virtual model display. Detection or identification of the relevant features, states, and attributes of the target object can be achieved using a convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0221] Figure 6 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0222] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0223] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0224] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0225] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0226] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0227] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0228] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0229] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0230] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0231] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0232] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0233] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0234] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0235] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0236] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying highlight videos, characterized in that: Applied to identification networks, including: Extracting video features of a video to be identified, wherein the video to be identified includes a plurality of video clips arranged in time sequence, and each of the video clips includes at least one video sub-feature; Identifying highlight videos based on the video sub-features of the plurality of video clips to obtain an identification result; The recognition network performs parameter update based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in the feature space, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update; The method further comprises: determining, in the video clip, a junction between a highlight video clip and a non-highlight video clip; Based on a third distance between the video sub-features of the highlight video clip and the non-highlight video clip before and after the adjacent portion, the parameters of the recognition network are updated so that the distance between the video sub-features of the highlight video clip and the non-highlight video clip output by the recognition network is greater than the distance before the update.

2. The method according to claim 1, characterized in that Parameter updating is performed based on a first distance between video sub-features of video clips with the same label in the feature space and a second distance between video sub-features of video clips with different labels, including: determining a first loss of the identification network according to the first distance and the second distance, wherein the first loss is positively correlated with the first distance and negatively correlated with the second distance; Based on the first loss, parameters of the recognition network are updated to reduce the first loss.

3. The method according to claim 1, characterized in that The video to be identified is a sample video, and the video clip is marked with a label for indicating whether the video clip is a highlight; The determining of the adjacent portion between the highlight video segment and the non-highlight video segment in the video segment includes: Determining, based on the labels of the video segments, the adjacency between the highlight video segment and the non-highlight video segment; Based on the highlight video segments and the non-highlight video segments before and after the adjacent portion, positive and negative sample pairs are constructed.

4. The method according to claim 2, characterized in that The updating of the parameters of the recognition network based on the third distance between the video sub-features of the highlight video segments and the non-highlight video segments before and after the adjacent portion includes: constructing a plurality of positive and negative sample pairs based on a plurality of highlight video segments and a plurality of non-highlight video segments before and after the adjacent portion, wherein a single positive and negative sample pair includes one highlight video segment and one non-highlight video segment; determining a second loss of the recognition network according to a third distance between the video sub-features of the highlight video segment and the non-highlight video segment in each positive and negative sample pair, where the second loss is negatively correlated with the third distance; Based on the second loss, the parameters of the recognition network are updated to reduce the second loss.

5. The method according to claim 4, characterized in that The video to be identified is a sample video, and the video clip is marked with a label for indicating whether the video clip is a highlight. The method further includes: determining a third loss based on a difference between the recognition result and the label of the video clip; Based on the third loss, the parameters of the recognition network are updated so that the recognition result output by the recognition network is consistent with the label.

6. The method according to claim 5, characterized in that The updating of the parameters of the identification network includes: Performing a weighted summation on at least two of the first loss, the second loss, and the third loss to obtain a target loss; Based on the target loss, the recognition network is updated to reduce the target loss.

7. The method according to any one of claims 1 to 6, characterized in that: Extract video features of the video to be identified, including: Obtaining visual features, audio features, and a splicing feature of the video to be identified after encoding the visual features and the audio features; wherein each video segment corresponds to at least one visual sub-feature, audio sub-feature, and splicing sub-feature; The visual features, audio features and splicing features are fused to obtain video features, wherein at least one visual sub-feature, audio sub-feature and splicing sub-feature corresponding to each video clip is fused to obtain the video sub-feature.

8. A highlight video recognition device, characterized in that: Applied to identification networks, including: An extraction module, configured to extract video features of a video to be identified, wherein the video to be identified comprises a plurality of video clips arranged in time sequence, and each of the video clips comprises at least one video sub-feature; an identification module, configured to identify highlight videos based on the video sub-features of the plurality of video clips to obtain an identification result; The recognition network performs parameter update based on a first distance between video sub-features of video clips with the same label and a second distance between video sub-features of video clips with different labels in the feature space, so that the distance between video sub-features of video clips with the same label output by the updated recognition network is smaller than the distance before the update, and the distance between video sub-features of video clips with different labels is larger than the distance before the update; The device further comprises: The second parameter updating module is used to determine the adjacent part of the highlight video segment and the non-highlight video segment in the video segment; based on the third distance between the video sub-features of the highlight video segment and the non-highlight video segment before and after the adjacent part, update the parameters of the recognition network so that the distance between the video sub-features of the highlight video segment and the non-highlight video segment output by the recognition network is greater than the distance before the update.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images