Video clustering method, device, and storage medium

Abstract

Embodiments of the present disclosure provide a video clustering method, a device, and a storage medium. The method comprises: obtaining all videos to be classified of any user from a video library; for each video to be classified among all of the videos to be classified, using a preset feature extraction model to extract a feature for determining an inter-video context relationship; on the basis of the feature of each video to be classified and a preset video context discrimination model, determining whether a context relationship exists between any two videos to be classified; and clustering and sorting videos to be classified having a context relationship, to obtain at least one video group. According to the embodiments of the present disclosure, clustering and sorting videos to be classified having a context relationship are automatically implemented on the basis of a preset model, without requiring a user to perform operations such as manually annotating labels for, classifying, or sorting the videos to be classified, so that the clustering and sorting efficiency of the videos to be classified is increased and the costs are reduced. In addition, the clustering and sorting of the videos to be classified provide convenience for video viewers to watch videos continuously, so that the video watching experience is improved.

Description

Video clustering method, device and storage medium

[0001] This disclosure claims priority to Chinese Patent Application No. 202311036112.8 filed on August 16, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this disclosure. Technical Field

[0002] The embodiments of the present disclosure relate to a video clustering method, device, and storage medium. Background Art

[0003] With the development of network technology and the popularization of smart devices, video has gradually become an important channel for people to entertain themselves and obtain information. Often, some videos belong to the same series, such as different episodes of the same TV series, or a long video split into different short videos. Therefore, it is necessary to cluster videos so that viewers can watch videos according to clusters.

[0004] In related technologies, video uploaders are usually required to label the uploaded videos and cluster them according to the labels. However, manual labeling and manual clustering are inefficient and cannot achieve rapid clustering of videos.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a video clustering method, device, and storage medium to achieve rapid clustering and sorting of videos to be classified.

[0007] In a first aspect, an embodiment of the present disclosure provides a video clustering method, comprising:

[0008] Get all videos to be classified by any user from the video library;

[0009] For each of all videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos;

[0010] Based on the characteristics of each video to be classified and the preset video context discrimination model, determine whether there is a contextual relationship between any two videos to be classified;

[0011] The videos to be classified that have contextual relationships are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted in contextual order.

[0012] In a second aspect, an embodiment of the present disclosure provides a video clustering device, including:

[0013] An acquisition unit, used to acquire all videos to be classified of any user from the video library;

[0014] A feature extraction unit is used to extract features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model;

[0015] A context discrimination unit is used to determine whether there is a contextual relationship between any two videos to be classified based on the characteristics of each video to be classified and a preset video context discrimination model;

[0016] The clustering unit is used to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted in context order.

[0017] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0018] The memory stores computer-executable instructions;

[0019] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video clustering method described in the first aspect and various possible designs of the first aspect.

[0020] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video clustering method described in the first aspect and various possible designs of the first aspect is implemented.

[0021] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the video clustering method as described in the first aspect and various possible designs of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] FIG1 is a schematic diagram of an application scenario of a video clustering method provided by an embodiment of the present disclosure;

[0024] FIG2 is a flow chart of a video clustering method according to an embodiment of the present disclosure;

[0025] FIG3 is a schematic diagram of feature extraction provided by an embodiment of the present disclosure;

[0026] FIG4 is a schematic diagram of a newly added video provided by an embodiment of the present disclosure;

[0027] FIG5 is a schematic diagram of a video deletion situation provided by an embodiment of the present disclosure;

[0028] FIG6 is a structural block diagram of a video clustering device provided by an embodiment of the present disclosure;

[0029] FIG7 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0031] At present, there are some videos that usually belong to the same series, such as different episodes of the same TV series, or a long video divided into different short videos, etc. Therefore, it is necessary to cluster the videos to facilitate video viewers to watch videos according to clustering, especially to watch videos in sequence. For example, after watching a certain video, the video viewer will continue to watch the next video connected to the video.

[0032] In related technologies, video uploaders are usually required to label the uploaded videos and cluster them according to the labels. However, manual labeling and manual clustering are inefficient and cannot achieve rapid clustering of videos. In addition, video serial numbers are added to the video titles, and then the videos are sorted according to the video serial numbers after clustering. Similarly, labeling video serial numbers also requires manpower and is inefficient. In addition, if the video serial number is wrong, it may cause errors in the sorting of the videos after clustering.

[0033] To address the above technical problems, the present disclosure provides a video clustering method that aims to quickly and efficiently cluster videos and also to sort the clustered videos in contextual order. Considering that a machine learning model can be trained to perform video classification, the present disclosure aims to cluster and sort videos using a machine learning model. A feature extraction model and a video context discrimination model can be pre-trained. For all videos to be classified by any user, the feature extraction model can be used to extract features for determining contextual relationships between videos. The features of each video to be classified are then input into the video context discrimination model to determine whether any two videos to be classified have a contextual relationship; and videos to be classified that have a contextual relationship are clustered and sorted. The present disclosure automatically clusters and sorts videos to be classified that have a contextual relationship based on a preset model, eliminating the need for users to manually label, classify, and sort the videos to be classified. This improves the efficiency and reduces the cost of clustering and sorting videos to be classified. Furthermore, the clustering and sorting of videos to be classified facilitates continuous viewing for viewers, thereby enhancing the video viewing experience.

[0034] The video clustering method disclosed in the present invention can be applied to scenarios where servers or terminal devices cluster and sort videos, for example, as shown in Figure 1, where a preset feature extraction model and video context discrimination model are pre-deployed in the server. The server can first perform a preliminary classification of the video library based on users (video uploaders), obtain all videos to be classified by any user, and then input them into the feature extraction model to extract features of each video to be classified for judging the contextual relationship between videos, input the features of each video to be classified into the video context discrimination model to judge whether there is a contextual relationship between any two videos to be classified; cluster and sort the videos to be classified with a contextual relationship to obtain at least one video group, wherein the videos included in each video group are sorted in context order.

[0035] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] The video clustering method disclosed herein will be described in detail below with reference to specific embodiments.

[0037] Referring to FIG2 , FIG2 is a flow chart of a video clustering method provided by an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server, and the video clustering method includes:

[0038] S201: Obtain all videos to be classified of any user from a video library.

[0039] In this embodiment, the video library includes videos uploaded by different users, and video clustering and sorting need to be performed on each video of the same user. Therefore, the videos to be classified included in the video library can be preliminarily classified on a user (video uploader) basis to obtain each user's videos to be classified, and then all videos to be classified of any user can be obtained, so as to perform corresponding operations on the videos to be classified of any user (video uploader) in the following steps.

[0040] S202: For each video to be classified among all videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos.

[0041] In this embodiment, all videos to be classified for any user are clustered based on contextual relationships, that is, videos with contextual relationships are divided into a video group. For example, all videos to be classified for this user include: video 1.1, video 1.2, video 1.4, video 1.5, video 2.1, video 2.2, where video 1.1, video 1.2, video 1.4, and video 1.5 belong to the same series of videos, video 1.1 and video 1.2 have a contextual relationship (adjacent), video 1.4 and video 1.5 have a contextual relationship (adjacent), and Video 1.2 and video 1.4 have no contextual relationship (not adjacent), video 2.1 and video 2.2 belong to another series of videos and have a contextual relationship (adjacent). The purpose of this embodiment is to cluster the above videos according to the contextual relationship, and 3 video groups can be obtained. Video group 1 includes video 1.1 and video 1.2, video group 2 includes video 1.4 and video 1.5 (although they belong to the same series of videos as video 1.1 and video 1.2, they cannot be divided into the same group because they have no contextual relationship), and video group 3 includes video 2.1 and video 2.2.

[0042] To achieve the above objectives, this embodiment extracts features from all videos to be classified by any user. Unlike conventional video clustering methods, the features extracted from videos in this embodiment are features used to determine the contextual relationship between videos. By comparing these features of any two videos to be classified, it can be determined whether the two videos to be classified have contextual features. If the two videos to be classified have contextual features, the order of the two videos to be classified can be further determined. The features used to determine the contextual relationship between videos can be implemented through machine learning. That is, in this embodiment, a feature extraction model is pre-trained. Through the training process, this feature extraction model is equipped with the function of extracting the features used to determine the contextual relationship between videos.

[0043] Optionally, for any video to be classified, a feature extraction model can be used to extract features for determining the contextual relationship between videos based on one or more of the visual information, audio information, title information, and video subtitles of the video to be classified. In this embodiment, as shown in FIG3 , one or more of the visual information (video image), audio information, title information, and video subtitles of the video to be classified can be obtained and input into the feature extraction model, and one or more of the visual features, audio features, title features, and subtitle features can be correspondingly extracted. The extracted features are then fused to obtain multimodal features, thereby facilitating a more accurate determination of the contextual relationship between videos.

[0044] S203: Determine whether there is a contextual relationship between any two videos to be classified based on the characteristics of each video to be classified and a preset video context discrimination model.

[0045] In this embodiment, after obtaining the features for judging the contextual relationship between videos of each video to be classified among all videos to be classified of any user, a judgment can be made as to whether there is a contextual relationship between each video based on the features of each video to be classified. In this embodiment, a video context discrimination model can be pre-trained. The video context discrimination model can output a judgment result on whether the two videos to be classified have a contextual relationship based on the features for judging the contextual relationship between videos of any two videos to be classified that are input; the features for judging the contextual relationship between videos of multiple videos to be classified can also be input in batches into the video context discrimination model to output a judgment result on whether any two videos to be classified among the multiple videos to be classified have a contextual relationship.

[0046] Among them, for the features of any two videos to be classified that are used to determine the contextual relationship between videos, the video context discrimination model can obtain the similarity of the two features, identify the size of the contextual relationship between the two videos to be classified based on the size of the similarity, and then determine whether the two videos to be classified have a contextual relationship based on the similarity, for example, determine whether the similarity value is greater than a preset threshold. If it is greater, it is determined that there is a contextual relationship, and then output the judgment result of whether the two videos to be classified have a contextual relationship.

[0047] In this embodiment, the features used to determine the contextual relationship between videos of each video to be classified among all videos to be classified by any user can be batch-inputted into the video context discrimination model. The video context discrimination model can generate a similarity matrix. Any similarity value in the similarity matrix is ​​used to identify the size of the contextual relationship between the corresponding two videos to be classified. Then, whether there is a contextual relationship between any two videos to be classified can be determined based on the similarity matrix.

[0048] It should be noted that, since the video contents with contextual relationships are connected, there is a sequence between the videos with contextual relationships, and when obtaining the similarity of the features of any two videos to be classified based on the features of the two videos to be classified, two similarity values ​​can be obtained, that is, assuming that any two videos to be classified are the first video to be classified and the second video to be classified, the similarity value corresponding to the first video to be classified and the second video to be classified, as well as the similarity value corresponding to the second video to be classified and the first video to be classified can be obtained, wherein the similarity value corresponding to the first video to be classified and the second video to be classified is used to indicate the probability that the first video to be classified and the second video to be classified have a contextual relationship, and the first video to be classified comes first and the second video to be classified comes later, and the similarity value corresponding to the second video to be classified and the first video to be classified is used to indicate the probability that the first video to be classified and the second video to be classified have a contextual relationship, and the second video to be classified comes first and the first video to be classified comes later.

[0049] Therefore, if it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified can be determined according to the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified. Specifically, if the corresponding similarity value between the first video to be classified and the second video to be classified is greater than the corresponding similarity value between the second video to be classified and the first video to be classified, then it is determined that the first video to be classified comes first and the second video to be classified comes later. If the corresponding similarity value between the first video to be classified and the second video to be classified is less than the corresponding similarity value between the second video to be classified and the first video to be classified, then it is determined that the second video to be classified comes first and the first video to be classified comes later.

[0050] S204: Cluster and sort the videos to be classified that have a contextual relationship to obtain at least one video group, wherein the videos included in each video group are sorted in context order.

[0051] In this embodiment, the above process can be used to determine whether there is a contextual relationship between any two videos to be classified, and if it is determined that the two videos to be classified have a contextual relationship, the order of the two videos to be classified, and then the videos to be classified with a contextual relationship are clustered and sorted. For example, after determining that video 3.1 and video 3.2 have a contextual relationship, and video 3.1 comes first and video 3.2 comes later, and video 3.2 and video 3.3 have a contextual relationship, and video 3.2 comes first and video 3.3 comes later, after clustering and sorting, video 3.1, video 3.2 and video 3.3 are classified into a video group, and the order is video 3.1, video 3.2 and video 3.3.

[0052] Furthermore, after obtaining at least one video group, the method further includes:

[0053] Determine whether the currently playing video is the last video in the video group;

[0054] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0055] If the currently playing video is the last video in the video group, the video will stop playing and / or display a prompt message after the currently playing video ends.

[0056] In this embodiment, before, after, or after the currently playing video ends, the terminal device of the video viewer can determine whether the currently playing video is the last video in the video group. If the currently playing video is not the last video in the video group, it means that the video group also includes a next video that is connected to (that is, has a contextual relationship with) the currently playing video. The terminal device can be controlled to continue playing the next video after the currently playing video ends until the last video in the video group is played. If the currently playing video is the last video in the video group, it means that there is no next video connected to (that is, has no contextual relationship with) the currently playing video. At this time, the playback of the currently playing video can be stopped after the playback of the currently playing video ends, and / or a prompt message can be displayed to prompt the video viewer that there is no next video connected to the currently playing video, or to prompt the video viewer to select another video group to play. Automatic video playback based on the above-mentioned video group can facilitate video viewers to watch multiple videos connected to each other continuously without having to manually search for and play the next video after one video ends. This is especially true in a video playback scenario based on a feed stream. For example, a feed stream server pushes a certain video A1 to a terminal device, and this video A1 belongs to a certain video group A. In this video group A, this video A1 has a next video A2 connected to it and has a contextual relationship with it. After video A1 is played, video A2 can be played directly. Video viewers do not need to manually search for and play video A2, which simplifies user operations and improves the video viewing experience.

[0057] The video clustering method provided in this embodiment obtains all videos to be classified of any user from a video library; for each video to be classified, a preset feature extraction model is used to extract features for determining the contextual relationship between videos; based on the features of each video to be classified and a preset video context discrimination model, it is determined whether any two videos to be classified have a contextual relationship; the videos to be classified with a contextual relationship are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted in context order. Based on the preset model, the videos to be classified with a contextual relationship are automatically clustered and sorted, without the need for the user to manually label, classify, sort, and perform other operations on the videos to be classified. This improves the efficiency of clustering and sorting the videos to be classified and reduces the cost. In addition, the clustering and sorting of the videos to be classified provides convenience for video viewers to watch videos continuously, which is conducive to improving the video viewing experience.

[0058] Based on any of the above embodiments, after obtaining at least one video group, the method further includes:

[0059] S301, obtaining the user's newly added video;

[0060] S302, extracting features for determining contextual relationships between videos from the newly added video using the feature extraction model;

[0061] S303: Determine, based on the features of the newly added video and the features of the first video and the last video in any of the video groups, whether the newly added video has a contextual relationship with the first video or the last video;

[0062] S304: If it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, the newly added video is added to the video group as the first video or the last video.

[0063] In this embodiment, after all videos to be classified of any user are clustered and sorted to obtain at least one video group, if the user uploads a new video to the video library, the user's new video can be obtained, and the same method as S202 is adopted to extract features for judging the contextual relationship between videos from the new video using the feature extraction model; and since the videos with contextual relationship have been clustered and sorted to obtain at least one video group, the new video can only have a contextual relationship with the first video or the last video of a certain video group, and cannot have a contextual relationship with any video in the middle of the video group (of course, the new video may also be a completely independent video, which has no contextual relationship with the first video or the last video of any video group). ), in order to improve the efficiency of judging the contextual relationship between the newly added video and the clustered videos, in this embodiment, it is not necessary to judge whether the newly added video has a contextual relationship with all the videos of the user except the newly added video. Instead, as shown in FIG4 , it is only necessary to determine whether the newly added video has a contextual relationship with the first video or the last video in each video group on the basis of obtaining at least one video group through clustering and sorting. If it is determined that the newly added video has a contextual relationship with the first video in a certain video group, the newly added video will be added to the video group as the new first video. If it is determined that the newly added video has a contextual relationship with the last video in a certain video group, the newly added video will be added to the video group as the new last video.

[0064] Based on any of the above embodiments, after obtaining at least one video group, the method further includes:

[0065] S401, receiving a deletion instruction from the user for any target video in any video group;

[0066] S402: If the target video is not the first video or the last video in the video group, delete the target video and divide the video group into different video groups starting from the target video.

[0067] In this embodiment, for any video group, the user can delete any target video of the video group, where the target video is the video that the user needs to delete. If the target video is the first video or the last video in the video group, the target video can be directly deleted from the video group; if the target video is not the first video or the last video in the video group, then after deleting the target video, the video before the target video and the video after the target video have no contextual relationship, and the video group needs to be divided into different video groups, that is, the division is performed from the point where the target video is deleted. As shown in Figure 5, after deleting video 1.3 from video group 1, video group 1 is divided into video group x and video group y from video 1.3 to ensure that the videos in each video group after the division still have a contextual relationship.

[0068] Based on any of the above embodiments, the feature extraction model and the video context discrimination model need to be trained in advance, and the training process is as follows:

[0069] S501, obtaining multiple positive sample video groups and multiple negative sample video groups;

[0070] S502: Jointly train the feature extraction model and the video context discrimination model according to the multiple positive sample video groups and the multiple negative sample video groups.

[0071] In this embodiment, for the joint training of the feature extraction model and the video context discrimination model, it is necessary to first obtain training samples, wherein the training samples include multiple positive sample video groups and multiple negative sample video groups. Optionally, the positive sample video group includes two videos with contextual relationship belonging to the same series of videos. For example, multiple video clips are segmented from a long video. Any two adjacent video clips have a contextual relationship and can be used as a positive sample video group. For example, any two adjacent videos in the same series of videos (such as a TV series) have a contextual relationship and can also be used as a positive sample video group. The negative sample video group may include two videos without contextual relationship belonging to the same series of videos. For example, two non-adjacent video clips in a long video can be used as a negative sample video group. For example, any two non-adjacent videos in the same series of videos (such as a TV series) have no contextual relationship and can also be used as a negative sample video group. In addition, the negative sample video group can also include two videos belonging to different series of videos, for example, a video clip is obtained from the first long video, and another video clip is obtained from the second long video. The two video clips have no contextual relationship and can be used as a negative sample video group. For example, any two videos in different series of videos (such as two TV series) have no contextual relationship and can also be used as a negative sample video group.

[0072] After obtaining the above-mentioned multiple positive sample video groups and multiple negative sample video groups, the feature extraction model and the video context discrimination model can be jointly trained based on the multiple positive sample video groups and multiple negative sample video groups. Specifically, during the forward propagation process, a positive sample video group or a negative sample video group is input into the feature extraction model, and features are extracted by the feature extraction model. The features output by the feature extraction model are input into the video context discrimination model for contextual relationship judgment to obtain a contextual relationship judgment result. Based on the contextual relationship judgment result and the positive sample video group or the negative sample video group input into the feature extraction model, the joint training loss of the feature extraction model and the video context discrimination model is calculated. Back propagation is performed based on the loss to adjust the parameters of the feature extraction model and the video context discrimination model. After repeated training iterations, the features extracted by the feature extraction model are features that can be used to judge the contextual relationship between videos, and the video context discrimination model can accurately judge whether there is a contextual relationship between videos based on the features used to judge the contextual relationship between videos.

[0073] It should be noted that the execution subject of the model training process can be the same execution subject as the above-mentioned video clustering method. Of course, it can also be that after completing the training on other execution subjects, the trained model is deployed to the execution subject of the above-mentioned video clustering method.

[0074] Corresponding to the video clustering method described in the preceding embodiment, FIG6 is a block diagram of the video clustering device provided in the present embodiment. For ease of illustration, only the portions relevant to the present embodiment are shown. Referring to FIG6 , the video clustering device 500 includes an acquisition unit 501, a feature extraction unit 502, a context determination unit 503, and a clustering unit 504.

[0075] The acquisition unit 501 is used to acquire all videos to be classified of any user from the video library;

[0076] A feature extraction unit 502 is configured to extract features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model;

[0077] The context determination unit 503 is used to determine whether there is a contextual relationship between any two videos to be classified based on the characteristics of each video to be classified and a preset video context determination model;

[0078] The clustering unit 504 is configured to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted in contextual order.

[0079] In one or more embodiments of the present disclosure, when the feature extraction unit 502 uses a preset feature extraction model to extract features for determining the contextual relationship between videos, it is configured to:

[0080] For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

[0081] In one or more embodiments of the present disclosure, when determining whether there is a contextual relationship between any two videos to be classified based on the features of each video to be classified and a preset video context discrimination model, the context determination unit 503 is configured to:

[0082] A similarity matrix is ​​generated based on the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined based on the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

[0083] In one or more embodiments of the present disclosure, when determining whether there is a contextual relationship between any two videos to be classified, the context determination unit 503 is configured to:

[0084] If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified is determined based on the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified.

[0085] In one or more embodiments of the present disclosure, the acquiring unit 501 is further configured to acquire a newly added video of the user;

[0086] The feature extraction unit 502 is further configured to extract features for determining contextual relationships between videos from the newly added video using the feature extraction model;

[0087] The context determination unit 503 is further configured to determine whether the newly added video has a contextual relationship with the first video or the last video according to the features of the newly added video and the features of the first video and the last video in any of the video groups;

[0088] The clustering unit 504 is further configured to, if it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, add the newly added video to the video group as the first video or the last video.

[0089] In one or more embodiments of the present disclosure, the clustering unit 504 is further configured to, after obtaining at least one video group, receive a deletion instruction from the user for any target video in any video group;

[0090] If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

[0091] In one or more embodiments of the present disclosure, the device further includes a control unit configured to:

[0092] Determine whether the currently playing video is the last video in the video group;

[0093] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0094] If the currently playing video is the last video in the video group, the video will stop playing and / or display a prompt message after the currently playing video ends.

[0095] In one or more embodiments of the present disclosure, the device further includes a training unit configured to:

[0096] Obtain multiple positive sample video groups and multiple negative sample video groups;

[0097] The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

[0098] In one or more embodiments of the present disclosure, the positive sample video group includes two videos with a contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without a contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

[0099] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0100] Referring to FIG7 , a schematic diagram of the structure of an electronic device 600 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 600 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG7 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0101] As shown in FIG7 , the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0102] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 7 shows an electronic device 600 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0103] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0104] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0105] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0106] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0107] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0110] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] In a first aspect, according to one or more embodiments of the present disclosure, a video clustering method is provided, comprising:

[0113] Get all videos to be classified by any user from the video library;

[0114] For each of all videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos;

[0115] Based on the characteristics of each video to be classified and the preset video context discrimination model, determine whether there is a contextual relationship between any two videos to be classified;

[0116] The videos to be classified that have contextual relationships are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted in contextual order.

[0117] According to one or more embodiments of the present disclosure, the step of extracting features for determining contextual relationships between videos using a preset feature extraction model includes:

[0118] For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

[0119] According to one or more embodiments of the present disclosure, the step of determining whether any two videos to be classified have a contextual relationship based on the features of each video to be classified and a preset video context discrimination model includes:

[0120] A similarity matrix is ​​generated based on the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined based on the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

[0121] According to one or more embodiments of the present disclosure, determining whether any two videos to be classified have a contextual relationship includes:

[0122] If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified is determined based on the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified.

[0123] According to one or more embodiments of the present disclosure, after obtaining at least one video group, the method further includes:

[0124] Get the newly added videos of the user;

[0125] Extracting features for determining contextual relationships between videos using the feature extraction model from the newly added video;

[0126] Determining, based on the features of the newly added video and the features of the first video and the last video in any of the video groups, whether the newly added video has a contextual relationship with the first video or the last video;

[0127] If it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, the newly added video is added to the video group as the first video or the last video.

[0128] According to one or more embodiments of the present disclosure, after obtaining at least one video group, the method further includes:

[0129] receiving a deletion instruction from the user for any target video in any video group;

[0130] If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

[0131] According to one or more embodiments of the present disclosure, after obtaining at least one video group, the method further includes:

[0132] Determine whether the currently playing video is the last video in the video group;

[0133] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0134] If the currently playing video is the last video in the video group, the video will stop playing and / or display a prompt message after the currently playing video ends.

[0135] According to one or more embodiments of the present disclosure, the feature extraction model and the video context discrimination model are trained through the following process:

[0136] Obtain multiple positive sample video groups and multiple negative sample video groups;

[0137] The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

[0138] According to one or more embodiments of the present disclosure, the positive sample video group includes two videos with a contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without a contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

[0139] In a second aspect, according to one or more embodiments of the present disclosure, a video clustering device is provided, including:

[0140] An acquisition unit, used to acquire all videos to be classified of any user from the video library;

[0141] A feature extraction unit is used to extract features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model;

[0142] A context discrimination unit is used to determine whether there is a contextual relationship between any two videos to be classified based on the characteristics of each video to be classified and a preset video context discrimination model;

[0143] The clustering unit is used to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted in context order.

[0144] According to one or more embodiments of the present disclosure, when the feature extraction unit uses a preset feature extraction model to extract features for determining the contextual relationship between videos, it is configured to:

[0145] For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

[0146] According to one or more embodiments of the present disclosure, when the context determination unit determines whether there is a contextual relationship between any two videos to be classified based on the features of each video to be classified and a preset video context determination model, it is configured to:

[0147] A similarity matrix is ​​generated based on the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined based on the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

[0148] According to one or more embodiments of the present disclosure, when determining whether there is a contextual relationship between any two videos to be classified, the context determination unit is configured to:

[0149] If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified is determined based on the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified.

[0150] According to one or more embodiments of the present disclosure, the acquiring unit is further configured to acquire a newly added video of the user;

[0151] The feature extraction unit is further configured to extract features for determining contextual relationships between videos from the newly added video using the feature extraction model;

[0152] The context determination unit is further configured to determine, based on the features of the newly added video and the features of the first video and the last video in any of the video groups, whether the newly added video has a contextual relationship with the first video or the last video;

[0153] The clustering unit is further configured to, if it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, add the newly added video to the video group as the first video or the last video.

[0154] According to one or more embodiments of the present disclosure, the clustering unit is further configured to, after obtaining at least one video group, receive a deletion instruction from the user for any target video in any video group;

[0155] If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

[0156] According to one or more embodiments of the present disclosure, the device further includes a control unit configured to:

[0157] Determine whether the currently playing video is the last video in the video group;

[0158] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0159] If the currently playing video is the last video in the video group, the video will stop playing and / or display a prompt message after the currently playing video ends.

[0160] According to one or more embodiments of the present disclosure, the device further includes a training unit configured to:

[0161] Obtain multiple positive sample video groups and multiple negative sample video groups;

[0162] The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

[0163] According to one or more embodiments of the present disclosure, the positive sample video group includes two videos with a contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without a contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

[0164] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0165] The memory stores computer-executable instructions;

[0166] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video clustering method described in the first aspect and various possible designs of the first aspect.

[0167] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the video clustering method described in the first aspect and various possible designs of the first aspect is implemented.

[0168] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the video clustering method as described in the first aspect and various possible designs of the first aspect is implemented.

[0169] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0170] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0171] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video clustering method, comprising: Get all videos to be classified from the video library of any user; For each of all the videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos; According to the characteristics of each video to be classified and the preset video context discrimination model, it is determined whether there is a context relationship between any two videos to be classified; The videos to be classified with contextual relationships are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

2. The method according to claim 1, wherein: The method of extracting features for determining contextual relationships between videos using a preset feature extraction model includes: For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

3. The method according to claim 1 or 2, wherein: The step of judging whether any two videos to be classified have a contextual relationship based on the features of each video to be classified and a preset video context discrimination model includes: A similarity matrix is ​​generated according to the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined according to the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

4. The method according to claim 3, wherein: The step of determining whether any two videos to be classified have a contextual relationship includes: If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the first video to be classified is determined according to the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified. The order of the second video to be classified.

5. The method according to any one of claims 1 to 4, wherein: After obtaining at least one video group, the method further includes: Get the newly added videos of the user; Extracting features for determining contextual relationships between videos using the feature extraction model from the newly added video; According to the features of the newly added video and the features of the first video and the last video in any of the video groups, determining whether the newly added video has a contextual relationship with the first video or the last video; If it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, the newly added video is added to the video group as the first video or the last video.

6. The method according to any one of claims 1 to 4, wherein: After obtaining at least one video group, the method further includes: receiving a deletion instruction from the user for any target video in any video group; If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

7. The method according to any one of claims 1 to 6, wherein: After obtaining at least one video group, the method further includes: Determine whether the currently playing video is the last video in the video group; If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or If the currently playing video is the last video in the video group, then the playback of the currently playing video is stopped and / or a prompt message is displayed after the currently playing video is finished.

8. The method according to any one of claims 1 to 4, wherein: The feature extraction model and the video context discrimination model are trained through the following process: Obtain multiple positive sample video groups and multiple negative sample video groups; The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

9. The method according to claim 8, wherein: The positive sample video group includes two videos with contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

10. A video clustering device, comprising: An acquisition unit, used for acquiring all videos to be classified of any user from a video library; A feature extraction unit, for extracting features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model; A context discrimination unit, used to determine whether there is a contextual relationship between any two videos to be classified according to the characteristics of each video to be classified and a preset video context discrimination model; The clustering unit is used to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

11. An electronic device, comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, wherein: The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

Images