Video clustering method and device and storage medium

Abstract

The embodiment of the invention provides a video clustering method and device and a storage medium. The method comprises the steps of obtaining all to-be-classified videos of any user from a video library; adopting a preset feature extraction model for each to-be-classified video in all the to-be-classified videos to extract features used for judging the context relationship between the videos; judging whether any two to-be-classified videos have a context relationship or not according to the features of each to-be-classified video and a preset video context judgment model; and clustering and sorting the to-be-classified videos with the context relationship to obtain at least one video group. The to-be-classified videos with the context relation are automatically clustered and sorted based on the preset model, operations such as labeling, classifying and sorting of the to-be-classified videos by a user are not needed, the clustering and sorting efficiency of the to-be-classified videos is improved, the cost is reduced, and the user experience is improved. And clustering and sorting of the to-be-classified videos provide convenience for video viewers to continuously watch the videos, and the video watching experience is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular to a video clustering method, device, and storage medium. Background Art

[0002] With the development of network technology and the popularization of smart terminals, videos have gradually become an important channel for people to entertain themselves and obtain information. Usually, there are some videos that belong to the same series, such as different episodes of the same TV series, or a long video divided into different short videos, etc. Therefore, it is necessary to cluster videos so that video viewers can watch videos according to clusters.

[0003] In the prior art, video uploaders are usually required to label the uploaded videos and cluster them according to the labels. However, manual labeling and manual clustering are inefficient and cannot achieve rapid clustering of videos. Summary of the invention

[0004] The embodiments of the present disclosure provide a video clustering method, device and storage medium to achieve fast clustering and sorting of videos to be classified.

[0005] In a first aspect, an embodiment of the present disclosure provides a video clustering method, including:

[0006] Get all videos to be classified from the video library of any user;

[0007] For each of all the videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos;

[0008] According to the characteristics of each video to be classified and the preset video context discrimination model, it is determined whether there is a context relationship between any two videos to be classified;

[0009] The videos to be classified with contextual relationships are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

[0010] In a second aspect, an embodiment of the present disclosure provides a video clustering device, including:

[0011] An acquisition unit, used for acquiring all videos to be classified of any user from a video library;

[0012] A feature extraction unit, for extracting features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model;

[0013] A context discrimination unit, used to determine whether there is a contextual relationship between any two videos to be classified according to the characteristics of each video to be classified and a preset video context discrimination model;

[0014] The clustering unit is used to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video clustering method as described in the first aspect and various possible designs of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the video clustering method described in the first aspect and various possible designs of the first aspect is implemented.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, the video clustering method described in the first aspect and various possible designs of the first aspect is implemented.

[0020] The video clustering method, device and storage medium provided by the disclosed embodiment obtain all videos to be classified of any user from the video library; for each video to be classified among all videos to be classified, a preset feature extraction model is used to extract features for judging the contextual relationship between videos; based on the features of each video to be classified and a preset video context discrimination model, it is judged whether there is a contextual relationship between any two videos to be classified; the videos to be classified with contextual relationship are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted in context order. Based on the preset model, the videos to be classified with contextual relationship are automatically clustered and sorted, without the need for users to manually label, classify, sort and other operations on the videos to be classified, thereby improving the efficiency of clustering and sorting the videos to be classified and reducing the cost, and the clustering and sorting of the videos to be classified provide convenience for video viewers to watch videos continuously, which is conducive to improving the video viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of an application scenario of a video clustering method provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a video clustering method flow chart provided by an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of feature extraction provided by an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a newly added video provided in an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of deleting a video according to an embodiment of the present disclosure;

[0027] Figure 6 A structural block diagram of a video clustering device provided in an embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] At present, there are some videos that usually belong to the same series, such as different episodes of the same TV series, or a long video divided into different short videos, etc. Therefore, it is necessary to cluster the videos to facilitate video viewers to watch videos according to clustering, especially to watch videos in sequence. For example, after watching a certain video, the video viewer will continue to watch the next video connected to the video.

[0031] In the prior art, video uploaders are usually required to label the uploaded videos and cluster them according to the labels. However, manual labeling and manual clustering are inefficient and cannot achieve rapid clustering of videos. In addition, video serial numbers are added to video titles, and then the videos are sorted according to the video serial numbers after clustering. Similarly, labeling video serial numbers also requires manpower and is inefficient. In addition, if the video serial number is wrong, it may cause errors in the video sorting after clustering.

[0032] In order to solve the above technical problems, the present disclosure provides a video clustering method, in the hope of being able to cluster videos quickly and efficiently, and also in the hope that the clustered videos can be sorted in context order. Considering that the machine learning model can realize the video classification function after training, the present disclosure hopes to realize the clustering and sorting of videos with the help of the machine learning model, and can pre-train the feature extraction model and the video context discrimination model, and then for all videos to be classified of any user, the feature extraction model can be used to extract the features for judging the contextual relationship between videos for each video to be classified, and then the features of each video to be classified can be input into the video context discrimination model to judge whether there is a contextual relationship between any two videos to be classified; the videos to be classified with contextual relationship are clustered and sorted. The present disclosure automatically realizes the clustering and sorting of videos to be classified with contextual relationship based on the preset model, without the need for users to manually label, classify, sort and other operations on the videos to be classified, thereby improving the efficiency of clustering and sorting of videos to be classified and reducing the cost, and the clustering and sorting of videos to be classified provide convenience for video viewers to watch videos continuously, which is conducive to improving the video viewing experience.

[0033] The video clustering method disclosed in the present invention can be applied to scenarios where a server or terminal device clusters and sorts videos, for example Figure 1 The application shown is to a server, in which a preset feature extraction model and a video context discrimination model are pre-deployed. The server can first perform a preliminary classification of the video library by user (video uploader), obtain all videos to be classified by any user, and then input them into the feature extraction model to extract features of each video to be classified for judging the contextual relationship between videos, input the features of each video to be classified into the video context discrimination model, and judge whether there is a contextual relationship between any two videos to be classified; cluster and sort the videos to be classified with the contextual relationship to obtain at least one video group, wherein the videos included in each video group are sorted in context order.

[0034] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] The video clustering method disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0036] refer to Figure 2 , Figure 2 A schematic diagram of a video clustering method flow chart provided in an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server, and the video clustering method includes:

[0037] S201, obtaining all videos to be classified of any user from a video library.

[0038] In this embodiment, the video library includes videos uploaded by different users, and video clustering and sorting need to be performed on each video of the same user. Therefore, the videos to be classified included in the video library can be preliminarily classified by user (video uploader) to obtain the videos to be classified of each user, and then all videos to be classified of any user can be obtained, so as to perform corresponding operations on the videos to be classified of any user (video uploader) in the following steps.

[0039] S202: For each video to be classified among all the videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos.

[0040] In this embodiment, all videos to be classified for any user are clustered based on contextual relationships, that is, videos with contextual relationships are classified into a video group. For example, all videos to be classified for the user include: video 1.1, video 1.2, video 1.4, video 1.5, video 2.1, video 2.2, wherein video 1.1, video 1.2, video 1.4, and video 1.5 belong to the same series of videos, video 1.1 and video 1.2 have a contextual relationship (adjacent), video 1.4 and video 1.5 have a contextual relationship (adjacent), and Video 1.2 and video 1.4 have no contextual relationship (not adjacent), video 2.1 and video 2.2 belong to another series of videos and have a contextual relationship (adjacent). The purpose of this embodiment is to cluster the above videos according to the contextual relationship, and 3 video groups can be obtained. Video group 1 includes video 1.1 and video 1.2, video group 2 includes video 1.4 and video 1.5 (although they belong to the same series of videos as video 1.1 and video 1.2, they cannot be divided into the same group because they have no contextual relationship), and video group 3 includes video 2.1 and video 2.2.

[0041] In order to achieve the above purpose, in this embodiment, features are extracted from each video to be classified among all videos to be classified of any user. Different from the common video clustering method, the features extracted from the videos in this embodiment are features used to determine the contextual relationship between videos. By comparing the features of any two videos to be classified, it can be determined whether the two videos to be classified have contextual features, and if the two videos to be classified have contextual features, the order of the two videos to be classified can be further determined. The features used to determine the contextual relationship between videos can be realized through machine learning, that is, in this embodiment, a feature extraction model needs to be pre-trained, and through the training process, the feature extraction model has the function of extracting the above-mentioned features used to determine the contextual relationship between videos.

[0042] Optionally, for any video to be classified, a feature extraction model may be used to extract features for determining the contextual relationship between videos based on one or more of the visual information, audio information, title information, and video subtitles of the video to be classified. Figure 3 As shown, one or more of the visual information (video image), audio information, title information and video subtitles of the video to be classified can be obtained and input into the feature extraction model to extract one or more of the visual features, audio features, title features and subtitle features accordingly, and then the extracted features are fused to obtain multimodal features, so as to more accurately judge the contextual relationship between videos.

[0043] S203: judging whether there is a contextual relationship between any two videos to be classified according to the characteristics of each video to be classified and a preset video context discrimination model.

[0044] In this embodiment, after obtaining the features for determining the contextual relationship between videos of each video to be classified among all videos to be classified of any user, it is possible to determine whether there is a contextual relationship between each video based on the features of each video to be classified. In this embodiment, a video context discrimination model can be pre-trained. The video context discrimination model can output a judgment result of whether the two videos to be classified have a contextual relationship based on the features for determining the contextual relationship between videos of any two videos to be classified that are input; it is also possible to batch input the features for determining the contextual relationship between videos of multiple videos to be classified into the video context discrimination model, and output a judgment result of whether any two videos to be classified among the multiple videos to be classified have a contextual relationship.

[0045] Among them, for the features of any two videos to be classified that are used to determine the contextual relationship between videos, the video context discrimination model can obtain the similarity of the two features, identify the size of the contextual relationship between the two videos to be classified based on the size of the similarity, and then determine whether the two videos to be classified have a contextual relationship based on the similarity, for example, determine whether the similarity value is greater than a preset threshold. If so, it is determined that there is a contextual relationship, and then output the judgment result of whether the two videos to be classified have a contextual relationship.

[0046] In this embodiment, the features used to determine the contextual relationship between videos of each video to be classified among all videos to be classified of any user can be batch-inputted into the video context discrimination model. The video context discrimination model can generate a similarity matrix. Any similarity value in the similarity matrix is ​​used to identify the size of the contextual relationship between the corresponding two videos to be classified. Then, whether there is a contextual relationship between any two videos to be classified can be determined based on the similarity matrix.

[0047] It should be noted that, since the video contents with contextual relationships are connected, there is a sequence between the videos with contextual relationships, and when obtaining the similarity of the features of any two videos to be classified based on the features of the two videos to be classified, two similarity values ​​can be obtained, that is, assuming that any two videos to be classified are the first video to be classified and the second video to be classified, the similarity value corresponding to the first video to be classified and the second video to be classified, and the similarity value corresponding to the second video to be classified and the first video to be classified can be obtained, wherein the similarity value corresponding to the first video to be classified and the second video to be classified is used to indicate the probability that the first video to be classified and the second video to be classified have a contextual relationship, and the first video to be classified comes first and the second video to be classified comes later, and the similarity value corresponding to the second video to be classified and the first video to be classified is used to indicate the probability that the first video to be classified and the second video to be classified have a contextual relationship, and the second video to be classified comes first and the first video to be classified comes later.

[0048] Therefore, if it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified can be determined according to the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified. Specifically, if the corresponding similarity value between the first video to be classified and the second video to be classified is greater than the corresponding similarity value between the second video to be classified and the first video to be classified, then the first video to be classified is determined to be first and the second video to be classified is determined to be later. If the corresponding similarity value between the first video to be classified and the second video to be classified is less than the corresponding similarity value between the second video to be classified and the first video to be classified, then the second video to be classified is determined to be first and the first video to be classified is determined to be later.

[0049] S204: Cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted in contextual order.

[0050] In this embodiment, the above process can be used to determine whether there is a contextual relationship between any two videos to be classified, as well as the order of the two videos to be classified if it is determined that the two videos to be classified have a contextual relationship, and then the videos to be classified with the contextual relationship are clustered and sorted. For example, after determining that video 3.1 and video 3.2 have a contextual relationship, and video 3.1 comes first and video 3.2 comes later, and video 3.2 and video 3.3 have a contextual relationship, and video 3.2 comes first and video 3.3 comes later, after clustering and sorting, video 3.1, video 3.2 and video 3.3 are classified into one video group, and the order is video 3.1, video 3.2 and video 3.3.

[0051] Furthermore, after obtaining at least one video group, the method further includes:

[0052] Determine whether the currently playing video is the last video in the video group;

[0053] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0054] If the currently playing video is the last video in the video group, then the playback of the currently playing video is stopped and / or a prompt message is displayed after the currently playing video is finished.

[0055] In this embodiment, before, after or after the currently playing video ends, the terminal device of the video viewer can determine whether the currently playing video is the last video in the video group. If the currently playing video is not the last video in the video group, it means that the video group also includes a next video connected to the currently playing video (that is, having a contextual relationship), and the terminal device can be controlled to continue playing the next video after the currently playing video ends until the last video in the video group is played. If the currently playing video is the last video in the video group, it means that there is no next video connected to the currently playing video (that is, there is no next video connected to the currently playing video). At this time, the playback of the currently playing video can be stopped after the playback of the currently playing video ends, and / or a prompt message can be displayed to prompt the video viewer that there is no next video connected to the currently playing video, or to prompt the video viewer to select other video groups for playback. Automatic video playback based on the above video group can facilitate video viewers to continuously watch multiple videos connected to each other without having to manually search for and play the next video after one video is finished playing, especially in a video playback scenario based on a feed stream. For example, a feed stream server pushes a certain video A1 to a terminal device, and the video A1 belongs to a certain video group A. In the video group A, the video A1 has a next video A2 connected to it and having a contextual relationship. After video A1 is played, video A2 can be played directly. Video viewers do not need to manually search for and play video A2, which simplifies user operations and improves video viewing experience.

[0056] The video clustering method provided in this embodiment obtains all videos to be classified of any user from the video library; for each video to be classified among all videos to be classified, a preset feature extraction model is used to extract features for judging the contextual relationship between videos; based on the features of each video to be classified and the preset video context discrimination model, it is judged whether there is a contextual relationship between any two videos to be classified; the videos to be classified with contextual relationship are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted in context order. Based on the preset model, the videos to be classified with contextual relationship are automatically clustered and sorted, without the need for users to manually label, classify, sort and other operations on the videos to be classified, thereby improving the efficiency of clustering and sorting the videos to be classified and reducing the cost. In addition, the clustering and sorting of the videos to be classified provide convenience for video viewers to watch videos continuously, which is conducive to improving the video viewing experience.

[0057] Based on any of the above embodiments, after obtaining at least one video group, the method further includes:

[0058] S301, obtaining a new video of the user;

[0059] S302, extracting features for determining contextual relationships between videos from the newly added video using the feature extraction model;

[0060] S303, judging whether the newly added video has a contextual relationship with the first video or the last video according to the characteristics of the newly added video and the characteristics of the first video and the last video in any of the video groups;

[0061] S304: If it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, the newly added video is added to the video group as the first video or the last video.

[0062] In this embodiment, after all videos to be classified of any user are clustered and sorted to obtain at least one video group, if the user uploads a new video to the video library, the user's newly added video can be obtained, and the feature extraction model is used to extract features for judging the contextual relationship between videos from the newly added video in the same manner as S202; and since the videos with contextual relationship have been clustered and sorted to obtain at least one video group, the newly added video can only have a contextual relationship with the first video or the last video of a certain video group, and cannot have a contextual relationship with any video in the middle of the video group (of course, the newly added video may also be a completely independent video and has no contextual relationship with the first video or the last video of any video group). In order to improve the efficiency of judging the contextual relationship between the newly added video and the clustered videos, in this embodiment, it is not necessary to judge whether the newly added video has a contextual relationship with all videos of the user except the newly added video separately, but instead Figure 4 As shown, it is only necessary to determine whether the newly added video has a contextual relationship with the first video or the last video in each video group based on at least one video group obtained through clustering and sorting. If it is determined that the newly added video has a contextual relationship with the first video in a certain video group, the newly added video is added to the video group and serves as the new first video. If it is determined that the newly added video has a contextual relationship with the last video in a certain video group, the newly added video is added to the video group and serves as the new last video.

[0063] Based on any of the above embodiments, after obtaining at least one video group, the method further includes:

[0064] S401, receiving a deletion instruction from the user for any target video in any video group;

[0065] S402: If the target video is not the first video or the last video in the video group, delete the target video, and divide the video group into different video groups starting from the target video.

[0066] In this embodiment, for any video group, the user can delete any target video of the video group, where the target video is the video that the user needs to delete. If the target video is the first video or the last video in the video group, the target video can be directly deleted from the video group; if the target video is not the first video or the last video in the video group, after deleting the target video, the video before the target video and the video after the target video have no contextual relationship, and the video group needs to be divided into different video groups, that is, the division is performed from the point where the target video is deleted, such as Figure 5 As shown, after deleting video 1.3 from video group 1, video group 1 is split into video group x and video group y from video 1.3, ensuring that the videos in each video group after the split still have a contextual relationship.

[0067] Based on any of the above embodiments, the feature extraction model and the video context discrimination model need to be trained in advance, and the training process is as follows:

[0068] S501, obtaining multiple positive sample video groups and multiple negative sample video groups;

[0069] S502: Jointly train the feature extraction model and the video context discrimination model according to the multiple positive sample video groups and the multiple negative sample video groups.

[0070] In this embodiment, for the joint training of the feature extraction model and the video context discrimination model, it is necessary to first obtain training samples, where the training samples include multiple positive sample video groups and multiple negative sample video groups. Optionally, the positive sample video group includes two videos with contextual relationship belonging to the same series of videos. For example, multiple video clips are segmented from a long video. Any two adjacent video clips have a contextual relationship and can be used as a positive sample video group. For example, any two adjacent videos in the same series of videos (such as a TV series) have a contextual relationship and can also be used as a positive sample video group. The negative sample video group may include two videos in the same series of videos that do not have a contextual relationship. Two videos in a series, for example, two non-adjacent video clips in a long video can be used as a negative sample video group. For example, any two non-adjacent videos in the same series of videos (such as a TV series) have no contextual relationship and can also be used as a negative sample video group. In addition, the negative sample video group may also include two videos belonging to different series of videos, for example, a video clip is obtained from the first long video, and another video clip is obtained from the second long video. The two video clips have no contextual relationship and can be used as a negative sample video group. For example, any two videos in different series of videos (such as two TV series) have no contextual relationship and can also be used as a negative sample video group.

[0071] After obtaining the above-mentioned multiple positive sample video groups and multiple negative sample video groups, the feature extraction model and the video context discrimination model can be jointly trained based on the multiple positive sample video groups and multiple negative sample video groups. Specifically, in the forward propagation process, a positive sample video group or a negative sample video group is input into the feature extraction model, and features are extracted by the feature extraction model, and the features output by the feature extraction model are input into the video context discrimination model for contextual relationship judgment to obtain the contextual relationship judgment result, and the joint training loss of the feature extraction model and the video context discrimination model is calculated based on the contextual relationship judgment result and the positive sample video group or the negative sample video group input into the feature extraction model, and the back propagation is performed based on the loss to adjust the parameters of the feature extraction model and the video context discrimination model. After repeated training iterations, the features extracted by the feature extraction model are features that can be used to judge the contextual relationship between videos, and the video context discrimination model can accurately judge whether there is a contextual relationship between videos based on the features used to judge the contextual relationship between videos.

[0072] It should be noted that the execution subject of the model training process can be the same execution subject as the above-mentioned video clustering method. Of course, it can also be that after completing the training on other execution subjects, the trained model is deployed to the execution subject of the above-mentioned video clustering method.

[0073] Corresponding to the video clustering method of the above embodiment, Figure 6The structure block diagram of the video clustering device provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 6 The video clustering device 500 includes: an acquisition unit 501, a feature extraction unit 502, a context determination unit 503, and a clustering unit 504.

[0074] The acquisition unit 501 is used to acquire all videos to be classified of any user from the video library;

[0075] A feature extraction unit 502 is used to extract features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model;

[0076] The context determination unit 503 is used to determine whether there is a context relationship between any two videos to be classified according to the characteristics of each video to be classified and a preset video context determination model;

[0077] The clustering unit 504 is used to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

[0078] In one or more embodiments of the present disclosure, when the feature extraction unit 502 uses a preset feature extraction model to extract features for determining the contextual relationship between videos, it is used to:

[0079] For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

[0080] In one or more embodiments of the present disclosure, when determining whether any two videos to be classified have a contextual relationship based on the features of each video to be classified and a preset video context discrimination model, the context discrimination unit 503 is used to:

[0081] A similarity matrix is ​​generated according to the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined according to the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

[0082] In one or more embodiments of the present disclosure, when determining whether any two videos to be classified have a contextual relationship, the context determination unit 503 is used to:

[0083] If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified is determined according to the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified.

[0084] In one or more embodiments of the present disclosure, the acquisition unit 501 is further used to acquire a newly added video of the user;

[0085] The feature extraction unit 502 is further used to extract features for determining contextual relationships between videos from the newly added video using the feature extraction model;

[0086] The context determination unit 503 is further configured to determine whether the newly added video has a contextual relationship with the first video or the last video according to the features of the newly added video and the features of the first video and the last video in any of the video groups;

[0087] The clustering unit 504 is further configured to, if it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, add the newly added video to the video group as the first video or the last video.

[0088] In one or more embodiments of the present disclosure, the clustering unit 504 is further configured to, after obtaining at least one video group, receive a deletion instruction from the user for any target video in any video group;

[0089] If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

[0090] In one or more embodiments of the present disclosure, the device further includes a control unit, configured to:

[0091] Determine whether the currently playing video is the last video in the video group;

[0092] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0093] If the currently playing video is the last video in the video group, then the playback of the currently playing video is stopped and / or a prompt message is displayed after the currently playing video is finished.

[0094] In one or more embodiments of the present disclosure, the device further includes a training unit, configured to:

[0095] Obtain multiple positive sample video groups and multiple negative sample video groups;

[0096] The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

[0097] In one or more embodiments of the present disclosure, the positive sample video group includes two videos with a contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without a contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

[0098] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0099] refer to Figure 7 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure, and the electronic device 600 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] like Figure 7 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0101] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0102] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0103] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0104] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0105] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0106] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0109] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0110] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] In a first aspect, according to one or more embodiments of the present disclosure, a video clustering method is provided, comprising:

[0112] Get all videos to be classified from the video library of any user;

[0113] For each of all the videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos;

[0114] According to the characteristics of each video to be classified and the preset video context discrimination model, it is determined whether there is a context relationship between any two videos to be classified;

[0115] The videos to be classified with contextual relationships are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

[0116] According to one or more embodiments of the present disclosure, the step of extracting features for determining contextual relationships between videos using a preset feature extraction model includes:

[0117] For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

[0118] According to one or more embodiments of the present disclosure, judging whether there is a contextual relationship between any two videos to be classified based on the features of each video to be classified and a preset video context discrimination model includes:

[0119] A similarity matrix is ​​generated according to the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined according to the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

[0120] According to one or more embodiments of the present disclosure, the step of determining whether any two videos to be classified have a contextual relationship includes:

[0121] If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified is determined according to the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified.

[0122] According to one or more embodiments of the present disclosure, after obtaining at least one video group, the method further includes:

[0123] Get the newly added videos of the user;

[0124] Extracting features for determining contextual relationships between videos using the feature extraction model for the newly added video;

[0125] According to the features of the newly added video and the features of the first video and the last video in any of the video groups, determining whether the newly added video has a contextual relationship with the first video or the last video;

[0126] If it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, the newly added video is added to the video group as the first video or the last video.

[0127] According to one or more embodiments of the present disclosure, after obtaining at least one video group, the method further includes:

[0128] receiving a deletion instruction from the user for any target video in any video group;

[0129] If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

[0130] According to one or more embodiments of the present disclosure, after obtaining at least one video group, the method further includes:

[0131] Determine whether the currently playing video is the last video in the video group;

[0132] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0133] If the currently playing video is the last video in the video group, then the playback of the currently playing video is stopped and / or a prompt message is displayed after the currently playing video is finished.

[0134] According to one or more embodiments of the present disclosure, the feature extraction model and the video context discrimination model are trained through the following process:

[0135] Obtain multiple positive sample video groups and multiple negative sample video groups;

[0136] The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

[0137] According to one or more embodiments of the present disclosure, the positive sample video group includes two videos with a contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without a contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

[0138] In a second aspect, according to one or more embodiments of the present disclosure, a video clustering device is provided, including:

[0139] An acquisition unit, used for acquiring all videos to be classified of any user from a video library;

[0140] A feature extraction unit, for extracting features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model;

[0141] A context discrimination unit, used to determine whether there is a contextual relationship between any two videos to be classified according to the characteristics of each video to be classified and a preset video context discrimination model;

[0142] The clustering unit is used to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

[0143] According to one or more embodiments of the present disclosure, when the feature extraction unit uses a preset feature extraction model to extract features for determining the contextual relationship between videos, it is used to:

[0144] For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

[0145] According to one or more embodiments of the present disclosure, when the context determination unit determines whether there is a contextual relationship between any two videos to be classified according to the features of each video to be classified and a preset video context determination model, it is used to:

[0146] A similarity matrix is ​​generated according to the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined according to the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

[0147] According to one or more embodiments of the present disclosure, when determining whether there is a contextual relationship between any two videos to be classified, the context determination unit is used to:

[0148] If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified is determined according to the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified.

[0149] According to one or more embodiments of the present disclosure, the acquisition unit is further used to acquire a newly added video of the user;

[0150] The feature extraction unit is further used to extract features for determining contextual relationships between videos from the newly added video using the feature extraction model;

[0151] The context determination unit is further used to determine whether the newly added video has a contextual relationship with the first video or the last video according to the features of the newly added video and the features of the first video and the last video in any of the video groups;

[0152] The clustering unit is further configured to, if it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, add the newly added video to the video group as the first video or the last video.

[0153] According to one or more embodiments of the present disclosure, the clustering unit is further configured to, after obtaining at least one video group, receive a deletion instruction from the user for any target video in any video group;

[0154] If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

[0155] According to one or more embodiments of the present disclosure, the device further includes a control unit, configured to:

[0156] Determine whether the currently playing video is the last video in the video group;

[0157] If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or

[0158] If the currently playing video is the last video in the video group, then the playback of the currently playing video is stopped and / or a prompt message is displayed after the currently playing video is finished.

[0159] According to one or more embodiments of the present disclosure, the device further includes a training unit, configured to:

[0160] Obtain multiple positive sample video groups and multiple negative sample video groups;

[0161] The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

[0162] According to one or more embodiments of the present disclosure, the positive sample video group includes two videos with a contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without a contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

[0163] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0164] The memory stores computer-executable instructions;

[0165] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video clustering method as described in the first aspect and various possible designs of the first aspect.

[0166] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the video clustering method described in the first aspect and various possible designs of the first aspect is implemented.

[0167] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the video clustering method as described in the first aspect and various possible designs of the first aspect is implemented.

[0168] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0169] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0170] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video clustering method, characterized in that: include: Get all videos to be classified from the video library of any user; For each of all the videos to be classified, a preset feature extraction model is used to extract features for determining contextual relationships between videos; According to the characteristics of each video to be classified and the preset video context discrimination model, it is determined whether there is a context relationship between any two videos to be classified; The videos to be classified with contextual relationships are clustered and sorted to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

2. The method according to claim 1, characterized in that: The method of extracting features for determining contextual relationships between videos using a preset feature extraction model includes: For any video to be classified, based on one or more of the visual information, audio information, title information and video subtitles of the video to be classified, the feature extraction model is used to extract features for determining the contextual relationship between videos.

3. The method according to claim 1, characterized in that The step of judging whether any two videos to be classified have a contextual relationship based on the features of each video to be classified and a preset video context discrimination model includes: A similarity matrix is ​​generated according to the characteristics of each video to be classified and the video context discrimination model, and whether there is a context relationship between any two videos to be classified is determined according to the similarity matrix, wherein any similarity value in the similarity matrix is ​​used to identify the size of the context relationship between the corresponding two videos to be classified.

4. The method according to claim 3, characterized in that The step of determining whether any two videos to be classified have a contextual relationship includes: If it is determined that there is a contextual relationship between the first video to be classified and the second video to be classified, the order of the first video to be classified and the second video to be classified is determined according to the corresponding similarity values ​​between the first video to be classified and the second video to be classified, and the corresponding similarity values ​​between the second video to be classified and the first video to be classified.

5. The method according to any one of claims 1 to 4, characterized in that: After obtaining at least one video group, the method further includes: Get the newly added videos of the user; Extracting features for determining contextual relationships between videos using the feature extraction model from the newly added video; According to the features of the newly added video and the features of the first video and the last video in any of the video groups, determining whether the newly added video has a contextual relationship with the first video or the last video; If it is determined that the newly added video has a contextual relationship with the first video or the last video in the video group, the newly added video is added to the video group as the first video or the last video.

6. The method according to any one of claims 1 to 4, characterized in that: After obtaining at least one video group, the method further includes: receiving a deletion instruction from the user for any target video in any video group; If the target video is not the first video or the last video in the video group, the target video is deleted, and the video group is divided into different video groups from the target video.

7. The method according to claim 1, characterized in that After obtaining at least one video group, the method further includes: Determine whether the currently playing video is the last video in the video group; If the currently playing video is not the last video in the video group, controlling the next video in the video group to continue playing after the currently playing video ends; or If the currently playing video is the last video in the video group, then the playback of the currently playing video is stopped and / or a prompt message is displayed after the currently playing video is finished.

8. The method according to any one of claims 1 to 4, characterized in that: The feature extraction model and the video context discrimination model are trained through the following process: Obtain multiple positive sample video groups and multiple negative sample video groups; The feature extraction model and the video context discrimination model are jointly trained according to the multiple positive sample video groups and the multiple negative sample video groups.

9. The method according to claim 8, characterized in that The positive sample video group includes two videos with contextual relationship belonging to the same series of videos, the negative sample video group includes two videos without contextual relationship belonging to the same series of videos, and / or the negative sample video group includes two videos belonging to different series of videos.

10. A video clustering device, characterized in that: include: An acquisition unit, used for acquiring all videos to be classified of any user from a video library; A feature extraction unit, for extracting features for determining contextual relationships between videos from each of all videos to be classified using a preset feature extraction model; A context discrimination unit, used to determine whether there is a contextual relationship between any two videos to be classified according to the characteristics of each video to be classified and a preset video context discrimination model; The clustering unit is used to cluster and sort the videos to be classified that have contextual relationships to obtain at least one video group, wherein the videos included in each video group are sorted according to the contextual order.

11. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

Images