Method for determining video tags, video recommendation method, and video search method

Abstract

The present disclosure provides a method for determining video tags, relates to the technical field of artificial intelligence, and particularly relates to the fields of image processing and computer vision. The specific implementation scheme is as follows: dividing a video sequence into a plurality of sub-videos according to shots; determining at least one video frame from the sub-videos as a key frame of the sub-video, and determining an initial tag sequence of the key frame; determining an effective tag of the sub-video according to the initial tag sequence; and determining a tag of the video sequence according to the effective tag of the sub-video. The present disclosure also provides a method for determining video tags, a video recommendation method, a video query method, an apparatus, an electronic device, and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of image processing and computer vision technology. More specifically, this disclosure provides a method for determining video tags, a video recommendation method, a video query method, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] Video tag extraction is a core function in media resource management, serving as a prerequisite for video storage, retrieval, and recommendation. During the video tag extraction process, it was found that video processing time increases linearly with video length, with long videos being processed very slowly, significantly limiting the application scenarios of related algorithms. Summary of the Invention

[0003] This disclosure provides a method for determining video tags, a video recommendation method, a video query method, an apparatus, a device, and a storage medium.

[0004] According to the first aspect, a method for determining video tags is provided, the method comprising: dividing a video sequence into multiple sub-videos according to shots; determining at least one video frame from the sub-videos as keyframes of the sub-videos, and determining an initial tag sequence of the keyframes; determining valid tags of the sub-videos based on the initial tag sequence; and determining tags of the video sequence based on the valid tags of the sub-videos.

[0005] According to the second aspect, a video recommendation method is provided, the method comprising: obtaining user tags; and determining a first target video from the video library for recommendation to the user based on a first similarity between the user tags and the tags of each video in the video library; wherein the video tags are obtained according to the method for determining video tags described above.

[0006] According to a third aspect, a video query method is provided, the method comprising: receiving a request to query a video, the request including a query term; and determining a second target video corresponding to the query term from the video library based on a second similarity between the query term and the tags of each video in the video library; wherein the video tags are obtained according to the method for determining video tags described above.

[0007] According to a fourth aspect, an apparatus for determining video tags is provided, the apparatus comprising: a segmentation module for dividing a video sequence into multiple sub-videos according to shots; a first determination module for determining at least one video frame from the sub-videos as a keyframe of the sub-video, and determining an initial tag sequence of the keyframes, and determining a valid tag of the sub-video based on the initial tag sequence; and a second determination module for determining tags of the video sequence based on the valid tags of the sub-videos.

[0008] According to a fifth aspect, a video recommendation apparatus is provided, the apparatus comprising: an acquisition module for acquiring user tags; and a fifth determination module for determining a first target video from the video library for recommendation to the user based on a first similarity between the user tags and the tags of each video in the video library; wherein the video tags are obtained according to the aforementioned apparatus for determining video tags.

[0009] According to a sixth aspect, a video query apparatus is provided, the apparatus comprising: a receiving module for receiving a request to query a video, the request including a query term; and a sixth determining module for determining a second target video corresponding to the query term from the video library based on a second similarity between the query term and the tags of each video in the video library; wherein the video tags are obtained according to the aforementioned apparatus for determining video tags.

[0010] According to a seventh aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to the present disclosure.

[0011] According to an eighth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the methods provided in this disclosure.

[0012] According to a ninth aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method provided in this disclosure when executed by a processor.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is an exemplary system architecture diagram illustrating a method for determining video tags, a video recommendation method, and a video query method that can be applied according to an embodiment of the present disclosure;

[0016] Figure 2 This is a flowchart of a method for determining video tags according to an embodiment of the present disclosure;

[0017] Figure 3This is a block diagram of a method for determining video tags according to an embodiment of the present disclosure;

[0018] Figure 4 This is a flowchart of a method for determining video tags according to another embodiment of the present disclosure;

[0019] Figure 5 This is a flowchart of a video recommendation method according to an embodiment of the present disclosure;

[0020] Figure 6 This is a flowchart of a video query method according to an embodiment of the present disclosure;

[0021] Figure 7 This is a block diagram of an apparatus for determining video tags according to an embodiment of the present disclosure;

[0022] Figure 8 This is a block diagram of a video recommendation device according to an embodiment of the present disclosure;

[0023] Figure 9 This is a block diagram of a video query device according to an embodiment of the present disclosure;

[0024] Figure 10 This is a block diagram of an electronic device comprising at least one of a method for determining video tags, a video recommendation method, and a video query method according to an embodiment of the present disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] The mainstream method for video tag extraction is to first classify the video frame by frame to obtain the tag determination result of each frame, and then use a series of strategies to merge the tags of each frame and generate video-level tags.

[0027] However, videos contain a large amount of redundant information; almost every frame within a shot contains the same elements. Some briefly appearing content is not the main focus of the video, resulting in irrelevant and distracting entries in the output tags. Processing frame by frame results in a significant amount of redundant information, consuming substantial resources and leading to low processing efficiency when processing long videos.

[0028] In addition, frame-by-frame processing results in a low video processing speed ratio (video duration / video processing time). Real-time processing can only be achieved when the video processing speed ratio is greater than a certain value (e.g., 1). Therefore, it is difficult to achieve real-time video processing.

[0029] One method for determining video tags involves extracting one or more keyframes from a video, classifying the keyframes to obtain their tags, and then fusing the keyframe tags to obtain video-level tags.

[0030] This method for determining video tags can improve the efficiency of video processing, but the selection of keyframes becomes a challenge. Random or equally spaced sampling cannot guarantee the importance of the extracted keyframes, resulting in low accuracy and poor stability of the tag determination results. Extracting keyframes based on the importance of the images contained in a single frame also consumes a lot of computing resources, thus also resulting in low efficiency.

[0031] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0032] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0033] Figure 1 This is an exemplary system architecture diagram illustrating a method for determining video tags, a video recommendation method, and a video query method that can be applied according to an embodiment of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, etc.

[0036] At least one of the methods for determining video tags, recommending videos, and querying videos provided in this disclosure can generally be executed by server 105. Correspondingly, at least one of the apparatuses for determining video tags, recommending videos, and querying videos provided in this disclosure can generally be located in server 105. At least one of the methods for determining video tags, recommending videos, and querying videos provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103, and / or server 105. Correspondingly, at least one of the apparatuses for determining video tags, recommending videos, and querying videos provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103, and / or server 105.

[0037] Figure 2 This is a flowchart of a method for determining video tags according to an embodiment of the present disclosure.

[0038] like Figure 2 As shown, the method 200 for determining video tags may include operations S210 to S240.

[0039] In operation S210, the video sequence is divided into multiple sub-videos according to the shot.

[0040] For example, a video sequence can consist of multiple shots, each shot containing multiple frames. Dividing the video sequence into sub-videos at the granularity of a single shot yields multiple sub-videos. Each sub-video corresponds to one shot, and the multiple frames within each sub-video contain only one shot's image; there are no shot transitions within the sub-video.

[0041] In operation S220, at least one video frame is determined from the sub-video as the keyframe of the sub-video, and the initial label sequence of the keyframe is determined. Based on the initial label sequence, the valid labels of the sub-video are determined.

[0042] For example, keyframes can be selected from a single sub-video among multiple sub-videos, depending on actual needs. Keyframes can also be selected from any portion of the multiple sub-videos (e.g., any two sub-videos, any three sub-videos, etc.). Furthermore, keyframes can be selected from each sub-video itself.

[0043] For example, for each sub-video, a preset number (e.g., n, where n is an integer greater than 1, e.g., n=3) of video frames can be selected from the sub-video as keyframes according to the principle of equal division.

[0044] For example, for each sub-video, a preset number of video frames can be selected as keyframes according to the duration of the sub-video. For instance, for a sub-video with a duration of 10 seconds, 2 keyframes can be selected. For a sub-video with a duration of 20 seconds, 4 keyframes can be selected, and so on.

[0045] For example, keyframes can also be selected from sub-videos based on other metrics, such as resolution. For instance, the higher the resolution of a sub-video, the more keyframes are selected, and the lower the resolution of a sub-video, the fewer keyframes are selected.

[0046] Each sub-video corresponds to one shot. Since almost every frame in a shot contains the same elements, selecting a preset number of keyframes from the sub-video for subsequent image classification and label determination can reduce redundancy.

[0047] For example, keyframes in a sub-video can be classified to obtain keyframe labels. This can be done by classifying each keyframe individually, obtaining a label for each keyframe, or by classifying a single keyframe individually, or by classifying any subset of keyframes from multiple keyframes to obtain a label for each keyframe in that subset. A keyframe can have multiple labels, and each label can have a confidence score, which can be generated by a deep learning model (image classification model) when classifying the keyframes. The deep learning model can be a CNN (Convolutional Neural Network), etc.

[0048] Therefore, for a keyframe within a sub-video, the keyframe can have an initial label sequence, which can contain multiple labels arranged in descending order of confidence evaluation value. For example, the initial label sequence of a keyframe might be {sports, football, match, World Cup, celebrity}, where the confidence evaluation values ​​of "sports", "football", "match", "World Cup", and "celebrity" decrease sequentially.

[0049] For example, each keyframe within a sub-video may have an initial tag sequence.

[0050] For example, for each sub-video, the valid labels for that sub-video can be determined based on the initial label sequence of at least one keyframe of that sub-video. For example, the label with the highest confidence in each of the at least one initial label sequence can be determined as the valid label for that sub-video, and so on.

[0051] It is understandable that the effective labels for sub-videos are shot-level labels. Shot-level labels reduce redundant information compared to single-frame image labels and cover information from the entire video sequence.

[0052] In operation S230, the labels of the video sequence are determined based on the valid labels of the sub-videos.

[0053] For example, the label of a video sequence can be determined based on the valid labels of each of the multiple sub-videos. Alternatively, the label of a video sequence can be determined based on the valid label of a single sub-video. Furthermore, the label of a video sequence can be determined based on the valid labels of any subset of sub-videos (e.g., any two sub-videos, any three sub-videos, etc.).

[0054] For example, the valid tags of each of the multiple sub-videos can be added to the valid tag list, and then the tags in the valid tag list can be filtered using blocking words to obtain the tags of the video sequence. The tags of the video sequence are the video-level tags.

[0055] The embodiments of this disclosure divide a video sequence into multiple sub-videos according to shots, select at least one keyframe from the sub-videos for image classification, obtain an initial label sequence of at least one keyframe, determine the effective labels of the sub-videos, i.e., shot-level labels, based on the initial label sequence of at least one keyframe, and determine the video-level labels based on the shot-level labels, thereby reducing information redundancy and significantly improving processing efficiency.

[0056] Figure 3 This is a block diagram of a method for determining video tags according to an embodiment of the present disclosure.

[0057] like Figure 3 As shown, video sequence 300 can be divided into sub-videos 310, ..., and sub-video 320 according to the shots.

[0058] For sub-video 310, keyframes 311 and 312 are selected. For keyframe 311, image classification processing yields initial label sequence 313. For keyframe 312, image classification processing yields initial label sequence 314. From initial label sequences 313 and 314, the k labels with the highest confidence (topk, e.g., k=4) are selected respectively, yielding target label sequences 315 and 316. Based on target label sequences 315 and 316, effective shot-level labels 317 can be determined.

[0059] Similarly, for sub-video 320, keyframes 321 and 322 are selected. For keyframe 321, image classification processing yields initial label sequence 323. For keyframe 322, image classification processing yields initial label sequence 324. From initial label sequences 323 and 324, the k labels with the highest confidence (topk, e.g., k=4) are selected respectively, yielding target label sequences 325 and 326. Based on target label sequences 325 and 326, shot-level valid labels 327 can be determined.

[0060] Based on shot-level valid labels 317 and 327, video-level label 330 can be determined. For example, the set of shot-level valid labels 317 and 327 can be used as video-level label 330.

[0061] This embodiment of the disclosure segments the video according to shots, determines effective tags at the shot level, and determines video-level tags based on the effective tags at the shot level. This can reduce redundant information in shots and ensure that each shot covers the entire video, avoiding the loss of important information. Therefore, it can improve processing efficiency and improve the accuracy of tag prediction.

[0062] Figure 4 This is a flowchart of a method for determining video tags according to another embodiment of the present disclosure.

[0063] like Figure 4 As shown, the method includes operations S410 to S480. For example, operations S420 to S450 can be operations applied to each sub-video, an operation applied to a specific sub-video, or an operation applied to any portion of a sub-video. Operation S440 can be an operation applied to each keyframe, an operation applied to a specific keyframe, or an operation applied to any portion of keyframes among multiple keyframes.

[0064] In operation S410, the video sequence is divided into multiple sub-videos according to shots. Operation S410 is similar to operation S210, and will not be described in detail here.

[0065] In operation S420, at least one keyframe is determined from the sub-video, and the target label sequence of the keyframe is determined.

[0066] For example, at least one keyframe can be determined from the sub-video according to the principle of equal division, video duration, or resolution index. The method of selecting keyframes can refer to operation S220, which will not be repeated here.

[0067] For example, keyframes can be classified to obtain an initial label sequence for the keyframes. The k labels with the highest confidence (topk, e.g., k=4) can be selected from the initial label sequence as the target label sequence for the keyframes.

[0068] For example, each keyframe can be classified into images, and an initial label sequence and a target label sequence can be determined to obtain the target label sequence for each keyframe.

[0069] For example, a sub-video includes n (n is an integer greater than 1, such as n=3) keyframes, each keyframe having a target tag sequence, therefore, the sub-video includes n target tag sequences.

[0070] In operation S430, it is determined whether the top-1 label with the highest confidence score in the target label sequence of each of the n keyframes in the sub-video is consistent. If they are consistent, the label with the highest confidence score is determined as the valid label of the sub-video, and operation S450 is executed. Otherwise, operation S440 is executed.

[0071] In one example, for the current sub-video, there are 3 keyframes. The target label sequence of the first keyframe is {a, b, c, d}, the target label sequence of the second keyframe is {a, c, e, d}, and the target label sequence of the third keyframe is {a, d, e, f}. Since the top-1 label of all three keyframes is "a", "a" can be identified as the valid label of the current sub-video. This valid label is a shot-level label.

[0072] In another example, for the current sub-video, there are 3 keyframes. The target label sequence of the first keyframe is {b, c, d, e}, the target label sequence of the second keyframe is {a, c, e, d}, and the target label sequence of the third keyframe is {e, d, f, a}. The top 1 labels of these three keyframes are inconsistent, so operation S440 is performed for these three keyframes.

[0073] In operation S440, it is determined whether the top1 label in the target label sequence of the current frame exists in the topk of other frames. If it does, the top1 label of the current frame is determined to be a valid label, and operation S450 is executed; otherwise, it is determined that the current frame does not contain a valid label.

[0074] In one example, the target label sequence for the first keyframe is {b, c, d, e}, for the second keyframe it is {a, c, e, d}, and for the third keyframe it is {e, d, f, a}. Since the top-ranked label "a" from the second frame exists in the target label sequence of the third frame, and the top-ranked label "e" from the third frame exists in the target label sequences of both the first and second frames, labels "a" and "e" can be identified as valid labels for the current sub-video.

[0075] This embodiment uses shot segmentation as a pre-operator for label extraction. At least one keyframe is selected from the sub-video corresponding to the shot for label extraction, reducing information redundancy and significantly improving the operator's execution speed. Simultaneously, the significantly reduced number of selected keyframes, along with the inclusion of keyframe label consistency checks, greatly reduces the probability of output noise, improving the accuracy and stability of the results.

[0076] In operation S450, add the valid tags of the sub-video to the list of valid tags.

[0077] In one example, after obtaining the labels "a" and "e", the labels "a" and "e" can be added to the list of valid labels.

[0078] In another example, for each keyframe, if the result of operation S440 is negative, a valid tag for the current sub-video cannot be obtained. If no valid tags are obtained for any sub-video, the list of valid tags is empty.

[0079] In operation S460, determine if the list of valid tags is empty. If it is, proceed to operation S470; otherwise, proceed to operation S480.

[0080] In operation S470, candidate tags are determined from at least one target tag sequence of the sub-video.

[0081] For example, if the list of valid tags is empty, meaning no valid tags at the shot level have been obtained, it is necessary to retrieve some tags from at least one target tag sequence in the sub-video as candidate tags.

[0082] One strategy for retrieving partial tags involves forming a target tag sequence set from at least one target tag sequence of each of the sub-videos, and then identifying the tag with the highest confidence from the tag sequence set as a candidate tag.

[0083] For example, the set of target label sequences, consisting of at least one target label sequence from each of the sub-videos, contains N (e.g., N = 10) target sequence labels. The top-1 label of each target label sequence has a confidence score; for example, the confidence score of the top-1 label of the first target label sequence is 90%, the confidence score of the top-1 label of the second target label sequence is 75%, and so on. The top-1 label with the highest confidence score can be selected as a candidate label.

[0084] A strategy for retrieving partial tags involves identifying the longest sub-video from multiple sub-videos as a candidate sub-video, and identifying the tag with the highest confidence evaluation value in at least one target tag sequence of the candidate sub-video as a candidate tag.

[0085] For example, if sub-video A has the longest duration among multiple sub-videos, it can be identified as a candidate sub-video. This candidate sub-video includes n (e.g., n=3) target feature sequences. The top 1 label of each target feature sequence has a confidence evaluation value. The top 1 label with the highest confidence evaluation value can be selected as the candidate label.

[0086] This disclosure embodiment sets up a tag retrieval strategy. When no valid tags at the shot level are obtained, candidate tags are retrieved based on the tag retrieval strategy. Video-level tags are then determined based on these candidate tags, avoiding situations where video tags are empty. In one example, candidate tags are retrieved from at least one target tag sequence for each of multiple sub-videos. The target tag sequences for each sub-video cover the entire video sequence, thus avoiding the loss of important information, ensuring the accuracy of candidate tags, and consequently ensuring the accuracy of video-level tags.

[0087] When operating the S480, the tags in the valid tag list are filtered using masked words to obtain the tags of the video sequence.

[0088] For example, valid tags that match the blocked words in the valid tag list (e.g., have a similarity greater than 95% with the blocked words) can be identified, and valid tags that match the blocked words can be removed from the valid tag list. The final valid tag list can then be used as the tag list for the video sequence.

[0089] Figure 5 This is a flowchart of the video recommendation method disclosed herein.

[0090] like Figure 5 As shown, the video recommendation method 500 includes operations S510 to S520.

[0091] When operating S510, obtain user tags.

[0092] In operation S520, based on the first similarity between the user's tags and the tags of each video in the video library, the first target video to be recommended to the user is determined from the video library.

[0093] For example, video tags are determined using the method described above. User tags can be determined based on a user's browsing history and historical actions (such as saving and forwarding). By determining the first similarity between the user's tags and the tags of each video in the video library, the first target video with a first similarity higher than a threshold (e.g., 80%) can be recommended to the user.

[0094] For example, videos in a video library can also be stored based on video tags. For instance, videos tagged with "sports" can be stored together, videos tagged with "education" can be stored together, and so on, which facilitates the recommendation of videos with similar tags.

[0095] Figure 6 This is a flowchart of the video query method disclosed herein.

[0096] like Figure 6 As shown, the video query method 600 includes operations S610 to S620.

[0097] When operating the S610, a request to query video is received, and the request includes query terms.

[0098] In operation S620, based on the second similarity between the query term and the tags of each video in the video library, the second target video corresponding to the query term is determined from the video library.

[0099] For example, video tags are determined using the method described above. Users can search for desired videos by entering query terms on the client side. For instance, if a user enters "World Cup," a query request containing "World Cup" is generated. The client sends this query request to the server. In response, the server calculates the second similarity between the query term "World Cup" and the tags of each video in the video library. Videos with a second similarity greater than a threshold (e.g., 90%) are designated as the second target videos corresponding to "World Cup," and these second target videos can be sent to the user's client.

[0100] For example, videos in a video library can also be stored based on video tags. For instance, videos tagged with "sports" are stored together, videos tagged with "education" are stored together, and so on. Based on the query term, the video set with the corresponding category tag can be quickly located. For example, the query term "World Cup" can quickly locate "sports", thereby improving the efficiency of video search.

[0101] Figure 7This is a block diagram of an apparatus for determining video tags according to an embodiment of the present disclosure.

[0102] like Figure 7 As shown, the device 700 for determining video tags includes a segmentation module 701, a first determination module 702, and a second determination module 703.

[0103] The segmentation module 701 is used to divide the video sequence into multiple sub-videos according to shots.

[0104] The first determining module 702 is used to determine at least one video frame from the sub-video as a key frame of the sub-video, determine the initial label sequence of the key frame, and determine the valid labels of the sub-video based on the initial label sequence.

[0105] The second determining module 703 is used to determine the tags of the video sequence based on the valid tags of the sub-videos.

[0106] The first determination module includes a classification submodule, a selection submodule, and a first determination submodule.

[0107] The classification submodule is used to classify keyframes and obtain the initial label sequence of the keyframes.

[0108] The selection submodule is used to select the k tags with the highest confidence from the initial tag sequence of the keyframe as the target tag sequence of the keyframe, where k is an integer greater than 1.

[0109] The first determination submodule is used to determine the valid tags of the sub-video based on the target tag sequence of the keyframes.

[0110] The first determining submodule includes a first determining unit and a second determining unit.

[0111] The first determining unit is used to determine the label with the highest confidence as a valid label of the sub-video if the labels with the highest confidence in the target label sequences of at least one keyframe are consistent with each other.

[0112] The second determining unit is used to determine the valid labels of the sub-video based on the relationship between the highest confidence labels in the target label sequences and other target label sequences when the labels with the highest confidence in the target label sequences of at least one keyframe are not completely consistent.

[0113] The second determining unit is used to determine the label with the highest confidence in each target label sequence as a valid label of the target label sequence if the label with the highest confidence in the target label sequence exists in other target label sequences; and to determine the valid label of the sub-video based on the valid label of at least one target label sequence.

[0114] The second determination module includes an addition submodule and a filtering submodule.

[0115] The submodule is used to add valid tags from sub-videos to the list of valid tags.

[0116] The filtering submodule is used to filter the tags in the valid tag list using masked words to obtain the tags of the video sequence.

[0117] The device 700 for determining video tags also includes a third determining module and a fourth determining module.

[0118] The third determining module is used to determine candidate tags from at least one target tag sequence of the sub-video when the list of valid tags is determined to be empty.

[0119] The fourth determination module is used to determine the tags of the video sequence based on the candidate tags.

[0120] According to embodiments of this disclosure, the tags in the target tag sequence have confidence assessment values. The third determining module includes a combination submodule and a second determining submodule.

[0121] The combination submodule is used to combine at least one target label sequence from each of multiple sub-videos into a target label sequence set.

[0122] The second determination submodule is used to determine the label with the highest confidence evaluation value from the set of target label sequences as candidate labels.

[0123] According to embodiments of this disclosure, the tags in the target tag sequence have confidence assessment values. The third determination module includes a third determination submodule and a fourth determination submodule.

[0124] The third determination submodule is used to determine the longest subvideo from multiple subvideos as a candidate subvideo.

[0125] The fourth determination submodule is used to determine the label with the highest confidence evaluation value in at least one target label sequence of the candidate sub-video as the candidate label.

[0126] Figure 8 This is a block diagram of a video recommendation device according to an embodiment of the present disclosure.

[0127] like Figure 8 As shown, the video recommendation device 800 includes an acquisition module 801 and a fifth determination module 802.

[0128] The acquisition module 801 is used to acquire user tags.

[0129] The fifth determining module 802 is used to determine the first target video to recommend to the user from the video library based on the first similarity between the user's tags and the tags of each video in the video library.

[0130] The video tags are obtained using the aforementioned device 700 for determining video tags.

[0131] Figure 9 This is a block diagram of a video query device according to an embodiment of the present disclosure.

[0132] like Figure 9 As shown, the video query device 900 includes a receiving module 901 and a sixth determining module 902.

[0133] The receiving module 901 is used to receive requests for video queries, which include query terms.

[0134] The sixth determining module 902 is used to determine the second target video corresponding to the query term from the video library based on the second similarity between the query term and the tags of each video in the video library.

[0135] The video tags are obtained using the aforementioned device 700 for determining video tags.

[0136] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0137] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0138] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0139] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs at least one of the methods and processes described above, such as video tagging methods, video recommendation methods, and video query methods. For example, in some embodiments, at least one of the video tagging methods, video recommendation methods, and video query methods can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of at least one of the video tagging methods, video recommendation methods, and video query methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured by any other suitable means (e.g., by means of firmware) to perform at least one of a video tagging method, a video recommendation method, and a video query method.

[0141] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0145] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0146] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0147] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining video tags, comprising: The video sequence is divided into multiple sub-videos according to shots; At least one video frame is determined from the sub-video as the keyframe of the sub-video, and an initial tag sequence of the keyframe is determined. Based on the initial tag sequence, the valid tags of the sub-video are determined. as well as Based on the valid tags of the sub-videos, determine the tags of the video sequence. The step of determining at least one video frame from the sub-video as a keyframe of the sub-video, determining an initial tag sequence for the keyframe, and determining valid tags for the sub-video based on the initial tag sequence includes: The keyframes are classified to obtain the initial label sequence of the keyframes; Select the k tags with the highest confidence from the initial tag sequence of the keyframe as the target tag sequence of the keyframe, where k is an integer greater than 1; Based on the target tag sequence of the keyframes, determine the valid tags of the sub-video. The step of determining the effective tags of the sub-video based on the target tag sequence of the keyframes includes: If it is determined that the labels with the highest confidence in the target label sequences of at least one of the keyframes are consistent with each other, the label with the highest confidence is determined as the valid label of the sub-video; If it is determined that the highest-confidence labels in the target label sequences of at least one of the keyframes are not completely consistent, the effective labels of the sub-video are determined based on the relationship between the highest-confidence labels in the target label sequences and other target label sequences.

2. The method according to claim 1, wherein, The step of determining the effective tags of the sub-video based on the relationship between the tag with the highest confidence in the target tag sequence and other target tag sequences includes: For each target label sequence, if the label with the highest confidence in that target label sequence exists in other target label sequences, then the label with the highest confidence in that target label sequence is determined as a valid label of that target label sequence; and The valid tags of the sub-video are determined based on at least one valid tag of the target tag sequence.

3. The method according to claim 1 or 2, wherein, Determining the tags of the video sequence based on the valid tags of the sub-videos includes: Add the valid tags of the sub-video to the valid tag list; and The tags in the list of valid tags are filtered using the blocking words to obtain the tags for the video sequence.

4. The method according to claim 3, further comprising: If the list of valid tags is determined to be empty, candidate tags are determined from at least one target tag sequence of the sub-video; as well as The tags of the video sequence are determined based on the candidate tags.

5. The method according to claim 4, wherein, The tags in the target tag sequence have confidence evaluation values; determining candidate tags from at least one target tag sequence of the sub-video includes: Each of the plurality of sub-videos is combined into a target label sequence set by forming at least one target label sequence from each of the sub-videos; and The label with the highest confidence score is selected from the set of target label sequences as the candidate label.

6. The method according to claim 4, wherein, The tags in the target tag sequence have confidence evaluation values; determining candidate tags from at least one target tag sequence of the sub-video includes: From the plurality of sub-videos, determine the sub-video with the longest duration as a candidate sub-video; and The label with the highest confidence evaluation value in at least one target label sequence of the candidate sub-video is determined as the candidate label.

7. A video recommendation method, comprising: Obtain user tags; as well as Based on the first similarity between the user's tags and the tags of each video in the video library, a first target video is determined from the video library to be recommended to the user; The video tags are obtained according to the method described in any one of claims 1 to 6.

8. A video query method, comprising: Receive a request to query a video, the request including query terms; as well as Based on the second similarity between the query term and the tags of each video in the video library, a second target video corresponding to the query term is determined from the video library; The video tags are obtained according to the method described in any one of claims 1 to 6.

9. An apparatus for determining video tags, comprising: The segmentation module is used to divide a video sequence into multiple sub-videos according to shots; The first determining module is configured to determine at least one video frame from the sub-video as a keyframe of the sub-video, determine an initial label sequence of the keyframe, and determine a valid label of the sub-video based on the initial label sequence. as well as The second determining module is used to determine the tags of the video sequence based on the valid tags of the sub-videos. The first determining module includes: A classification submodule is used to classify the keyframes to obtain an initial label sequence for the keyframes; The selection submodule is used to select the k tags with the highest confidence from the initial tag sequence of the key frame as the target tag sequence of the key frame, where k is an integer greater than 1; The first determining submodule is used to determine the valid tags of the sub-video based on the target tag sequence of the keyframes. The first determining submodule includes: The first determining unit is configured to determine the label with the highest confidence as a valid label of the sub-video if the labels with the highest confidence in the target label sequences of at least one of the keyframes are consistent with each other. The second determining unit is configured to determine the valid label of the sub-video based on the relationship between the highest confidence label in the target label sequence and other target label sequences when it is determined that the highest confidence label in the target label sequence is not completely consistent in the target label sequence of at least one of the key frames.

10. The apparatus according to claim 9, wherein, The second determining unit is used to determine the label with the highest confidence in each target label sequence as a valid label of the target label sequence if the label with the highest confidence in the target label sequence exists in other target label sequences. And determine the valid tags of the sub-video based on at least one valid tag of the target tag sequence.

11. The apparatus according to any one of claims 9 to 10, wherein, The second determining module includes: Add a submodule to add the valid tags of the sub-video to the valid tag list; and The filtering submodule is used to filter the tags in the list of valid tags using masked words to obtain the tags of the video sequence.

12. The apparatus of claim 11, further comprising: The third determining module is used to determine candidate tags from at least one target tag sequence of the sub-video when the effective tag list is determined to be empty; as well as The fourth determining module is used to determine the tags of the video sequence based on the candidate tags.

13. The apparatus according to claim 12, wherein, The tags in the target tag sequence have confidence evaluation values; the third determining module includes: A combination submodule is used to combine at least one target tag sequence from each of the plurality of sub-videos into a target tag sequence set; and The second determining submodule is used to determine the label with the highest confidence evaluation value from the target label sequence set as a candidate label.

14. The apparatus according to claim 12, wherein, The tags in the target tag sequence have confidence evaluation values; the third determining module includes: The third determining submodule is used to determine the longest subvideo from the plurality of subvideos as a candidate subvideo; and The fourth determination submodule is used to determine the label with the highest confidence evaluation value in at least one target label sequence of the candidate sub-video as the candidate label.

15. A video recommendation device, comprising: The acquisition module is used to acquire user tags; as well as The fifth determining module is used to determine a first target video to recommend to the user from the video library based on the first similarity between the user tag and the tags of each video in the video library; The video tags are obtained according to the apparatus as described in any one of claims 9 to 14.

16. A video query device, comprising: A receiving module is used to receive requests to query videos, the requests including query terms; as well as The sixth determining module is used to determine a second target video corresponding to the query term from the video library based on the second similarity between the query term and the tags of each video in the video library; The video tags are obtained according to the apparatus as described in any one of claims 9 to 14.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1 to 8 when executed by a processor.

Images