Short video tag determination method and device, equipment and medium

By inputting the frame candidate images of short videos into the video tag determination model for vectorization and object detection, combined with the confidence combination of the image blocks with or without labels, the video tag is automatically determined, which solves the inefficient and insufficient accuracy in the prior art. Problem, efficient short video tag extraction is achieved.

CN118945394APending Publication Date: 2024-11-12GUANGZHOU FANGZHOU INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411138245.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing text-based short video tag extraction methods are inefficient, rely on manual annotation, and fail to effectively utilize the diverse information in the video content, resulting in insufficient accuracy and efficiency of tag extraction.

Method used

By inputting the frame candidate video images of the short video to be marked with the trained video tag to determine the model, perform vectorization processing and object detection, combining the confidence combination of labeled and unlabeled image blocks, the video tag is automatically determined, and the tag extraction efficiency is improved .

Benefits of technology

Automatic short video tag determination is realized, which improves the efficiency and accuracy of tag extraction, reduces the dependence on manual tagging, and improves the service quality and content management level of the short video platform.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945394A_ABST
    Figure CN118945394A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a short video tag determination method and device, equipment and a medium. The method comprises the following steps: inputting a current video image into a video tag determination model to obtain a current image vector; determining current video image blocks of the current video image, and performing target detection on each current video image block to obtain a labeled image block and an unlabeled image block; determining a label-free image block vector, and determining a label-free confidence coefficient vector according to the current image vector, the label-free image block vector and a preset label vector; according to the label-free confidence vector, determining a first confidence group of the label-free image block, and determining a second confidence group of the label image block; and outputting an image confidence group according to the first confidence group and the second confidence group, and determining a target video tag according to the image confidence group of each candidate video image. The label of the short video is automatically determined, and the determining efficiency of the label of the short video is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of video processing technology, and in particular, to a method, device, equipment and medium for determining a short video tag. Background Art

[0002] With the rise of short video platforms, there is a growing demand for efficient and accurate video content labels. Although video content itself carries a lot of information, the extraction process of this information is complex and costly, resulting in the fact that label extraction methods based on video content are not common. At present, the industry generally adopts label extraction methods based on text information, because text information is relatively easy to process and has low cost. However, existing text-based label extraction technology relies on manual annotation and is inefficient. Therefore, how to improve the determination of short video labels is crucial. Summary of the invention

[0003] The present invention provides a short video tag determination method, device, equipment and medium to automatically determine the tags of short videos and improve the efficiency of determining the short video tags.

[0004] According to one aspect of the present invention, a method for determining a short video tag is provided, comprising:

[0005] Input any candidate video image frame in the short video to be labeled as the current video image into the trained video label determination model, and perform vectorization processing on the current video image to obtain a current image vector;

[0006] Determine a current video image block of the current video image, perform target detection on each of the current video image blocks, and use the current video image block with label associated data as a labeled image block, and use the current video image block without label associated data as an unlabeled image block;

[0007] Determine an unlabeled image block vector of the unlabeled image block, and determine an unlabeled confidence vector according to the current image vector, the unlabeled image block vector and a preset label vector;

[0008] Determine a first confidence group of the unlabeled image block according to the unlabeled confidence vector, and determine a corresponding second confidence group according to the label association data in the labeled image block;

[0009] According to the first confidence group and the second confidence group, the image confidence group of the current video image is output, and according to the image confidence groups of each of the candidate video images, the target video label of the short video to be labeled is determined.

[0010] According to another aspect of the present invention, a short video tag determination device is provided, comprising:

[0011] A current image vector determination module is used to input any candidate video image frame in the short video to be labeled as the current video image into the trained video label determination model, and perform vectorization processing on the current video image to obtain a current image vector;

[0012] An image segmentation module, used to determine a current video image block of the current video image, perform target detection on each of the current video image blocks, and regard the current video image block with label associated data as a labeled image block, and regard the current video image block without label associated data as an unlabeled image block;

[0013] an unlabeled vector determination module, configured to determine an unlabeled image block vector of the unlabeled image block, and determine an unlabeled confidence vector according to the current image vector, the unlabeled image block vector and a preset label vector;

[0014] A confidence group determination module, configured to determine a first confidence group of the unlabeled image block according to the unlabeled confidence vector, and determine a corresponding second confidence group according to the label association data in the labeled image block;

[0015] A video label determination module is used to output the image confidence group of the current video image according to the first confidence group and the second confidence group, and determine the target video label of the short video to be labeled according to the image confidence group of each candidate video image.

[0016] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When one or more programs are executed by one or more processors, the one or more processors can execute any one of the short video tag determination methods provided by the embodiments of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement any short video tag determination method provided by an embodiment of the present invention when executed.

[0021] The embodiment of the present invention provides a short video label determination scheme, which comprises the following steps: inputting any frame of candidate video image in a short video to be labeled as a current video image into a trained video label determination model, performing vectorization processing on the current video image to obtain a current image vector; determining a current video image block of the current video image, performing target detection on each current video image block, treating the current video image block with label associated data as a labeled image block, and treating the current video image block without label associated data as an unlabeled image block; determining an unlabeled image block vector of the unlabeled image block, and determining an unlabeled confidence vector based on the current image vector, the unlabeled image block vector and a preset label vector; determining a first confidence group of the unlabeled image block based on the unlabeled confidence vector, and determining a corresponding second confidence group based on the label associated data in the labeled image block; outputting an image confidence group of the current video image based on the first confidence group and the second confidence group, and determining a target video label of the short video to be labeled based on the image confidence groups of each candidate video image. The above scheme, by introducing a video label determination model and outputting an image confidence group, further determines the target video label, thereby realizing automatic determination of short video labels and improving the efficiency of determining short video labels.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a short video tag determination method provided by Embodiment 1 of the present invention;

[0025] Figure 2A is a flow chart of a short video tag determination method provided by Embodiment 2 of the present invention;

[0026] Figure 2B is a schematic diagram of video image segmentation provided by Embodiment 2 of the present invention;

[0027] Figure 2C is a schematic diagram of a process for determining an unlabeled confidence vector provided in Embodiment 2 of the present invention;

[0028] Figure 3It is a structural schematic diagram of a short video tag determination device provided by Embodiment 3 of the present invention;

[0029] Figure 4 It is a structural schematic diagram of an electronic device for implementing a short video tag determination method provided by a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Deep learning models in the existing technology, such as convolutional neural networks (CNNs) and self-attention mechanisms, have been tried to automatically generate video labels, but these models require a large amount of training data, and due to the diversity and subjectivity of video labels, the model training process is complicated and difficult to meet the requirements of practical applications. In addition, existing methods fail to fully consider the differences between different types of information in the video screen, such as subtitles, brand logos, advertising content, and character images, which leads to a waste of computing resources and a decrease in the accuracy of label extraction.

[0032] Therefore, developing a new method that can automatically generate short video tags to improve the efficiency and accuracy of tag extraction is of great significance to improving the service quality and content management level of short video platforms.

[0033] Embodiment 1

[0034] Figure 1 It is a flowchart of a short video tag determination method provided in Example 1 of the present invention. This embodiment can be applicable to the situation of determining the tag of a short video. The method can be executed by a short video tag determination device, which can be implemented in software and / or hardware and can be configured in an electronic device that carries the short video tag determination function.

[0035] See also Figure 1 The short video tag determination method shown includes:

[0036] S110, inputting any frame of candidate video image in the short video to be labeled as the current video image into the trained video label determination model, performing vectorization processing on the current video image, and obtaining a current image vector.

[0037] Among them, the short video to be labeled refers to the short video whose video label needs to be determined. The embodiment of the present invention does not impose any restrictions on the short video to be labeled, and it can be set by the technician based on experience. The candidate video image refers to each frame image in the short video to be labeled. The current video image refers to the candidate video image input into the video label determination model at the current moment.

[0038] The video tag determination model refers to a model that can be used to determine the confidence of an image in a short video under each video tag. The current image vector refers to the image vector of the current video image.

[0039] Exemplarily, the current video image is resized according to the preset image size to obtain the adjusted current video image; the adjusted current video image is vector-converted to obtain the current image vector. The embodiment of the present invention does not impose any limitation on the size of the preset image size, which can be set by a technician based on experience or repeatedly determined through a large number of experiments. The benefit of introducing the preset image size is that the current video image can be compressed to improve the efficiency of subsequent processing.

[0040] S120, determining the current video image blocks of the current video image, performing target detection on each current video image block, treating the current video image blocks with label associated data as labeled image blocks, and treating the current video image blocks without label associated data as unlabeled image blocks.

[0041] The current video image block refers to an image block including at least part of the current video image content. Exemplarily, the current video image is processed into blocks to obtain a plurality of current video image blocks.

[0042] The present invention does not impose any limitation on the target detection method, which can be set by a technician based on experience. For example, the target detection can be a target detection algorithm of the YOLO (You Only Look Once) series.

[0043] Among them, the tag-associated data refers to the associated data that can directly determine the video tag. Exemplarily, the tag-associated data may include at least one of trademarks, subtitles, text descriptions, and advertisements. A labeled image block refers to a current video image block that has tag-associated data. An unlabeled image block refers to a current video image block that does not have tag-associated data.

[0044] S130, determining an unlabeled image block vector of the unlabeled image block, and determining an unlabeled confidence vector according to the current image vector, the unlabeled image block vector and the preset label vector.

[0045] Among them, the unlabeled image block vector refers to the vector corresponding to the unlabeled image block. The preset label vector refers to the vector of the candidate video label. The candidate video label refers to the video label pre-set in the video label determination model. The embodiment of the present invention does not impose any restrictions on the setting of the candidate video label, which can be set by technicians based on experience. The video label can be understood as the category to which the short video belongs.

[0046] The unlabeled confidence vector refers to a video label confidence vector determined based on the unlabeled image block.

[0047] In an optional embodiment, an unlabeled confidence vector is determined based on a current image vector, an unlabeled image block vector, and a preset label vector, including: performing feature extraction on the unlabeled image block vector to obtain an unlabeled key vector; and determining an unlabeled confidence vector based on the current image vector, the unlabeled key vector, and the preset label vector.

[0048] Among them, the unlabeled key vector refers to the key information vector extracted from the unlabeled image block vector.

[0049] It can be understood that by performing feature extraction on the unlabeled image block vector to obtain the unlabeled key vector, the extraction of key information in the unlabeled image block vector is achieved, irrelevant data is avoided from participating in subsequent processing, and the efficiency of determining the unlabeled confidence vector is improved.

[0050] S140 . Determine a first confidence group of the unlabeled image block according to the unlabeled confidence vector, and determine a corresponding second confidence group according to the label association data in the labeled image block.

[0051] The first confidence group refers to a data group of corresponding confidences of each candidate video label determined based on the unlabeled confidence vector, and the second confidence group refers to a data group of confidences of labeled image blocks under each candidate video label.

[0052] It should be noted that the sum of the confidences in the first confidence group is 1, and the sum of the confidences in the second confidence group is 1. For example, if the number of candidate video tags is 4, the first confidence group may be {0.62, 0.22, 0.07, 0.09}, and the second confidence group may be {0, 0, 1, 0}.

[0053] It should be noted that, since the number of labeled image blocks may be at least one, the second confidence group corresponding to the labeled image block is determined based on the label association data in each labeled image block, that is, the number of second confidence groups may be multiple. Specifically, the number of second confidence groups is determined based on the number of labeled image blocks, and one labeled image block corresponds to one second confidence group.

[0054] It should be noted that there may be multiple unlabeled image blocks, and multiple unlabeled image blocks correspond to one first confidence group, that is, there is only one first confidence group.

[0055] It should be noted that if all current video image blocks are unlabeled image blocks, the number of the first confidence group is one and the number of the second confidence group is zero.

[0056] It should be noted that the confidences corresponding to the candidate video labels have the same position in the first confidence group and the second confidence group and are pre-set, that is, the confidence corresponding to which candidate video label can be determined according to the position of the confidence in the first confidence group or the second confidence group.

[0057] Exemplarily, for any labeled image block, the unique candidate video label corresponding to the labeled image block is determined according to the label association data in the labeled image block, and the confidence of the unique candidate video label is set to 1 to generate a second confidence group containing 0 and 1, such as {0,1,0,0}.

[0058] S150: outputting an image confidence group of the current video image according to the first confidence group and the second confidence group, and determining a target video label of the short video to be labeled according to the image confidence groups of each candidate video image.

[0059] The image confidence group refers to a combination of the confidences of the current video image under each candidate video label.

[0060] The target video tag refers to the video tag of the short video to be labeled.

[0061] In an optional embodiment, based on the first confidence group and the second confidence group, an image confidence group of the current video image is output, including: averaging the confidences corresponding to the same candidate video tag in the first confidence group and the second confidence group to obtain a mean confidence; and outputting the image confidence group of the current video image including the mean confidence.

[0062] The mean confidence can be used to quantify the possibility that the current video image belongs to a candidate video tag. A candidate video tag refers to a pre-set short video tag. There can be multiple candidate video tags.

[0063] It can be understood that by introducing the mean confidence, a comprehensive consideration of labeled image blocks and unlabeled image blocks is achieved, which avoids the occurrence of extreme confidence in the image confidence group and improves the reliability and accuracy of the image confidence group.

[0064] In an optional embodiment, the target video label of the short video to be labeled is determined according to the image confidence group of each candidate video image, including: averaging the mean confidence corresponding to the same candidate video label in the image confidence group of each candidate video image to obtain the candidate image confidence; and determining the target video label of the short video to be labeled according to the candidate image confidence corresponding to each candidate video label.

[0065] Among them, the candidate image confidence can be used to quantify the possibility that the short video to be labeled belongs to a certain candidate video label.

[0066] Exemplarily, the target video label of the short video to be labeled is determined according to the candidate image confidences corresponding to each candidate video label, including: sorting the candidate image confidences, and selecting the candidate video label corresponding to the largest candidate image confidence from the sorting results as the target video label of the short video to be labeled.

[0067] It can be understood that by introducing the confidence of the candidate image, comprehensive consideration of each frame of the candidate video image is achieved, thereby improving the accuracy and reliability of the determined target video label.

[0068] The embodiment of the present invention provides a short video label determination scheme, which comprises the following steps: inputting any frame of candidate video image in a short video to be labeled as a current video image into a trained video label determination model, performing vectorization processing on the current video image to obtain a current image vector; determining a current video image block of the current video image, performing target detection on each current video image block, treating the current video image block with label associated data as a labeled image block, and treating the current video image block without label associated data as an unlabeled image block; determining an unlabeled image block vector of the unlabeled image block, and determining an unlabeled confidence vector based on the current image vector, the unlabeled image block vector and a preset label vector; determining a first confidence group of the unlabeled image block based on the unlabeled confidence vector, and determining a corresponding second confidence group based on the label associated data in the labeled image block; outputting an image confidence group of the current video image based on the first confidence group and the second confidence group, and determining a target video label of the short video to be labeled based on the image confidence groups of each candidate video image. The above scheme, by introducing a video label determination model and outputting an image confidence group, further determines the target video label, thereby realizing automatic determination of short video labels and improving the efficiency of determining short video labels.

[0069] On the basis of the above technical solution, an embodiment of the present invention further provides another short video label determination method, including: inputting each frame of candidate video image in the short video to be labeled into a trained video label determination model, and performing vectorization processing on the candidate video image to obtain a candidate image vector; according to the video resolution of the short video to be labeled, each candidate video image is divided into blocks to obtain a candidate video image block; performing target detection on each candidate video image block, treating the candidate video image block with label associated data as a labeled image block, and treating the candidate video image block without label associated data as an unlabeled image block; determining The unlabeled image block vector of the unlabeled image block is obtained, and the unlabeled confidence vector is determined according to the candidate image vector, the unlabeled image block vector and the preset label vector; the first confidence group of the unlabeled image block is determined according to the unlabeled confidence vector, and the corresponding second confidence group is determined according to the label association data in the labeled image block; the confidence corresponding to the same candidate video label in the first confidence group and the second confidence group is averaged to obtain the video confidence; the video confidence group of the short video to be labeled including the video confidence is output; the candidate video label corresponding to the maximum video confidence in the video confidence group is used as the target video label.

[0070] Among them, the candidate image vector refers to the image vector determined based on each candidate video image. The video resolution refers to the resolution of the short video to be labeled. The candidate video image block refers to the image block containing at least part of the candidate video image content. The video confidence group refers to the combination of the confidence of the short video to be labeled under each candidate video label. The video confidence can be used to quantify the possibility that the short video to be labeled belongs to a certain candidate video label.

[0071] Embodiment 2

[0072] Figure 2A This is a flow chart of a short-time frequency label determination method provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment further refines the operation of "determining the current video image block of the current video image" into "determining the image resolution of the current video image, and according to the image resolution, dividing the current video image into blocks to obtain the current video image block" to improve the determination mechanism of the current video image block. It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments.

[0073] See also Figure 2A The short video tag determination method shown includes:

[0074] S210, input any frame of candidate video image in the short video to be labeled as the current video image into the trained video label determination model, and perform vectorization processing on the current video image to obtain a current image vector.

[0075] S220: Determine the image resolution of the current video image, and divide the current video image into blocks according to the image resolution to obtain current video image blocks.

[0076] The image resolution refers to the resolution of the current video image. For example, the image resolution may be the video resolution of the short video to be annotated.

[0077] In an optional embodiment, the current video image is divided into blocks according to the image resolution to obtain the current video image blocks, including: if the length in the image resolution satisfies a preset first blocking condition, the current video image is divided into blocks according to a first blocking strategy to obtain the current video image blocks; wherein the first blocking condition is that the length in the image resolution is greater than or equal to twice the height in the image resolution; if the height in the image resolution satisfies a preset second blocking condition, the current video image is divided into blocks according to the second blocking strategy to obtain the current video image blocks; wherein the second blocking condition is that the height in the image resolution is greater than or equal to twice the length in the image resolution; if the image resolution does not satisfy any one of the preset first blocking condition and the preset second blocking condition, the current video image is divided into blocks according to a third blocking strategy to obtain the current video image blocks.

[0078] Among them, the first block strategy refers to dividing the current video image into blocks according to 2*4 (i.e., 2 rows and 4 columns) to obtain eight current video image blocks. The second block strategy refers to dividing the current video image into blocks according to 4*2 (i.e., 4 rows and 2 columns) to obtain eight current video image blocks. The third block strategy refers to dividing the current video image into blocks according to 4*4 (i.e., 4 rows and 4 columns) to obtain 16 current video image blocks.

[0079] For example, see Figure 2B Schematic diagram of video image segmentation shown in FIG. 1 , wherein (a) represents the first segmentation strategy, (b) represents the second segmentation strategy, and (c) represents the third segmentation strategy.

[0080] The preset first block condition refers to that the length in the image resolution is greater than or equal to twice the height. The preset second block condition refers to that the height in the image resolution is greater than or equal to twice the length.

[0081] It can be understood that by introducing the first blocking strategy, the second blocking strategy and the third blocking strategy, the current video image in different situations is blocked respectively, which improves the diversity of the blocking strategies and improves the applicability of the determined current video image blocks.

[0082] S230 , performing target detection on each current video image block, treating the current video image block with label associated data as a labeled image block, and treating the current video image block without label associated data as an unlabeled image block.

[0083] S240, determining an unlabeled image block vector of the unlabeled image block, and determining an unlabeled confidence vector according to the current image vector, the unlabeled image block vector and the preset label vector.

[0084] S250 . Determine a first confidence group of the unlabeled image block according to the unlabeled confidence vector, and determine a corresponding second confidence group according to the label association data in the labeled image block.

[0085] S260: outputting an image confidence group of the current video image according to the first confidence group and the second confidence group, and determining a target video label of the short video to be labeled according to the image confidence groups of each candidate video image.

[0086] The embodiment of the present invention provides a short video label determination scheme, which refines the operation of determining the current video image block of the current video image into determining the image resolution of the current video image, and divides the current video image into blocks according to the image resolution to obtain the current video image block, thereby improving the determination mechanism of the current video image block. The above scheme improves the accuracy of the determined current video image block by dividing the current video image into blocks according to the image resolution to obtain the current video image block.

[0087] Based on the above technical solution, a short video label determination method based on visual segmentation is provided, which includes five steps: video adaptive segmentation algorithm, target detection algorithm, labeled image block processing, unlabeled image block processing and determining target video labels.

[0088] Exemplary, video adaptive block algorithm: an adaptive block algorithm based on the short video resolution is proposed to block the current video image. When the short video resolution is greater than or equal to 2 times the length, the current video image is divided into 2*4 current video image blocks; when the short video resolution is greater than or equal to 2 times the length, the current video image is divided into 4*2 current video image blocks; for other cases, the current video image is divided into 4*4 current video image blocks.

[0089] Exemplary target detection algorithm: perform target detection on several current video image blocks after segmentation, such as using the target detection algorithm of the YOLO series, determine whether the current video image block contains label-related data, mark all current video image blocks containing label-related data as labeled image blocks, and mark other current video image blocks as unlabeled image blocks. After the target detection is completed, determine whether to perform target detection again for the unlabeled image blocks based on the detection situation.

[0090] Exemplarily, the labeled image block processing includes determining a second confidence group of the labeled image block based on the label association data in the labeled image block.

[0091] For example, the unlabeled image block processing process can be found in Figure 2C , where the unlabeled image block vector first passes through N temporal sparse attention layers to extract shallow features and obtain an unlabeled key vector; the current image vector, the unlabeled key vector and the preset label vector (obtained by Gaussian initialization of the candidate video labels) of the current video image pass through M temporal cross attention layers to fuse the information of the original image and the block image, and finally output the unlabeled confidence vector.

[0092] Exemplarily, the target video label is determined: for the unlabeled confidence vector, an MLP (Multilayer Perceptron) module is passed to obtain an image confidence group; according to the image confidence group of each candidate video image, the target video label of the short video to be labeled is determined. Among them, the MLP can be trained using a preset label library. The preset label library refers to a pre-set database for storing video labels.

[0093] Embodiment 3

[0094] Figure 3 1 is a schematic diagram of the structure of a short video tag determination device provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of determining the tag of a short video, and the method can be performed by a short video tag determination device, which can be implemented in software and / or hardware, and can be configured in an electronic device that carries the short video tag determination function.

[0095] like Figure 3 As shown, the device includes: a current image vector determination module 310, an image block module 320, an unlabeled vector determination module 330, a confidence group determination module 340 and a video label determination module 350. Among them,

[0096] The current image vector determination module 310 is used to input any candidate video image frame in the short video to be labeled as the current video image into the trained video label determination model, and perform vectorization processing on the current video image to obtain the current image vector;

[0097] An image segmentation module 320 is used to determine a current video image block of the current video image, perform target detection on each of the current video image blocks, and use the current video image block with label associated data as a labeled image block, and use the current video image block without label associated data as an unlabeled image block;

[0098] An unlabeled vector determination module 330, configured to determine an unlabeled image block vector of the unlabeled image block, and determine an unlabeled confidence vector according to the current image vector, the unlabeled image block vector and a preset label vector;

[0099] A confidence group determination module 340 is used to determine a first confidence group of the unlabeled image block according to the unlabeled confidence vector, and to determine a corresponding second confidence group according to the label association data in the labeled image block;

[0100] The video label determination module 350 is used to output the image confidence group of the current video image according to the first confidence group and the second confidence group, and determine the target video label of the short video to be labeled according to the image confidence group of each candidate video image.

[0101] The embodiment of the present invention provides a short video label determination scheme, which comprises the following steps: inputting any frame of candidate video image in a short video to be labeled as a current video image into a trained video label determination model, performing vectorization processing on the current video image to obtain a current image vector; determining a current video image block of the current video image, performing target detection on each current video image block, treating the current video image block with label associated data as a labeled image block, and treating the current video image block without label associated data as an unlabeled image block; determining an unlabeled image block vector of the unlabeled image block, and determining an unlabeled confidence vector based on the current image vector, the unlabeled image block vector and a preset label vector; determining a first confidence group of the unlabeled image block based on the unlabeled confidence vector, and determining a corresponding second confidence group based on the label associated data in the labeled image block; outputting an image confidence group of the current video image based on the first confidence group and the second confidence group, and determining a target video label of the short video to be labeled based on the image confidence groups of each candidate video image. The above scheme, by introducing a video label determination model and outputting an image confidence group, further determines the target video label, thereby realizing automatic determination of short video labels and improving the efficiency of determining short video labels.

[0102] Optionally, the image segmentation module 320 includes:

[0103] The image blocking unit is used to determine the image resolution of the current video image, and according to the image resolution, block the current video image to obtain the current video image block.

[0104] Optionally, the image segmentation unit is specifically used for:

[0105] If the length in the image resolution satisfies a preset first block condition, the current video image is block processed according to a first block strategy to obtain the current video image block; wherein the first block condition is that the length in the image resolution is greater than or equal to twice the height in the image resolution;

[0106] If the high part of the image resolution satisfies a preset second block condition, the current video image is block processed according to a second block strategy to obtain the current video image block; wherein the second block condition is that the high part of the image resolution is greater than or equal to twice the length of the image resolution;

[0107] If the image resolution does not satisfy any one of the preset first blocking condition and the preset second blocking condition, the current video image is subjected to blocking processing according to a third blocking strategy to obtain the current video image block.

[0108] Optionally, the video tag determination module 350 includes:

[0109] A mean confidence determination unit, configured to average the confidences corresponding to the same candidate video tag in the first confidence group and the second confidence group to obtain a mean confidence;

[0110] An image confidence group determining unit is used to output an image confidence group of the current video image including the mean confidence.

[0111] Optionally, the video tag determination module 350 includes:

[0112] A candidate confidence determination unit, configured to average the mean confidences corresponding to the same candidate video label in the image confidence groups of the candidate video images to obtain the candidate image confidences;

[0113] The video label determination unit is used to determine the target video label of the short video to be labeled according to the confidence of the candidate image corresponding to each candidate video label.

[0114] Optionally, the unlabeled vector determination module 330 includes:

[0115] Performing feature extraction on the unlabeled image block vector to obtain an unlabeled key vector;

[0116] The unlabeled confidence vector is determined according to the current image vector, the unlabeled key vector and the preset label vector.

[0117] The short video tag determination device provided in the embodiment of the present invention can execute the short video tag determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each short video tag determination method.

[0118] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of short videos to be marked, etc., are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0119] Embodiment 4

[0120] Figure 4 It is a structural diagram of an electronic device for implementing a short video tag determination method provided by Embodiment 4 of the present invention. Electronic device 410 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0121] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0122] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 411 performs the various methods and processes described above, such as a short video tag determination method.

[0124] In some embodiments, the short video tag determination method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the short video tag determination method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to perform the short video tag determination method in any other appropriate manner (e.g., by means of firmware).

[0125] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0130] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0131] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0132] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining short video tags, characterized in that: include: Input any candidate video image frame in the short video to be labeled as the current video image into the trained video label determination model, and perform vectorization processing on the current video image to obtain a current image vector; Determine a current video image block of the current video image, perform target detection on each of the current video image blocks, and use the current video image block with label associated data as a labeled image block, and use the current video image block without label associated data as an unlabeled image block; Determine an unlabeled image block vector of the unlabeled image block, and determine an unlabeled confidence vector according to the current image vector, the unlabeled image block vector and a preset label vector; Determine a first confidence group of the unlabeled image block according to the unlabeled confidence vector, and determine a corresponding second confidence group according to the label association data in the labeled image block; According to the first confidence group and the second confidence group, the image confidence group of the current video image is output, and according to the image confidence groups of each of the candidate video images, the target video label of the short video to be labeled is determined.

2. The method according to claim 1, characterized in that The determining of the current video image block of the current video image comprises: The image resolution of the current video image is determined, and according to the image resolution, the current video image is processed into blocks to obtain the current video image blocks.

3. The method according to claim 2, characterized in that The step of performing block processing on the current video image according to the image resolution to obtain the current video image block comprises: If the length in the image resolution satisfies a preset first block condition, the current video image is block processed according to a first block strategy to obtain the current video image block; wherein the first block condition is that the length in the image resolution is greater than or equal to twice the height in the image resolution; If the high part of the image resolution satisfies a preset second block condition, the current video image is block processed according to a second block strategy to obtain the current video image block; wherein the second block condition is that the high part of the image resolution is greater than or equal to twice the length of the image resolution; If the image resolution does not satisfy any one of the preset first blocking condition and the preset second blocking condition, the current video image is subjected to blocking processing according to a third blocking strategy to obtain the current video image block.

4. The method according to claim 1, characterized in that: Outputting the image confidence group of the current video image according to the first confidence group and the second confidence group includes: averaging the confidences corresponding to the same candidate video label in the first confidence group and the second confidence group to obtain an average confidence; An image confidence group of the current video image including the mean confidence is output.

5. The method according to claim 4, characterized in that The step of determining the target video label of the short video to be labeled according to the image confidence group of each candidate video image includes: averaging the mean confidences corresponding to the same candidate video label in the image confidence groups of the candidate video images to obtain the candidate image confidences; According to the confidence level of the candidate images corresponding to the candidate video tags, the target video tag of the short video to be labeled is determined.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining the unlabeled confidence vector according to the current image vector, the unlabeled image block vector and the preset label vector comprises: Performing feature extraction on the unlabeled image block vector to obtain an unlabeled key vector; The unlabeled confidence vector is determined according to the current image vector, the unlabeled key vector and the preset label vector.

7. A short video tag determination device, characterized in that: include: A current image vector determination module is used to input any candidate video image frame in the short video to be labeled as the current video image into the trained video label determination model, and perform vectorization processing on the current video image to obtain a current image vector; An image segmentation module, used to determine a current video image block of the current video image, perform target detection on each of the current video image blocks, and regard the current video image block with label associated data as a labeled image block, and regard the current video image block without label associated data as an unlabeled image block; an unlabeled vector determination module, configured to determine an unlabeled image block vector of the unlabeled image block, and determine an unlabeled confidence vector according to the current image vector, the unlabeled image block vector and a preset label vector; A confidence group determination module, configured to determine a first confidence group of the unlabeled image block according to the unlabeled confidence vector, and determine a corresponding second confidence group according to the label association data in the labeled image block; A video label determination module is used to output the image confidence group of the current video image according to the first confidence group and the second confidence group, and determine the target video label of the short video to be labeled according to the image confidence group of each candidate video image.

8. The device according to claim 7, characterized in that The image segmentation module comprises: The image blocking unit is used to determine the image resolution of the current video image, and to block the current video image according to the image resolution to obtain the current video image blocks.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a short video tag determination method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a short video tag determination method as described in any one of claims 1 to 6 is implemented.