Short video tag extraction method and device, equipment and medium

Abstract

The embodiment of the invention discloses a short video tag extraction method and device, equipment and a medium. The method comprises the following steps: determining at least one key frame picture from a short video to be processed, and vectorizing each key frame picture to obtain a key picture vector; determining candidate vector similarities according to candidate picture vectors corresponding to the candidate video tags stored in a vector database and the key picture vectors, and selecting a preset first selection threshold number of key vector similarities from the candidate vector similarities; taking the candidate video tag corresponding to the key vector similarity as a reference video tag, and determining a reference video picture corresponding to the reference video tag; and vectorizing the reference video picture to obtain a reference picture vector, and determining a target video tag of the to-be-processed short video according to the reference picture vector and the candidate picture vector. According to the scheme, the short video tag extraction efficiency and accuracy are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video processing technology, and in particular to a short video label extraction method, device, equipment and medium. Background Art

[0002] With the rise of short video platforms, the demand for efficient and accurate video content labels is growing. In the prior art, text-based label extraction methods are usually used because text information is relatively easy to process and has low cost. However, existing text-based label extraction technologies rely on manual annotation, which has low efficiency and accuracy. Therefore, it is crucial to improve the efficiency and accuracy of short video label extraction. Summary of the invention

[0003] The present invention provides a short video tag extraction method, device, equipment and medium to improve the efficiency and accuracy of short video tag extraction.

[0004] According to one aspect of the present invention, a short video tag extraction method is provided, comprising:

[0005] Determine at least one key frame image from the short video to be processed, and vectorize each of the key frame images to obtain a key image vector;

[0006] Determine the candidate vector similarity according to the candidate image vectors corresponding to each candidate video tag stored in the vector database, and the key image vector, and select a preset first selection threshold key vector similarities from the candidate vector similarities;

[0007] Taking the candidate video tag corresponding to the key vector similarity as the reference video tag, and determining the reference video picture corresponding to the reference video tag;

[0008] The reference video picture is vectorized to obtain a reference picture vector, and a target video label of the short video to be processed is determined according to the reference picture vector and the candidate picture vector.

[0009] According to another aspect of the present invention, a short video tag extraction device is provided, comprising:

[0010] A picture vector determination module, used to determine at least one key frame picture from the short video to be processed, and vectorize each of the key frame pictures to obtain a key picture vector;

[0011] A vector similarity determination module is used to determine the candidate vector similarities according to the candidate picture vectors corresponding to each candidate video tag stored in the vector database and the key picture vector, and select a preset first selection threshold number of key vector similarities from the candidate vector similarities;

[0012] A reference video picture determination module, used to use the candidate video tag corresponding to the key vector similarity as a reference video tag, and determine a reference video picture corresponding to the reference video tag;

[0013] The target video label determination module is used to vectorize the reference video picture to obtain a reference picture vector, and determine the target video label of the short video to be processed based on the reference picture vector and the candidate picture vector.

[0014] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0015] one or more processors;

[0016] A memory for storing one or more programs;

[0017] When one or more programs are executed by one or more processors, the one or more processors can execute any short video label extraction method provided by the embodiments of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement any short video label extraction method provided by an embodiment of the present invention when executed.

[0019] The embodiment of the present invention provides a short video label extraction scheme, which determines at least one key frame image from the short video to be processed, and vectorizes each key frame image to obtain a key image vector; determines the candidate vector similarity according to the candidate image vectors corresponding to each candidate video label stored in the vector database, and the key image vector, and selects a preset first selection threshold key vector similarities from the candidate vector similarities; uses the candidate video label corresponding to the key vector similarity as a reference video label, and determines the reference video image corresponding to the reference video label; vectorizes the reference video image to obtain a reference image vector, and determines the target video label of the short video to be processed according to the reference image vector and the candidate image vector. The above scheme improves the efficiency of short video label extraction by automatically extracting and determining the target video label of the short video to be processed from the candidate video labels; at the same time, by first determining the reference video label according to the key image vector and the candidate image vector, and then determining the target video label according to the reference video label, the accuracy of the determined target video label is improved.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of a short video label extraction method provided by Embodiment 1 of the present invention;

[0023] Figure 2A This is a flow chart of a short video label extraction method provided by Embodiment 2 of the present invention;

[0024] Figure 2B is a schematic diagram of a short video label extraction method provided by Embodiment 2 of the present invention;

[0025] Figure 3 It is a structural schematic diagram of a short video label extraction device provided by Embodiment 3 of the present invention;

[0026] Figure 4 It is a structural schematic diagram of an electronic device for implementing a short video tag extraction method provided by a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Video content itself carries a lot of information, but the process of extracting this information is complex and costly, resulting in the fact that label extraction methods based on video content are not common. At present, the industry generally adopts label extraction methods based on text information, because text information is relatively easy to process and has low cost. However, existing text-based label extraction technologies rely on manual annotation, which is not only inefficient, but also difficult to keep up with the speed of content updates and cannot effectively respond to the challenges of emerging content.

[0029] Although deep learning models, such as convolutional neural networks (CNNs) and self-attention mechanisms, have been tried to automate the generation of video labels, these models require a large amount of training data, and due to the diversity and subjectivity of video labels, the model training process is complicated and difficult to meet the requirements of practical applications. In addition, existing methods fail to fully consider the differences between different types of information in video images, such as subtitles, brand logos, advertising content, and character images, which leads to a waste of computing resources and a decrease in the accuracy of label extraction.

[0030] Therefore, developing a new method that can automatically generate short video content tags to improve the efficiency and accuracy of tag extraction is of great significance to improving the service quality and content management level of short video platforms.

[0031] Embodiment 1

[0032] Figure 1 It is a flow chart of a short video label extraction method provided in Example 1 of the present invention. This embodiment can be applied to the case of extracting labels corresponding to the short video to be processed from the video label library. The method can be executed by a short video label extraction device, which can be implemented in software and / or hardware and can be configured in an electronic device that carries the short video label extraction function.

[0033] See also Figure 1 The short video label extraction method shown includes:

[0034] S110, determining at least one key frame image from the short video to be processed, and vectorizing each key frame image to obtain a key image vector.

[0035] The short video to be processed refers to a short video for which a video tag needs to be determined. The key frame image refers to an image extracted from the short video to be processed and having information for determining the video tag. Exemplarily, the number of key frame images can be one or more.

[0036] The key picture vector refers to a picture vector obtained by performing vector conversion on the key frame picture. It should be noted that one key frame picture corresponds to one key picture vector.

[0037] In an optional embodiment, at least one key frame image is determined from the short video to be processed, and each key frame image is vectorized to obtain a key image vector, including: inputting each frame of the short video to be processed into a trained key frame extraction model to obtain a key frame image; inputting the key frame image into a trained image embedding model to obtain a key image vector.

[0038] The video images to be processed refer to the frames of images that constitute the short video to be processed.

[0039] The key frame extraction model can be used to extract key frame images from the short video to be processed. The embodiment of the present invention does not impose any limitation on the network structure of the key frame extraction model, which can be set by technicians according to experience or needs.

[0040] The picture embedding model can be used to convert the key frame picture into a vector to obtain a key picture vector. The embodiment of the present invention does not impose any limitation on the picture embedding model, which can be set by a technician based on experience. For example, CLIP (Contrastive Language-Image Pre-training, multimodal pre-training model) can be used as the picture embedding model.

[0041] It can be understood that in the embodiment of the present invention, by introducing the key frame extraction model and the image embedding model, the comprehensive use of the two models is achieved, thereby improving the efficiency and accuracy of determining the key frame images and key image vectors.

[0042] In the embodiment of the present invention, a preset extraction period can be set to extract key frame images from the short video to be processed. The embodiment of the present invention does not impose any restrictions on the size of the preset extraction period, which can be set by technicians based on experience, and only needs to ensure that the preset extraction period is less than the default duration of the short video to be processed. The default duration refers to the complete playback duration of the short video to be processed without interference.

[0043] S120, determining candidate vector similarities according to the candidate picture vectors corresponding to the candidate video tags stored in the vector database, and the key picture vectors, and selecting a preset first selection threshold number of key vector similarities from the candidate vector similarities.

[0044] The vector database may be used to store candidate image vectors. The embodiment of the present invention does not impose any limitation on the vector database, which may be set by a technician according to experience or needs. Exemplarily, the vector database may be a RAG (Retrieval-Augmented Generation) vector database.

[0045] The candidate video tag refers to a pre-set tag of a short video. Exemplarily, the candidate video tag is a text tag. The candidate image vector refers to an image vector obtained by performing vector conversion on the candidate video image corresponding to the candidate video tag.

[0046] The candidate vector similarity can be used to quantify the similarity between the candidate picture vector and the key picture vector. The embodiment of the present invention does not impose any limitation on the size of the preset first selection threshold, which can be set by technicians based on experience or needs, or determined repeatedly through a large number of experiments.

[0047] The key vector similarity refers to the candidate vector similarity that can be used to determine the reference video label.

[0048] In an optional embodiment, the candidate image vector is determined based on the following method: each candidate video tag in the video tag library is input into a trained text-generated image model to obtain a candidate video image corresponding to the candidate video tag; the candidate video image is input into an image embedding model to obtain a candidate image vector corresponding to the candidate video tag.

[0049] The video tag library may be used to store candidate video tags. The embodiment of the present invention does not impose any limitation on the video tag library, and the video tag library may be set by technicians according to experience or needs.

[0050] The text generation picture model can be used to generate multiple candidate video pictures corresponding to the candidate video tags. The embodiment of the present invention does not impose any limitation on the network structure of the text generation picture model, which can be set by technicians according to experience or needs. For example, an open source model such as Stable Diffusion XL can be used as the text generation picture model.

[0051] The candidate video picture refers to the picture corresponding to the candidate video tag.

[0052] In the embodiment of the present invention, one candidate video tag corresponds to at least two candidate video images, and one candidate video image corresponds to one candidate image vector, so that one candidate video tag corresponds to at least two candidate image vectors. Multiple candidate image vectors corresponding to each candidate video tag are stored in a vector database.

[0053] It can be understood that the embodiment of the present invention generates multiple candidate video images for a candidate video tag by introducing a text-to-image model, thereby improving the content represented by a candidate video tag and further improving the comprehensiveness and richness of the candidate image vector corresponding to the candidate video tag.

[0054] Exemplarily, selecting a preset first selection threshold number of key vector similarities from the candidate vector similarities includes: sorting the candidate vector similarities to obtain a candidate sorting result; and selecting a preset first selection threshold number of candidate vector similarities from the candidate sorting result as the key vector similarities.

[0055] The candidate ranking result refers to the ranking result obtained by sorting the candidate vector similarities from large to small.

[0056] Specifically, the candidate vector similarities are sorted from large to small to obtain a candidate sorting result; and a preset first selection threshold number of candidate vector similarities are selected from the candidate sorting result as the key vector similarities.

[0057] S130: Taking the candidate video tag corresponding to the key vector similarity as a reference video tag, and determining a reference video picture corresponding to the reference video tag.

[0058] The reference video tag refers to the candidate video tag corresponding to the key vector similarity. The reference video picture refers to the picture corresponding to the reference video tag. Exemplarily, the reference video tag can correspond to multiple reference video pictures.

[0059] Exemplarily, the reference video label is input into the text generation picture model to obtain at least two reference video pictures corresponding to the reference video label.

[0060] S140, vectorizing the reference video picture to obtain a reference picture vector, and determining a target video label of the short video to be processed according to the reference picture vector and the candidate picture vector.

[0061] The reference image vector refers to the image vector obtained by vector conversion of the reference video image. The target video tag refers to the candidate video tag required for the short video to be processed, that is, the target video tag can be understood as the candidate video tag used to summarize the content of the short video to be processed.

[0062] The embodiment of the present invention provides a short video label extraction scheme, which determines at least one key frame image from the short video to be processed, and vectorizes each key frame image to obtain a key image vector; determines the candidate vector similarity according to the candidate image vectors corresponding to each candidate video label stored in the vector database, and the key image vector, and selects a preset first selection threshold key vector similarities from the candidate vector similarities; uses the candidate video label corresponding to the key vector similarity as a reference video label, and determines the reference video image corresponding to the reference video label; vectorizes the reference video image to obtain a reference image vector, and determines the target video label of the short video to be processed according to the reference image vector and the candidate image vector. The above scheme improves the efficiency of short video label extraction by automatically extracting and determining the target video label of the short video to be processed from the candidate video labels; at the same time, by first determining the reference video label according to the key image vector and the candidate image vector, and then determining the target video label according to the reference video label, the accuracy of the determined target video label is improved.

[0063] Embodiment 2

[0064] Figure 2AIt is a flow chart of a short video label extraction method provided by the second embodiment of the present invention. Based on the above embodiments, this embodiment further refines the operation of "determining the target video label of the short video to be processed based on the reference picture vector and the candidate picture vector" into "determining the reference vector similarity between the reference picture vector and the candidate picture vector; selecting a preset second selection threshold number of target vector similarities from the reference vector similarities, and using the candidate video label corresponding to the target vector similarity as the target video label" to improve the determination mechanism of the target video label. It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments.

[0065] See also Figure 2A The short video label extraction method shown includes:

[0066] S210: Determine at least one key frame image from the short video to be processed, and vectorize each key frame image to obtain a key image vector.

[0067] S220, determining candidate vector similarities according to the candidate picture vectors corresponding to the candidate video tags stored in the vector database, and the key picture vectors, and selecting a preset first selection threshold number of key vector similarities from the candidate vector similarities.

[0068] S230: Taking the candidate video tag corresponding to the key vector similarity as a reference video tag, and determining a reference video picture corresponding to the reference video tag.

[0069] S240. Vectorize the reference video picture to obtain a reference picture vector.

[0070] S250: Determine the reference vector similarity between the reference picture vector and the candidate picture vector.

[0071] The reference vector similarity may be used to quantify the similarity between the reference picture vector and the candidate picture vector.

[0072] S260 , selecting a preset second selection threshold number of target vector similarities from the reference vector similarities, and taking the candidate video tags corresponding to the target vector similarities as the target video tags.

[0073] The embodiment of the present invention does not impose any limitation on the size of the preset second selection threshold, which can be set by a technician based on experience or needs, or determined repeatedly through a large number of experiments. It should be noted that the preset first selection threshold and the preset second selection threshold in the embodiment of the present invention can be the same or different, and the embodiment of the present invention does not impose any specific limitation on this.

[0074] In an optional embodiment, the preset first selection threshold may be greater than the preset second selection threshold. Accordingly, the preset first selection threshold can realize the initial screening of candidate video tags to obtain reference video tags; and then the reference video tags are secondary screened according to the preset second selection threshold to obtain the target video tags.

[0075] In another optional embodiment, the preset first selection threshold can be equal to the preset second selection threshold. Accordingly, the target video tag determined by the preset second selection threshold can be understood as a verification of the reference video tag determined by the preset first selection threshold; if the reference video tag is the same as the target video tag, it indicates that the reference video tag is accurate; if the reference video tag and the target video tag are at least partially different, the target video tag shall prevail, indicating that the reference video tag is at least partially inaccurate.

[0076] The target vector similarity refers to the reference vector similarity that can be used to determine the target video label.

[0077] In an optional embodiment, selecting a preset second selection threshold number of target vector similarities from the reference vector similarities includes: sorting the reference vector similarities to obtain a reference sorting result; and selecting a preset second selection threshold number of reference vector similarities from the reference sorting result as the target vector similarity.

[0078] The reference sorting result refers to the sorting result obtained by sorting the reference vector similarities from large to small.

[0079] Specifically, the reference vector similarities are sorted from large to small to obtain a reference sorting result; and a preset second selection threshold number of reference vector similarities are selected from the reference sorting result as the target vector similarity.

[0080] It can be understood that by sorting the reference vector similarities and selecting the target vector similarity from the sorted results, the accuracy of the selected target vector similarity is improved.

[0081] The embodiment of the present invention provides a short video label extraction scheme, which refines the operation of determining the target video label of the short video to be processed based on the reference picture vector and the candidate picture vector into determining the reference vector similarity between the reference picture vector and the candidate picture vector; selects a preset second selection threshold number of target vector similarities from the reference vector similarities, and uses the candidate video label corresponding to the target vector similarity as the target video label, thereby improving the target video label determination mechanism. The above scheme improves the accuracy of the determined target video label by introducing the reference vector similarity and the preset second selection threshold to determine the target vector similarity.

[0082] On the basis of the above technical solution, an embodiment of the present invention further provides a short video tag extraction method based on RAG (Retrieval-Augmented Generation), including the creation and maintenance of a RAG vector database.

[0083] For example, see Figure 2B Schematic diagram of the short video tag extraction method shown in FIG. In the preparation stage, first, each candidate video tag I1, I2, and I3 in the video tag library is passed through the T2I (text to image) model to generate a batch of synthetic images (i.e., candidate video images), and a synthetic image library is created. Then, these candidate video images are sequentially converted into candidate image vectors through the image embedding model, and a RAG vector database is maintained to save these candidate image vectors.

[0084] Furthermore, in the use stage, the short video to be processed is subjected to a key frame extraction model or a key frame is extracted according to a preset extraction cycle to obtain a key frame image of the short video to be processed, such as Figure 2B F1, F2 and F3 in the preparation stage; after these key frame images are processed through the same picture embedding model as in the preparation stage to obtain key picture vectors, cosine similarity calculation is performed with the candidate picture vectors in the RAG vector database to obtain candidate vector similarity; the candidate vector similarity with the largest value preset first selected threshold is used as the key vector similarity, and the candidate video label corresponding to the key vector similarity is used as the reference video label; then the reference video label is converted into a reference video image through the same T2I model as in the preparation stage; it is converted into a reference video vector through the same picture embedding model as in the preparation stage, and cosine similarity calculation is performed with the candidate picture vectors in the RAG vector database to obtain reference vector similarity; the candidate video label corresponding to the vector with the largest value preset second selected threshold is used as the target video label of the short video to be processed. The number of reference video labels can be the same as or different from the number of target video labels.

[0085] Embodiment 3

[0086] Figure 3 This is a schematic diagram of the structure of a video segment label extraction device provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of extracting labels corresponding to the short video to be processed from the video label library. The method can be executed by a short video label extraction device, which can be implemented in software and / or hardware and can be configured in an electronic device that carries the short video label extraction function.

[0087] like Figure 3As shown, the device includes: a picture vector determination module 310, a vector similarity determination module 320, a reference video picture determination module 330 and a target video label determination module 340.

[0088] The picture vector determination module 310 is used to determine at least one key frame picture from the short video to be processed, and vectorize each of the key frame pictures to obtain a key picture vector;

[0089] A vector similarity determination module 320 is used to determine the candidate vector similarities according to the candidate picture vectors corresponding to the candidate video tags stored in the vector database and the key picture vectors, and select a preset first selection threshold number of key vector similarities from the candidate vector similarities;

[0090] A reference video picture determination module 330 is used to use the candidate video tag corresponding to the key vector similarity as a reference video tag, and determine a reference video picture corresponding to the reference video tag;

[0091] The target video label determination module 340 is used to vectorize the reference video picture to obtain a reference picture vector, and determine the target video label of the short video to be processed based on the reference picture vector and the candidate picture vector.

[0092] The embodiment of the present invention provides a short video label extraction scheme, which determines at least one key frame image from the short video to be processed, and vectorizes each key frame image to obtain a key image vector; determines the candidate vector similarity according to the candidate image vectors corresponding to each candidate video label stored in the vector database, and the key image vector, and selects a preset first selection threshold key vector similarities from the candidate vector similarities; uses the candidate video label corresponding to the key vector similarity as a reference video label, and determines the reference video image corresponding to the reference video label; vectorizes the reference video image to obtain a reference image vector, and determines the target video label of the short video to be processed according to the reference image vector and the candidate image vector. The above scheme improves the efficiency of short video label extraction by automatically extracting and determining the target video label of the short video to be processed from the candidate video labels; at the same time, by first determining the reference video label according to the key image vector and the candidate image vector, and then determining the target video label according to the reference video label, the accuracy of the determined target video label is improved.

[0093] Optionally, the target video tag determination module 340 includes:

[0094] a reference similarity determination unit, configured to determine a reference vector similarity between the reference picture vector and the candidate picture vector;

[0095] The target video label determination unit is used to select a preset second selection threshold number of target vector similarities from the reference vector similarities, and use the candidate video labels corresponding to the target vector similarities as the target video labels.

[0096] Optionally, the target video tag determination unit is specifically used to:

[0097] Sorting the reference vector similarities to obtain a reference sorting result;

[0098] A preset second selection threshold number of reference vector similarities are selected from the reference sorting results as target vector similarities.

[0099] Optionally, the picture vector determination module 310 includes:

[0100] A key frame image determination unit, used for inputting each frame of the to-be-processed video image of the to-be-processed short video into a trained key frame extraction model to obtain a key frame image;

[0101] The key picture vector determination unit is used to input the key frame picture into the trained picture embedding model to obtain the key picture vector.

[0102] Optionally, the candidate picture vector is determined based on the following means:

[0103] A candidate video image determination module is used to input each candidate video tag in the video tag library into the trained text-generated image model to obtain a candidate video image corresponding to the candidate video tag;

[0104] The candidate picture vector determination module is used to input the candidate video picture into the picture embedding model to obtain the candidate picture vector corresponding to the candidate video label.

[0105] Optionally, one candidate video tag corresponds to at least two candidate video images.

[0106] The short video label extraction device provided in the embodiment of the present invention can execute the short video label extraction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each short video label extraction method.

[0107] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of the short videos to be processed and the candidate video tags, etc., are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0108] Embodiment 4

[0109] Figure 4It is a structural diagram of an electronic device for implementing a short video tag extraction method provided by Embodiment 4 of the present invention. Electronic device 410 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0110] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0111] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0112] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 411 executes the various methods and processes described above, such as a short video tag extraction method.

[0113] In some embodiments, the short video tag extraction method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the short video tag extraction method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to perform the short video tag extraction method in any other appropriate manner (e.g., by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0116] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0119] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A short video label extraction method, characterized in that: include: Determine at least one key frame image from the short video to be processed, and vectorize each of the key frame images to obtain a key image vector; Determine the candidate vector similarity according to the candidate image vectors corresponding to each candidate video tag stored in the vector database, and the key image vector, and select a preset first selection threshold key vector similarities from the candidate vector similarities; Taking the candidate video tag corresponding to the key vector similarity as the reference video tag, and determining the reference video picture corresponding to the reference video tag; The reference video picture is vectorized to obtain a reference picture vector, and a target video label of the short video to be processed is determined according to the reference picture vector and the candidate picture vector.

2. The method according to claim 1, characterized in that The step of determining a target video tag of the short video to be processed according to the reference picture vector and the candidate picture vector includes: determining a reference vector similarity between the reference picture vector and the candidate picture vector; A preset second selection threshold number of target vector similarities are selected from the reference vector similarities, and the candidate video tags corresponding to the target vector similarities are used as target video tags.

3. The method according to claim 2, characterized in that The selecting a preset second selection threshold number of target vector similarities from the reference vector similarities comprises: Sorting the reference vector similarities to obtain a reference sorting result; A preset second selection threshold number of reference vector similarities is selected from the reference sorting results as the target vector similarity.

4. The method according to claim 1, characterized in that: The step of determining at least one key frame image from the short video to be processed and vectorizing each of the key frame images to obtain a key image vector includes: Inputting each frame of the short video to be processed into the trained key frame extraction model to obtain a key frame image; The key frame image is input into the trained image embedding model to obtain a key image vector.

5. The method according to claim 1, characterized in that: The candidate picture vector is determined based on the following method: Input each candidate video tag in the video tag library into the trained text-to-image model to obtain a candidate video image corresponding to the candidate video tag; The candidate video picture is input into the picture embedding model to obtain the candidate picture vector corresponding to the candidate video label.

6. The method according to claim 5, characterized in that One candidate video tag corresponds to at least two candidate video images.

7. A short video label extraction device, characterized in that: include: A picture vector determination module, used to determine at least one key frame picture from the short video to be processed, and vectorize each of the key frame pictures to obtain a key picture vector; A vector similarity determination module is used to determine the candidate vector similarities according to the candidate picture vectors corresponding to each candidate video tag stored in the vector database and the key picture vector, and select a preset first selection threshold number of key vector similarities from the candidate vector similarities; A reference video picture determination module, used to use the candidate video tag corresponding to the key vector similarity as a reference video tag, and determine a reference video picture corresponding to the reference video tag; The target video label determination module is used to vectorize the reference video picture to obtain a reference picture vector, and determine the target video label of the short video to be processed based on the reference picture vector and the candidate picture vector.

8. The device according to claim 7, characterized in that The target video tag determination module includes: a reference similarity determination unit, configured to determine a reference vector similarity between the reference picture vector and the candidate picture vector; The target video label determination unit is used to select a preset second selection threshold number of target vector similarities from the reference vector similarities, and use the candidate video labels corresponding to the target vector similarities as the target video labels.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a short video label extraction method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a short video label extraction method as described in any one of claims 1 to 6 is implemented.

Images