Method and device for generating video description information, and storage medium

Abstract

Embodiments of the present disclosure provide a method and device for generating video description information, and a storage medium, the method comprising: using a preset feature extraction model to extract video features from a video to be processed; on the basis of the video features, using a preset entity recognition model to perform entity recognition, determining a target entity included in the video to be processed, and obtaining local features of the target entity; from a preset knowledge graph, obtaining knowledge information of the target entity; and, on the basis of the video features, the local features of the target entity, the knowledge information of the target entity, and a preset video description model, generating description information of the video to be processed. According to the embodiments of the present disclosure, by combining the video to be processed with the knowledge graph, the information of the video to be processed can be supplemented or expanded. Thus, content-enriched video description information which is more accurate and which conforms more to reality can be generated, and the quality of the video description information can be improved.

Description

Method, device and storage medium for generating video description information

[0001] This application claims priority to Chinese patent application No. 202311038349.X filed on August 17, 2023, the entire text of which is incorporated herein by reference as a part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to a method, device, and storage medium for generating video description information. Background Art

[0003] Video captioning involves asking a computer to automatically generate a text description of a video given it's video. This task is a hot research area in deep learning and has broad application prospects. It can help machines better understand the content of videos, enabling automated video understanding, video search, and video content management.

[0004] In some cases, a video description model is often used to automatically generate video description information for a video. However, the video description model usually only relies on features extracted from the video, and the resulting video description information is relatively rough and may contain errors, resulting in low quality of the video description information.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a method, device, and storage medium for generating video description information, so as to improve the generation quality of video description information.

[0007] In a first aspect, an embodiment of the present disclosure provides a method for generating video description information, comprising:

[0008] Use a preset feature extraction model to extract video features from the video to be processed;

[0009] Using a preset entity recognition model to perform entity recognition based on the video features, determine the target entity included in the video to be processed, and obtain local features of the target entity;

[0010] Acquire knowledge information of the target entity from a preset knowledge graph;

[0011] The description information of the video to be processed is generated according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

[0012] In a second aspect, an embodiment of the present disclosure provides a device for generating video description information, including:

[0013] A feature extraction unit, configured to extract video features from the video to be processed using a preset feature extraction model;

[0014] An entity recognition unit, configured to perform entity recognition based on the video features using a preset entity recognition model to determine a target entity included in the video to be processed;

[0015] The feature extraction unit is further used to obtain local features of the target entity;

[0016] A knowledge graph unit, configured to obtain knowledge information of the target entity from a preset knowledge graph;

[0017] The video description unit is used to generate description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

[0018] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0019] The memory stores computer-executable instructions;

[0020] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method for generating video description information as described in the first aspect and various possible designs of the first aspect.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented.

[0022] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] FIG1 is a scene example diagram of a method for generating video description information provided by an embodiment of the present disclosure;

[0025] FIG2 is a flow chart of a method for generating video description information according to an embodiment of the present disclosure;

[0026] FIG3 is a schematic diagram of video feature extraction provided by an embodiment of the present disclosure;

[0027] FIG4 is a schematic diagram of the structure of a video description model provided by an embodiment of the present disclosure;

[0028] FIG5 is a structural block diagram of a device for generating video description information provided by an embodiment of the present disclosure;

[0029] FIG6 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0031] In some cases, video description models are often used to automatically generate video descriptions for videos. However, these models typically rely solely on features extracted from the video, resulting in rough and potentially erroneous video descriptions, leading to low-quality video descriptions. In particular, video description models are trained on manually annotated data, requiring annotators to understand the video content and annotate the descriptions. This annotated video description is often rough and erroneous. Training the video description model based on this rough annotated data limits the quality of the generated video descriptions and introduces erroneous information into the model.

[0032] In order to solve the above technical problems and enable the model to generate high-quality video description information, the present disclosure provides a method for generating video description information by introducing a knowledge graph. A knowledge graph is a graphical model used to describe knowledge such as entities, concepts, and events in the real world. It is usually composed of multiple nodes and edges, where each node represents an entity or concept, and the edges represent the relationship between them. By combining the video to be processed with the knowledge graph, the information of the video to be processed can be supplemented or expanded. Based on the information from the video and the information from the knowledge graph, the video description model can generate more accurate, more practical, and richer video description information, thereby improving the quality of the video description information.

[0033] The method for generating video description information disclosed in the present invention can be applied to scenarios such as servers or terminal devices. For example, as shown in Figure 1, it is applied to a server, and the required model is pre-deployed in the server. After the server obtains the video to be processed, it inputs it into the feature extraction model to extract video features from the video to be processed; then a preset entity recognition model is used to perform entity recognition based on the video features, determine the target entity included in the video to be processed, and obtain the local features of the target entity; obtain the knowledge information of the target entity from the preset knowledge graph; finally, the video features, the local features of the target entity, and the knowledge information of the target entity are input into the video description model to generate description information of the video to be processed.

[0034] The method for generating video description information disclosed in the present invention is described in detail below with reference to specific embodiments.

[0035] Referring to FIG2 , FIG2 is a flow chart of a method for generating video description information provided by an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server, and the method for generating video description information includes:

[0036] S201: Use a preset feature extraction model to extract video features from the video to be processed.

[0037] In this embodiment, after obtaining the video to be processed, video features can be extracted from the video to be processed. Specifically, a feature extraction model for extracting video features can be pre-trained, and then video features can be extracted from the video to be processed based on the feature extraction model.

[0038] Optionally, as shown in Figure 3, in this embodiment, one or more of the visual information (video image), audio information, title information and video subtitles of the video to be processed can be obtained and input into the feature extraction model. The feature extraction model can correspondingly extract one or more of the visual features, audio features, title features and subtitle features, and then the feature extraction results are aggregated to obtain the video features, that is, one or more of the extracted visual features, audio features, title features and subtitle features are fused into the video features, wherein the fusion method can directly add the feature extraction results, for example, using the concat function to add the feature extraction results.

[0039] S202: Using a preset entity recognition model to perform entity recognition based on the video features, determine a target entity included in the video to be processed, and obtain local features of the target entity.

[0040] In this embodiment, there are different entities in the video to be processed, including but not limited to regions, people, animals, plants, vehicles, buildings, etc., and the entities included in the video to be processed can be identified, which are recorded here as target entities. Specifically, an entity recognition model based on video features can be pre-trained, and the video features can be input into the entity recognition model to output the entities included therein, that is, the target entities included in the video to be processed. It should be noted that since the video features include one or more of the visual features, audio features, title features and subtitle features, the entity recognition model can simultaneously support the recognition of entities for one or more of the visual features, audio features, title features and subtitle features. Of course, different entity recognition models can also be used to recognize entities for one or more of the visual features, audio features, title features and subtitle features.

[0041] After determining the target entity included in the video to be processed, the local features of the target entity can be obtained. Optionally, one or more of the images, audio, and subtitles related to the target entity can be obtained. The feature extraction model can be used to extract one or more of the visual features, audio features, and title features, and then the feature extraction results can be aggregated to obtain the local features of the target entity.

[0042] S203: Acquire knowledge information of the target entity from a preset knowledge graph.

[0043] In this embodiment, since the information contained in the video features extracted only from the video to be processed and the local features of the target entity is limited, a preset knowledge graph can be combined to achieve information supplementation or expansion based on the preset knowledge graph that is consistent with the content of the video to be processed. Specifically, matching and linking can be performed based on the entity of the target entity in the preset knowledge graph. If the target entity is matched in the preset knowledge graph, the knowledge information of the matched target entity is retrieved from the preset knowledge graph, including but not limited to the relevant attribute information of the target entity. For example, if the target entity is a target person, the target person can be matched and linked with the preset person in the preset knowledge graph. If the target person matches a preset person in the preset knowledge graph, the knowledge information of the preset person is retrieved from the preset knowledge graph, that is, the knowledge information of the target person is obtained; for example, if the target entity is an apple, "red apples are sweeter, green apples are sourer", "there are many types of apples, such as Red Fuji apples, Guoguang apples and Huang Yuanshuai apples" can be retrieved from the preset knowledge graph. In addition, a piece of knowledge information can be represented by a triple vector, which can be represented as (head entity, relationship, tail entity).

[0044] Optionally, the preset knowledge graph in this embodiment may adopt a general knowledge graph, such as the ConceptNet knowledge graph.

[0045] Alternatively, the preset indication graph may also be a part of a certain knowledge graph (recorded as the first knowledge graph, for example, the first knowledge graph may be a general knowledge graph, etc.). Specifically, the theme or keywords of the video to be processed may be obtained, wherein the theme or keywords may be input by the user, or the theme identification or keyword identification may be performed on the video to be processed (for example, identification based on video features); and then, a part of the knowledge graph related to the theme or the keyword may be obtained from the first knowledge graph, that is, a part of the knowledge graph may be segmented from the first knowledge graph, and the part of the knowledge graph may be determined as the preset knowledge graph.

[0046] Alternatively, the preset knowledge graph can be constructed in advance. Specifically, before obtaining knowledge information about the target entity from the preset knowledge graph, an information library with the same theme as the video to be processed can be obtained. For example, if the video to be processed is a food-themed video, a food-themed information library can be obtained. For example, food-related text content can be captured from the Internet or a preset database as the food-themed information library. Furthermore, the preset knowledge graph can be constructed based on the information library and a preset relationship extraction model. The relationship extraction model is any model used to construct the knowledge graph, and the relationship extraction model is used to extract entity relationships from the knowledge library to construct the knowledge graph.

[0047] S204: Generate description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity, and a preset video description model.

[0048] In this embodiment, after obtaining the video features of the video to be processed, the local features of the target entity, and the knowledge information of the target entity, processing can be performed based on a pre-trained video description model to generate description information of the video to be processed. The input of the video description model is the video features, the local features of the target entity, and the knowledge information of the target entity, and the output is the description information of the video to be processed. By fusing the video features and the local features of the target entity from the video to be processed with the knowledge information of the target entity from the preset knowledge graph, more accurate, more practical, and richer content video description information can be generated. The video description model in this embodiment can be a Transformer model, and of course it can also be other machine learning models, which are not limited here.

[0049] The method for generating video description information of this embodiment uses a preset feature extraction model to extract video features from the video to be processed; uses a preset entity recognition model to perform entity recognition based on the video features, determines the target entity included in the video to be processed, and obtains local features of the target entity; obtains knowledge information of the target entity from a preset knowledge graph; and generates description information of the video to be processed based on the video features, local features of the target entity, knowledge information of the target entity, and the preset video description model. By combining the video to be processed with the knowledge graph, information supplementation or expansion of the video to be processed is achieved, thereby generating more accurate, more practical, and content-rich video description information, thereby improving the quality of the video description information.

[0050] Based on the above embodiment, the step S204 of generating description information of the video to be processed based on the video features, the local features of the target entity, the knowledge information of the target entity, and a preset video description model may specifically include:

[0051] S301, using different encoders of the video description model to respectively encode the video features, the local features of the target entity, and the knowledge information of the target entity to obtain corresponding encoding results;

[0052] S302. Using the decoder of the video description model to decode the encoding result based on the attention mechanism, to generate description information of the video to be processed.

[0053] In this embodiment, as shown in Figure 4, the video description model can adopt a Transformer model. The video description model may include an encoder (Encoder) and a decoder (Decoder), where different encoders can be configured to encode video features, local features of the target entity, and knowledge information of the target entity respectively to obtain corresponding encoding results. The encoding results can be expressed in the form of feature vectors.

[0054] Furthermore, the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity are input into the decoder for decoding into description information of the video to be processed. During the decoding process, the decoder can associate and / or shield some of the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the attention mechanism, especially the encoding results of the local features and the encoding results of the knowledge information of the same target entity are associated, while the encoding results of the local features and the encoding results of the knowledge information of different target entities are shielded, so that the knowledge information of a target entity can only act on the target entity without interfering with other target entities.

[0055] Furthermore, when the decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate the description information of the video to be processed, a preset attention matrix (or mask) can be obtained. In the preset attention matrix, the encoding result of the knowledge information of any target entity is only related to the encoding result of the local features of the same target entity. The local features of any target entity are also related to the video features. For example, the preset attention matrix is ​​as follows:

[0056] In the preset attention matrix, V1 and V2 represent the encoding results of video features, R1 and R2 represent the encoding results of local features of the target entity, K1, K2, and K3 represent the encoding results of the knowledge information of the target entity, where R1 and K1 correspond to the same target entity, and R2 and K2, K3 correspond to the same target entity. In the preset attention matrix, 1 indicates that there is a connection, and 0 indicates that there is no connection (needs to be masked). It can be seen that R1 and K1 are connected, K1 has no connection with the other encoding results except R1, R2 has a connection with K2 and K3, and K2 has no connection with the other encoding results except R2, K3 has no connection with the other encoding results except R2, and V1, V2, R1, and R2 are connected to each other.

[0057] After obtaining the preset attention matrix, the decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate decoding words in sequence. During the decoding process, the preset attention matrix is ​​used to focus on the related encoding results, and natural language text is finally generated according to the decoding words, that is, the description information of the video to be processed.

[0058] Based on any of the above embodiments, the step of obtaining the knowledge information of the target entity from the preset knowledge graph in S203 may specifically include:

[0059] S2031. Acquire all knowledge information related to the target entity from the preset knowledge graph;

[0060] S2032, obtaining similarities between each piece of knowledge information in all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0061] S2033: Select a preset amount of knowledge information from all knowledge information related to the target entity according to the similarity, and determine it as the knowledge information of the target entity.

[0062] In this embodiment, all knowledge information related to the target entity is obtained from a preset knowledge graph. However, this knowledge information may not be completely consistent with the target entity. Therefore, all knowledge information related to the target entity can be screened. Specifically, the similarity between each piece of knowledge information related to any target entity and the video features or the local features of the target entity can be obtained. According to the size of the similarity, a preset number of knowledge information with the greatest similarity is selected from all the knowledge information related to the target entity. The selected knowledge information is more consistent with the target entity and can ultimately be used as the knowledge information of the target entity to avoid interference from some inconsistent knowledge information.

[0063] Based on any of the above embodiments, the feature extraction model, the entity recognition model, and the video description model can be trained through the following process:

[0064] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0065] In this embodiment, end-to-end training can be adopted for the entire link, that is, the feature extraction model, the entity recognition model, and the video description model are jointly trained. It is only necessary to obtain multiple sample videos and corresponding description information, and use the multiple sample videos as input. The predicted video description information is obtained by sequentially passing through the feature extraction model, the entity recognition model, and the video description model, and the loss is obtained with the description information of the sample video. Back propagation is performed based on the loss to optimize the feature extraction model, the entity recognition model, and the video description model, and finally the trained feature extraction model, the entity recognition model, and the video description model are obtained. End-to-end training does not require separate training of each model, nor does it require preparation of training data for each model, and the training cost is low.

[0066] In another training method, the feature extraction model and entity recognition model can be trained separately in advance, or the trained feature extraction model and entity recognition model can be directly used and frozen. That is, the parameters of the feature extraction model and entity recognition model are not changed, and then the video description model can be trained separately. The specific training process is as follows:

[0067] S301, obtaining sample videos and corresponding description information;

[0068] S302, extracting video features from the sample video using the feature extraction model;

[0069] S303: Using the entity recognition model to perform entity recognition based on the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities;

[0070] S304: Acquire knowledge information of the sample entity from the preset knowledge graph;

[0071] S305 , using the video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity, and the description information corresponding to the sample video as training data to train the video description model.

[0072] In this embodiment, a plurality of sample videos and corresponding description information are obtained, and the sample videos are processed by a feature extraction model and an entity recognition model in sequence, and finally the video features of the sample videos, the local features of the sample entities (entities included in the sample videos) and the knowledge information of the sample entities are obtained, and then the video features of the sample videos, the local features of the sample entities, the knowledge information of the sample entities and the description information corresponding to the sample videos are used as training data to train the video description model, and the video features of the sample videos, the local features of the sample entities and the knowledge information of the sample entities are input into the video description model to obtain predicted video description information, and the loss obtained with the description information of the sample videos is obtained, and back propagation is performed based on the loss to optimize the video description model, and finally the trained video description model is obtained. This training method is more targeted, the trained model is more accurate, and higher-quality video description information can be obtained.

[0073] It should be noted that the above model training process can be executed on the same device as S201 or on a different device, and there is no limitation here.

[0074] Corresponding to the video description information generation method described in the above embodiment, Figure 5 is a block diagram of the video description information generation device provided in the present embodiment. For ease of illustration, only the portions relevant to the present embodiment are shown. Referring to Figure 5 , the video description information generation device 500 includes a feature extraction unit 501, an entity recognition unit 502, a knowledge graph unit 503, and a video description unit 504.

[0075] The feature extraction unit 501 is used to extract video features from the video to be processed using a preset feature extraction model;

[0076] An entity recognition unit 502 is configured to perform entity recognition based on the video features using a preset entity recognition model to determine a target entity included in the video to be processed;

[0077] The feature extraction unit 501 is further configured to obtain local features of the target entity;

[0078] The knowledge graph unit 503 is used to obtain the knowledge information of the target entity from a preset knowledge graph;

[0079] The video description unit 504 is configured to generate description information of the video to be processed based on the video features, the local features of the target entity, the knowledge information of the target entity, and a preset video description model.

[0080] In one or more embodiments of the present disclosure, when the video description unit 504 generates description information of the video to be processed based on the video features, the local features of the target entity, the knowledge information of the target entity, and a preset video description model, it is configured to:

[0081] Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity respectively to obtain corresponding encoding results;

[0082] The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

[0083] In one or more embodiments of the present disclosure, when the decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed, the video description unit 504 is configured to:

[0084] Obtaining a preset attention matrix, wherein an encoding result of knowledge information of any target entity in the preset attention matrix is ​​only associated with an encoding result of a local feature of the same target entity, and the local feature of any target entity is also associated with the video feature;

[0085] The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

[0086] In one or more embodiments of the present disclosure, when extracting video features from a video to be processed using a preset feature extraction model, the feature extraction unit 501 is configured to:

[0087] A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

[0088] In one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit 503 is further configured to:

[0089] Acquire an information database having the same subject as the video to be processed;

[0090] The preset knowledge graph is constructed based on the information database and the preset relationship extraction model.

[0091] In one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit 503 is further configured to:

[0092] Obtaining the subject or keyword of the video to be processed;

[0093] Obtain a partial knowledge graph related to the topic or the keyword from the first knowledge graph, and determine the partial knowledge graph as the preset knowledge graph.

[0094] In one or more embodiments of the present disclosure, when the knowledge graph unit 503 obtains the knowledge information of the target entity from the preset knowledge graph, it is used to:

[0095] Acquire all knowledge information related to the target entity from the preset knowledge graph;

[0096] Obtaining similarity between each piece of knowledge information in all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0097] A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

[0098] In one or more embodiments of the present disclosure, the feature extraction model, the entity recognition model, and the video description model are trained through the following process:

[0099] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0100] In one or more embodiments of the present disclosure, the video description model is trained through the following process:

[0101] Get sample videos and corresponding description information;

[0102] Extracting video features from the sample video using the feature extraction model;

[0103] Using the entity recognition model to perform entity recognition based on the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities;

[0104] Acquire knowledge information of the sample entity from the preset knowledge graph;

[0105] The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description model.

[0106] The device for generating video description information provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0107] Referring to FIG6 , a schematic diagram of the structure of an electronic device 600 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 600 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG6 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0108] As shown in Figure 6, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 shows an electronic device 600 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0110] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0111] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0112] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0113] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0114] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0117] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0118] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] In a first aspect, according to one or more embodiments of the present disclosure, a method for generating video description information is provided, comprising:

[0120] Use a preset feature extraction model to extract video features from the video to be processed;

[0121] Using a preset entity recognition model to perform entity recognition based on the video features, determine the target entity included in the video to be processed, and obtain local features of the target entity;

[0122] Acquire knowledge information of the target entity from a preset knowledge graph;

[0123] The description information of the video to be processed is generated according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

[0124] According to one or more embodiments of the present disclosure, generating description information of the video to be processed based on the video features, the local features of the target entity, the knowledge information of the target entity, and a preset video description model includes:

[0125] Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity respectively to obtain corresponding encoding results;

[0126] The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

[0127] According to one or more embodiments of the present disclosure, the decoder using the video description model decodes the encoding result based on the attention mechanism to generate description information of the video to be processed, including:

[0128] Obtaining a preset attention matrix, wherein an encoding result of knowledge information of any target entity in the preset attention matrix is ​​only associated with an encoding result of a local feature of the same target entity, and the local feature of any target entity is also associated with the video feature;

[0129] The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

[0130] According to one or more embodiments of the present disclosure, extracting video features from the video to be processed using a preset feature extraction model includes:

[0131] A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

[0132] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the process further includes:

[0133] Acquire an information database having the same subject as the video to be processed;

[0134] The preset knowledge graph is constructed based on the information database and the preset relationship extraction model.

[0135] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the process further includes:

[0136] Obtaining the subject or keyword of the video to be processed;

[0137] Obtain a partial knowledge graph related to the topic or the keyword from the first knowledge graph, and determine the partial knowledge graph as the preset knowledge graph.

[0138] According to one or more embodiments of the present disclosure, the step of obtaining the knowledge information of the target entity from a preset knowledge graph includes:

[0139] Acquire all knowledge information related to the target entity from the preset knowledge graph;

[0140] Obtaining similarity between each piece of knowledge information in all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0141] A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

[0142] According to one or more embodiments of the present disclosure, the feature extraction model, the entity recognition model, and the video description model are trained through the following process:

[0143] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0144] According to one or more embodiments of the present disclosure, the video description model is trained through the following process:

[0145] Get sample videos and corresponding description information;

[0146] Extracting video features from the sample video using the feature extraction model;

[0147] Using the entity recognition model to perform entity recognition based on the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities;

[0148] Acquire knowledge information of the sample entity from the preset knowledge graph;

[0149] The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description model.

[0150] In a second aspect, according to one or more embodiments of the present disclosure, a device for generating video description information is provided, including:

[0151] A feature extraction unit, configured to extract video features from the video to be processed using a preset feature extraction model;

[0152] An entity recognition unit, configured to perform entity recognition based on the video features using a preset entity recognition model to determine a target entity included in the video to be processed;

[0153] The feature extraction unit is further used to obtain local features of the target entity;

[0154] A knowledge graph unit, configured to obtain knowledge information of the target entity from a preset knowledge graph;

[0155] The video description unit is used to generate description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

[0156] According to one or more embodiments of the present disclosure, when the video description unit generates description information of the video to be processed based on the video features, the local features of the target entity, the knowledge information of the target entity, and a preset video description model, it is used to:

[0157] Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity respectively to obtain corresponding encoding results;

[0158] The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

[0159] According to one or more embodiments of the present disclosure, when the video description unit uses the decoder of the video description model to decode the encoding result based on the attention mechanism to generate the description information of the video to be processed, it is configured to:

[0160] Obtaining a preset attention matrix, wherein an encoding result of knowledge information of any target entity in the preset attention matrix is ​​only associated with an encoding result of a local feature of the same target entity, and the local feature of any target entity is also associated with the video feature;

[0161] The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

[0162] According to one or more embodiments of the present disclosure, when the feature extraction unit extracts video features from the video to be processed using a preset feature extraction model, it is configured to:

[0163] A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

[0164] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit is further configured to:

[0165] Acquire an information database having the same subject as the video to be processed;

[0166] The preset knowledge graph is constructed based on the information database and the preset relationship extraction model.

[0167] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit is further configured to:

[0168] Obtaining the subject or keyword of the video to be processed;

[0169] Obtain a partial knowledge graph related to the topic or the keyword from the first knowledge graph, and determine the partial knowledge graph as the preset knowledge graph.

[0170] According to one or more embodiments of the present disclosure, when the knowledge graph unit obtains the knowledge information of the target entity from the preset knowledge graph, it is used to:

[0171] Acquire all knowledge information related to the target entity from the preset knowledge graph;

[0172] Obtaining similarity between each piece of knowledge information in all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0173] A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

[0174] According to one or more embodiments of the present disclosure, the feature extraction model, the entity recognition model, and the video description model are trained through the following process:

[0175] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0176] According to one or more embodiments of the present disclosure, the video description model is trained through the following process:

[0177] Get sample videos and corresponding description information;

[0178] Extracting video features from the sample video using the feature extraction model;

[0179] Using the entity recognition model to perform entity recognition based on the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities;

[0180] Acquire knowledge information of the sample entity from the preset knowledge graph;

[0181] The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description model.

[0182] The device for generating video description information provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0183] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0184] The memory stores computer-executable instructions;

[0185] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method for generating video description information as described in the first aspect and various possible designs of the first aspect.

[0186] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented.

[0187] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented.

[0188] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0189] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0190] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for generating video description information, comprising: Use a preset feature extraction model to extract video features from the video to be processed; Using a preset entity recognition model to perform entity recognition according to the video features, determine the target entity included in the video to be processed, and obtain local features of the target entity; Acquire knowledge information of the target entity from a preset knowledge graph; The description information of the video to be processed is generated according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

2. The method according to claim 1, wherein: The generating the description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model includes: Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity, respectively, to obtain corresponding encoding results; The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

3. The method according to claim 2, wherein: The decoder using the video description model decodes the encoding result based on the attention mechanism to generate description information of the video to be processed, including: Obtaining a preset attention matrix, in which the encoding result of the knowledge information of any target entity is only related to the encoding result of the local feature of the same target entity, and the local feature of any target entity is also related to the video feature; The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

4. The method according to any one of claims 1 to 3, wherein: The method of extracting video features from the video to be processed by using a preset feature extraction model includes: A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

5. The method according to any one of claims 1 to 4, wherein: Before acquiring the knowledge information of the target entity from the preset knowledge graph, the method further includes: Acquire an information library having the same subject as the video to be processed; The preset knowledge graph is constructed based on the information base and the preset relationship extraction model.

6. The method according to any one of claims 1 to 4, wherein: Before acquiring the knowledge information of the target entity from the preset knowledge graph, the method further includes: Obtaining the subject or keyword of the video to be processed; A partial knowledge graph related to the topic or the keyword is obtained from the first knowledge graph, and the partial knowledge graph is determined as the preset knowledge graph.

7. The method according to any one of claims 1 to 6, wherein: The acquiring the knowledge information of the target entity from the preset knowledge graph includes: Acquire all knowledge information related to the target entity from the preset knowledge graph; Obtaining the similarity between each piece of knowledge information of all knowledge information related to the target entity and the video feature or the local feature of the target entity; A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

8. The method according to any one of claims 1 to 7, wherein: The feature extraction model, the entity recognition model, and the video description model are trained through the following process: A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

9. The method according to any one of claims 1 to 7, wherein: The video description model is trained through the following process: Get sample videos and corresponding description information; Extracting video features from the sample video using the feature extraction model; Using the entity recognition model to perform entity recognition according to the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities; Acquire knowledge information of the sample entity from the preset knowledge graph; The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description. The model is trained.

10. A device for generating video description information, comprising: A feature extraction unit is configured to extract video features from the video to be processed using a preset feature extraction model; An entity recognition unit is configured to use a preset entity recognition model to perform entity recognition according to the video features to determine a target entity included in the video to be processed; The feature extraction unit is also used to obtain local features of the target entity; A knowledge graph unit, configured to obtain knowledge information of the target entity from a preset knowledge graph; The video description unit is configured to generate description information of the video to be processed according to the video features, local features of the target entity, knowledge information of the target entity and a preset video description model.

11. An electronic device, comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising computer executable instructions, and when a processor executes the computer executable instructions, the method according to any one of claims 1 to 9 is implemented.

Images