Video description information generation method and device and storage medium

Abstract

Embodiments of the invention provide a video description information generation method and device, and a storage medium. The method comprises the steps of extracting video features of a to-be-processed video by adopting a preset feature extraction model; performing entity recognition according to the video features by adopting a preset entity recognition model, determining a target entity included in the to-be-processed video, and obtaining local features of the target entity; obtaining knowledge information of the target entity from a preset knowledge graph; and generating description information of the to-be-processed video according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model. The video to be processed is combined with the knowledge graph to realize information supplement or expansion of the video to be processed, so that the video description information which is more accurate, more practical and rich in content can be generated, and the quality of the video description information is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular, to a method, device and storage medium for generating video description information. Background Art

[0002] The video description task is to let the computer automatically generate a text description of a video given a video. The video description task is a hot research direction in the field of deep learning, with a very broad application prospect. It can help machines better understand the content in the video, thereby realizing automated video understanding, video search and video content management.

[0003] In the prior art, a video description model is usually used to automatically generate video description information for a video. However, the video description model usually only relies on features extracted from the video, and the video description information finally generated is relatively rough and may contain errors, resulting in low quality of the video description information. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device and storage medium for generating video description information, so as to improve the generation quality of video description information.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating video description information, including:

[0006] Use a preset feature extraction model to extract video features from the video to be processed;

[0007] Using a preset entity recognition model to perform entity recognition according to the video features, determine the target entity included in the video to be processed, and obtain local features of the target entity;

[0008] Acquire knowledge information of the target entity from a preset knowledge graph;

[0009] The description information of the video to be processed is generated according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

[0010] In a second aspect, an embodiment of the present disclosure provides a device for generating video description information, including:

[0011] A feature extraction unit, used for extracting video features from the video to be processed by using a preset feature extraction model;

[0012] An entity recognition unit, configured to use a preset entity recognition model to perform entity recognition according to the video features, and determine a target entity included in the video to be processed;

[0013] The feature extraction unit is also used to obtain local features of the target entity;

[0014] A knowledge graph unit, used to obtain knowledge information of the target entity from a preset knowledge graph;

[0015] The video description unit is used to generate description information of the video to be processed according to the video features, local features of the target entity, knowledge information of the target entity and a preset video description model.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0017] The memory stores computer-executable instructions;

[0018] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for generating video description information as described in the first aspect and various possible designs of the first aspect.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented.

[0021] The method, device and storage medium for generating video description information provided by the embodiments of the present disclosure extract video features from the video to be processed by using a preset feature extraction model; perform entity recognition based on the video features using a preset entity recognition model to determine the target entity included in the video to be processed, and obtain the local features of the target entity; obtain the knowledge information of the target entity from a preset knowledge graph; and generate description information of the video to be processed based on the video features, the local features of the target entity, the knowledge information of the target entity and the preset video description model. By combining the video to be processed with the knowledge graph, the information of the video to be processed can be supplemented or expanded, so that more accurate, more practical and rich video description information can be generated, thereby improving the quality of the video description information. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 A scene example diagram of a method for generating video description information provided by an embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a flow chart of a method for generating video description information provided by an embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of video feature extraction provided by an embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram of a video description model structure provided in one embodiment of the present disclosure;

[0027] Figure 5 A structural block diagram of a device for generating video description information provided by an embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] In the prior art, a video description model is usually used to automatically generate video description information for a video. However, the video description model usually only relies on features extracted from the video, and the resulting video description information is relatively rough and may contain errors, resulting in low quality of the video description information. In particular, the video description model is trained based on manually annotated data. The annotators need to understand the video content and annotate the video description information themselves. The annotated video description information is often relatively rough and contains errors. When the video description model is trained based on these rough annotated data, the quality of the generated video description information is limited, and the model is given erroneous information.

[0031] In order to solve the above technical problems and enable the model to generate high-quality video description information, the present disclosure provides a method for generating video description information, which introduces a knowledge graph. The knowledge graph is a graphical model used to describe knowledge such as entities, concepts, and events in the real world. It is usually composed of multiple nodes and edges, each node represents an entity or concept, and the edge represents the relationship between them. By combining the video to be processed with the knowledge graph, the information of the video to be processed can be supplemented or expanded, and then based on the information from the video and the information from the knowledge graph, the video description model can generate more accurate, more practical and rich video description information, thereby improving the quality of the video description information.

[0032] The method for generating video description information disclosed in the present invention can be applied to scenarios such as servers or terminal devices, for example Figure 1 The application shown is applied to a server, in which the required models are pre-deployed. After the server obtains the video to be processed, it is input into the feature extraction model to extract the video features of the video to be processed; then a preset entity recognition model is used to perform entity recognition based on the video features, determine the target entity included in the video to be processed, and obtain the local features of the target entity; obtain the knowledge information of the target entity from the preset knowledge graph; finally, the video features, the local features of the target entity and the knowledge information of the target entity are input into the video description model to generate the description information of the video to be processed.

[0033] The method for generating video description information disclosed in the present invention is described in detail below in conjunction with specific embodiments.

[0034] refer to Figure 2 , Figure 2 A schematic flow chart of a method for generating video description information provided in an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server, and the method for generating video description information includes:

[0035] S201, extracting video features from the video to be processed using a preset feature extraction model.

[0036] In this embodiment, after the video to be processed is obtained, video features can be extracted from the video to be processed. Specifically, a feature extraction model for extracting features from the video can be pre-trained, and then video features can be extracted from the video to be processed based on the feature extraction model.

[0037] Optional, such as Figure 3As shown, in this embodiment, one or more of the visual information (video image), audio information, title information and video subtitles of the video to be processed can be obtained and input into the feature extraction model, and one or more of the visual features, audio features, title features and subtitle features can be correspondingly extracted through the feature extraction model, and then the feature extraction results are aggregated to obtain the video features, that is, one or more of the extracted visual features, audio features, title features and subtitle features are fused into the video features, wherein the fusion method can directly add each feature extraction result, for example, using the concat function to add each feature extraction result.

[0038] S202: Use a preset entity recognition model to perform entity recognition according to the video features, determine a target entity included in the video to be processed, and obtain local features of the target entity.

[0039] In this embodiment, there are different entities in the video to be processed, including but not limited to regions, people, animals, plants, vehicles, buildings, etc. The entities included in the video to be processed can be identified, which are recorded as target entities here. Specifically, an entity recognition model based on video features can be pre-trained, and the video features can be input into the entity recognition model to output the entities included therein, that is, the target entities included in the video to be processed. It should be noted that since the video features include one or more of the visual features, audio features, title features and subtitle features, the entity recognition model can simultaneously support the recognition of one or more of the visual features, audio features, title features and subtitle features. Of course, different entity recognition models can also be used to recognize entities for one or more of the visual features, audio features, title features and subtitle features.

[0040] After determining the target entity included in the video to be processed, local features of the target entity can be obtained. Optionally, one or more of the images, audios, and subtitles related to the target entity can be obtained. One or more of the visual features, audio features, and title features can be correspondingly extracted through a feature extraction model, and the feature extraction results can be aggregated to obtain the local features of the target entity.

[0041] S203: Acquire knowledge information of the target entity from a preset knowledge graph.

[0042] In this embodiment, since the information contained in the video features extracted only from the video to be processed and the local features of the target entity is limited, the preset knowledge graph can be combined to realize the information supplement or expansion that conforms to the content of the video to be processed based on the preset knowledge graph. Specifically, matching and linking can be performed based on the entity of the target entity in the preset knowledge graph. If the target entity is matched in the preset knowledge graph, the knowledge information of the matched target entity is retrieved from the preset knowledge graph, including but not limited to the relevant attribute information of the target entity. For example, if the target entity is a certain target person, the target person can be matched and linked with the preset person in the preset knowledge graph. If the target person matches a certain preset person in the preset knowledge graph, the knowledge information of the preset person is retrieved from the preset knowledge graph, that is, the knowledge information of the target person is obtained; for another example, if the target entity is an apple, "red apples are sweeter, green apples are sourer", "there are many types of apples, such as Red Fuji apples, Guoguang apples and Huang Yuanshuai apples" can be retrieved from the preset knowledge graph. In addition, a piece of knowledge information can be represented by a triple vector, which can be represented as (head entity, relationship, tail entity).

[0043] Optionally, the preset knowledge graph in this embodiment may adopt a general knowledge graph, such as the ConceptNet knowledge graph.

[0044] Alternatively, the preset indication graph may also be a part of a certain knowledge graph (recorded as the first knowledge graph, for example, the first knowledge graph may be a general knowledge graph, etc.). Specifically, the theme or keywords of the video to be processed may be obtained, wherein the theme or keywords may be input by the user, or the theme recognition or keyword recognition may be performed on the video to be processed (for example, recognition based on video features); then, a partial knowledge graph related to the theme or the keywords may be obtained from the first knowledge graph, that is, a partial knowledge graph may be segmented from the first knowledge graph, and the partial knowledge graph may be determined as the preset knowledge graph.

[0045] Alternatively, the preset knowledge graph can also be constructed in advance. Specifically, before obtaining the knowledge information of the target entity from the preset knowledge graph, an information base with the same theme as the video to be processed can be obtained. For example, if the video to be processed is a food-themed video, a food-themed information base can be obtained. For example, food-related text content can be captured from the Internet or from a preset database as a food-themed information base; further, the preset knowledge graph can be constructed based on the information base and a preset relationship extraction model. The relationship extraction model is any model used to construct a knowledge graph, and entity relationships are extracted from the knowledge base through the relationship extraction model to construct a knowledge graph.

[0046] S204: Generate description information of the video to be processed according to the video features, local features of the target entity, knowledge information of the target entity, and a preset video description model.

[0047] In this embodiment, after obtaining the video features of the video to be processed, the local features of the target entity, and the knowledge information of the target entity, processing can be performed based on a pre-trained video description model to generate description information of the video to be processed. The input of the video description model is the video features, the local features of the target entity, and the knowledge information of the target entity, and the output is the description information of the video to be processed. By fusing the video features and the local features of the target entity from the video to be processed with the knowledge information of the target entity from the preset knowledge graph, a more accurate, more practical, and richer video description information can be generated. The video description model in this embodiment can be a Transformer model, and of course it can also be other machine learning models, which are not limited here.

[0048] The method for generating video description information of this embodiment extracts video features from the video to be processed by using a preset feature extraction model; performs entity recognition based on the video features using a preset entity recognition model to determine the target entity included in the video to be processed, and obtains the local features of the target entity; obtains the knowledge information of the target entity from a preset knowledge graph; generates description information of the video to be processed based on the video features, the local features of the target entity, the knowledge information of the target entity, and the preset video description model. By combining the video to be processed with the knowledge graph, the information of the video to be processed can be supplemented or expanded, so that more accurate, more practical, and richer video description information can be generated, thereby improving the quality of the video description information.

[0049] On the basis of the above embodiment, the step S204 of generating description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and the preset video description model may specifically include:

[0050] S301, using different encoders of the video description model to respectively encode the video features, the local features of the target entity, and the knowledge information of the target entity to obtain corresponding encoding results;

[0051] S302. Using the decoder of the video description model to decode the encoding result based on an attention mechanism, to generate description information of the video to be processed.

[0052] In this embodiment, if Figure 4As shown, the video description model can adopt a Transformer model, and the video description model can include an encoder and a decoder, wherein different encoders can be configured to encode video features, local features of the target entity, and knowledge information of the target entity, respectively, to obtain corresponding encoding results, and the encoding results can be expressed in the form of feature vectors.

[0053] Furthermore, the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity are input into the decoder to be decoded into description information of the video to be processed. During the decoding process, the decoder can associate and / or shield some of the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the attention mechanism, especially the encoding results of the local features and the encoding results of the knowledge information of the same target entity are associated, while the encoding results of the local features and the encoding results of the knowledge information of different target entities are shielded, so that the knowledge information of a target entity can only act on the target entity without interfering with other target entities.

[0054] Furthermore, when the decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate the description information of the video to be processed, a preset attention matrix (or mask) can be obtained. In the preset attention matrix, the encoding result of the knowledge information of any target entity is only related to the encoding result of the local features of the same target entity. The local features of any target entity are also related to the video features. For example, the preset attention matrix is ​​as follows:

[0055] V1 V2 R1 R2 K1 K2 K3 V1 1 1 1 1 0 0 0 V2 1 1 1 1 0 0 0 R1 1 1 1 1 1 0 0 R2 1 1 1 1 0 1 1 K1 0 0 1 0 1 0 0 K2 0 0 0 1 0 1 0 K3 0 0 0 1 0 0 1

[0056] In the preset attention matrix, V1 and V2 represent the encoding results of video features, R1 and R2 represent the encoding results of local features of the target entity, K1, K2, and K3 represent the encoding results of the knowledge information of the target entity, where R1 and K1 correspond to the same target entity, and R2 and K2, K3 correspond to the same target entity. In the preset attention matrix, 1 indicates that there is a connection, and 0 indicates that there is no connection (needs to be shielded). It can be seen that R1 and K1 are related, K1 is not related to the other encoding results except R1, R2 is related to K2, K3, and K2 is not related to the other encoding results except R2, K3 is not related to the other encoding results except R2, and V1, V2, R1, and R2 are related to each other.

[0057] After obtaining the preset attention matrix, the decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate decoding words in sequence. During the decoding process, the preset attention matrix is ​​used to focus on the related encoding results, and natural language text is finally generated according to the decoding words, that is, the description information of the video to be processed.

[0058] Based on any of the above embodiments, the step of acquiring the knowledge information of the target entity from the preset knowledge graph in S203 may specifically include:

[0059] S2031, acquiring all knowledge information related to the target entity from the preset knowledge graph;

[0060] S2032, obtaining the similarity between each piece of knowledge information of all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0061] S2033: Select a preset amount of knowledge information from all knowledge information related to the target entity according to the similarity, and determine it as the knowledge information of the target entity.

[0062] In this embodiment, all knowledge information related to the target entity is obtained from a preset knowledge graph, but this knowledge information may not be completely consistent with the target entity. Therefore, all knowledge information related to the target entity can be screened. Specifically, the similarity between each piece of knowledge information related to any target entity and the video feature or the local feature of the target entity can be obtained. According to the size of the similarity, a preset number of knowledge information with the greatest similarity is selected from all the knowledge information related to the target entity. The selected knowledge information is more consistent with the target entity and can ultimately be used as the knowledge information of the target entity to avoid interference from some inconsistent knowledge information.

[0063] Based on any of the above embodiments, the feature extraction model, the entity recognition model, and the video description model can be trained through the following process:

[0064] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0065] In this embodiment, end-to-end training can be adopted for the entire link, that is, the feature extraction model, the entity recognition model, and the video description model are jointly trained. It is only necessary to obtain multiple sample videos and corresponding description information, and use the multiple sample videos as input. The predicted video description information is obtained by sequentially passing through the feature extraction model, the entity recognition model, and the video description model, and the loss is obtained with the description information of the sample video. Back propagation is performed based on the loss to optimize the feature extraction model, the entity recognition model, and the video description model, and finally the trained feature extraction model, the entity recognition model, and the video description model are obtained. The end-to-end training does not require separate training of each model, nor does it require preparation of training data for each model, and the training cost is low.

[0066] In another training method, the feature extraction model and the entity recognition model can be trained separately in advance, or the trained feature extraction model and the entity recognition model can be directly used and frozen, that is, the parameters of the feature extraction model and the entity recognition model are no longer changed, and then the video description model is trained separately. The specific training process is as follows:

[0067] S301, obtaining sample videos and corresponding description information;

[0068] S302, extracting video features from the sample video using the feature extraction model;

[0069] S303, using the entity recognition model to perform entity recognition according to the video features of the sample video, determining the sample entities included in the sample video, and obtaining local features of the sample entities;

[0070] S304, acquiring knowledge information of the sample entity from the preset knowledge graph;

[0071] S305: Use the video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity, and the description information corresponding to the sample video as training data to train the video description model.

[0072] In this embodiment, a plurality of sample videos and corresponding description information are obtained, and the sample videos are processed by a feature extraction model and an entity recognition model in turn, and finally the video features of the sample videos, the local features of the sample entities (entities included in the sample videos) and the knowledge information of the sample entities are obtained, and then the video features of the sample videos, the local features of the sample entities, the knowledge information of the sample entities and the description information corresponding to the sample videos are used as training data to train the video description model, and the video features of the sample videos, the local features of the sample entities and the knowledge information of the sample entities are input into the video description model to obtain predicted video description information, and the loss obtained with the description information of the sample videos is obtained, and back propagation is performed based on the loss to optimize the video description model, and finally the trained video description model is obtained. This training method is more targeted, the trained model is more accurate, and higher quality video description information can be obtained.

[0073] It should be noted that the above model training process can be executed on the same device as S201 or on a different device, and there is no limitation here.

[0074] Corresponding to the method for generating video description information in the above embodiment, Figure 5 A structural block diagram of a device for generating video description information provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 The video description information generation device 500 includes: a feature extraction unit 501, an entity recognition unit 502, a knowledge graph unit 503, and a video description unit 504.

[0075] The feature extraction unit 501 is used to extract video features from the video to be processed using a preset feature extraction model;

[0076] An entity recognition unit 502 is used to perform entity recognition according to the video features using a preset entity recognition model to determine a target entity included in the video to be processed;

[0077] The feature extraction unit 501 is also used to obtain local features of the target entity;

[0078] The knowledge graph unit 503 is used to obtain the knowledge information of the target entity from a preset knowledge graph;

[0079] The video description unit 504 is used to generate description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

[0080] In one or more embodiments of the present disclosure, when the video description unit 504 generates the description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and the preset video description model, it is used to:

[0081] Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity, respectively, to obtain corresponding encoding results;

[0082] The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

[0083] In one or more embodiments of the present disclosure, when the decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate the description information of the video to be processed, the video description unit 504 is used to:

[0084] Obtaining a preset attention matrix, in which the encoding result of the knowledge information of any target entity is only related to the encoding result of the local feature of the same target entity, and the local feature of any target entity is also related to the video feature;

[0085] The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

[0086] In one or more embodiments of the present disclosure, when the feature extraction unit 501 extracts video features from the video to be processed using a preset feature extraction model, it is used to:

[0087] A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

[0088] In one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit 503 is further used to:

[0089] Acquire an information library having the same subject as the video to be processed;

[0090] The preset knowledge graph is constructed based on the information base and the preset relationship extraction model.

[0091] In one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit 503 is further used to:

[0092] Obtaining the subject or keyword of the video to be processed;

[0093] A partial knowledge graph related to the topic or the keyword is obtained from the first knowledge graph, and the partial knowledge graph is determined as the preset knowledge graph.

[0094] In one or more embodiments of the present disclosure, when the knowledge graph unit 503 acquires the knowledge information of the target entity from the preset knowledge graph, it is used to:

[0095] Acquire all knowledge information related to the target entity from the preset knowledge graph;

[0096] Obtaining the similarity between each piece of knowledge information of all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0097] A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

[0098] In one or more embodiments of the present disclosure, the feature extraction model, the entity recognition model, and the video description model are trained through the following process:

[0099] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0100] In one or more embodiments of the present disclosure, the video description model is trained through the following process:

[0101] Get sample videos and corresponding description information;

[0102] Extracting video features from the sample video using the feature extraction model;

[0103] Using the entity recognition model to perform entity recognition according to the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities;

[0104] Acquire knowledge information of the sample entity from the preset knowledge graph;

[0105] The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description model.

[0106] The device for generating video description information provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0107] refer to Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure, and the electronic device 600 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0110] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0111] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0113] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0114] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0117] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0118] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] In a first aspect, according to one or more embodiments of the present disclosure, a method for generating video description information is provided, comprising:

[0120] Use a preset feature extraction model to extract video features from the video to be processed;

[0121] Using a preset entity recognition model to perform entity recognition according to the video features, determine the target entity included in the video to be processed, and obtain local features of the target entity;

[0122] Acquire knowledge information of the target entity from a preset knowledge graph;

[0123] The description information of the video to be processed is generated according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

[0124] According to one or more embodiments of the present disclosure, generating description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model includes:

[0125] Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity, respectively, to obtain corresponding encoding results;

[0126] The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

[0127] According to one or more embodiments of the present disclosure, the decoder using the video description model decodes the encoding result based on an attention mechanism to generate description information of the video to be processed, including:

[0128] Obtaining a preset attention matrix, in which the encoding result of the knowledge information of any target entity is only related to the encoding result of the local feature of the same target entity, and the local feature of any target entity is also related to the video feature;

[0129] The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

[0130] According to one or more embodiments of the present disclosure, extracting video features from the video to be processed by using a preset feature extraction model includes:

[0131] A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

[0132] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the process further includes:

[0133] Acquire an information library having the same subject as the video to be processed;

[0134] The preset knowledge graph is constructed based on the information base and the preset relationship extraction model.

[0135] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the process further includes:

[0136] Obtaining the subject or keyword of the video to be processed;

[0137] A partial knowledge graph related to the topic or the keyword is obtained from the first knowledge graph, and the partial knowledge graph is determined as the preset knowledge graph.

[0138] According to one or more embodiments of the present disclosure, the step of acquiring knowledge information of the target entity from a preset knowledge graph includes:

[0139] Acquire all knowledge information related to the target entity from the preset knowledge graph;

[0140] Obtaining the similarity between each piece of knowledge information of all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0141] A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

[0142] According to one or more embodiments of the present disclosure, the feature extraction model, the entity recognition model, and the video description model are trained through the following process:

[0143] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0144] According to one or more embodiments of the present disclosure, the video description model is trained through the following process:

[0145] Get sample videos and corresponding description information;

[0146] Extracting video features from the sample video using the feature extraction model;

[0147] Using the entity recognition model to perform entity recognition according to the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities;

[0148] Acquire knowledge information of the sample entity from the preset knowledge graph;

[0149] The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description model.

[0150] In a second aspect, according to one or more embodiments of the present disclosure, a device for generating video description information is provided, including:

[0151] A feature extraction unit, used for extracting video features from the video to be processed by using a preset feature extraction model;

[0152] An entity recognition unit, configured to use a preset entity recognition model to perform entity recognition according to the video features, and determine a target entity included in the video to be processed;

[0153] The feature extraction unit is also used to obtain local features of the target entity;

[0154] A knowledge graph unit, used to obtain knowledge information of the target entity from a preset knowledge graph;

[0155] The video description unit is used to generate description information of the video to be processed according to the video features, local features of the target entity, knowledge information of the target entity and a preset video description model.

[0156] According to one or more embodiments of the present disclosure, when the video description unit generates description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and the preset video description model, it is used to:

[0157] Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity, respectively, to obtain corresponding encoding results;

[0158] The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

[0159] According to one or more embodiments of the present disclosure, when the video description unit uses the decoder of the video description model to decode the encoding result based on the attention mechanism to generate the description information of the video to be processed, it is used to:

[0160] Obtaining a preset attention matrix, in which the encoding result of the knowledge information of any target entity is only related to the encoding result of the local feature of the same target entity, and the local feature of any target entity is also related to the video feature;

[0161] The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

[0162] According to one or more embodiments of the present disclosure, when the feature extraction unit extracts video features from the video to be processed using a preset feature extraction model, it is used to:

[0163] A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

[0164] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit is further used to:

[0165] Acquire an information library having the same subject as the video to be processed;

[0166] The preset knowledge graph is constructed based on the information base and the preset relationship extraction model.

[0167] According to one or more embodiments of the present disclosure, before acquiring the knowledge information of the target entity from the preset knowledge graph, the knowledge graph unit is further used to:

[0168] Obtaining the subject or keyword of the video to be processed;

[0169] A partial knowledge graph related to the topic or the keyword is obtained from the first knowledge graph, and the partial knowledge graph is determined as the preset knowledge graph.

[0170] According to one or more embodiments of the present disclosure, when the knowledge graph unit acquires the knowledge information of the target entity from the preset knowledge graph, it is used to:

[0171] Acquire all knowledge information related to the target entity from the preset knowledge graph;

[0172] Obtaining the similarity between each piece of knowledge information of all knowledge information related to the target entity and the video feature or the local feature of the target entity;

[0173] A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

[0174] According to one or more embodiments of the present disclosure, the feature extraction model, the entity recognition model, and the video description model are trained through the following process:

[0175] A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

[0176] According to one or more embodiments of the present disclosure, the video description model is trained through the following process:

[0177] Get sample videos and corresponding description information;

[0178] Extracting video features from the sample video using the feature extraction model;

[0179] Using the entity recognition model to perform entity recognition according to the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities;

[0180] Acquire knowledge information of the sample entity from the preset knowledge graph;

[0181] The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description model.

[0182] The device for generating video description information provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0183] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0184] The memory stores computer-executable instructions;

[0185] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for generating video description information as described in the first aspect and various possible designs of the first aspect.

[0186] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented.

[0187] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the method for generating video description information as described in the first aspect and various possible designs of the first aspect is implemented.

[0188] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0189] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0190] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating video description information, characterized in that: include: Use a preset feature extraction model to extract video features from the video to be processed; Using a preset entity recognition model to perform entity recognition according to the video features, determine the target entity included in the video to be processed, and obtain local features of the target entity; Acquire knowledge information of the target entity from a preset knowledge graph; The description information of the video to be processed is generated according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model.

2. The method according to claim 1, characterized in that The generating the description information of the video to be processed according to the video features, the local features of the target entity, the knowledge information of the target entity and a preset video description model includes: Using different encoders of the video description model to encode the video features, the local features of the target entity, and the knowledge information of the target entity, respectively, to obtain corresponding encoding results; The decoder of the video description model is used to decode the encoding result based on the attention mechanism to generate description information of the video to be processed.

3. The method according to claim 2, characterized in that The decoder using the video description model decodes the encoding result based on the attention mechanism to generate description information of the video to be processed, including: Obtaining a preset attention matrix, in which the encoding result of the knowledge information of any target entity is only related to the encoding result of the local feature of the same target entity, and the local feature of any target entity is also related to the video feature; The decoder of the video description model is used to decode the encoding results of the video features, the encoding results of the local features of the target entity, and the encoding results of the knowledge information of the target entity based on the preset attention matrix to generate description information of the video to be processed.

4. The method according to any one of claims 1 to 3, characterized in that: The method of extracting video features from the video to be processed by using a preset feature extraction model includes: A preset feature extraction model is used to extract features of one or more of the visual information, audio information, title information and video subtitles of the video to be processed, and the feature extraction results are aggregated to obtain the video features.

5. The method according to any one of claims 1 to 3, characterized in that: Before acquiring the knowledge information of the target entity from the preset knowledge graph, the method further includes: Acquire an information library having the same subject as the video to be processed; The preset knowledge graph is constructed based on the information base and the preset relationship extraction model.

6. The method according to any one of claims 1 to 3, characterized in that: Before acquiring the knowledge information of the target entity from the preset knowledge graph, the method further includes: Obtaining the subject or keyword of the video to be processed; A partial knowledge graph related to the topic or the keyword is obtained from the first knowledge graph, and the partial knowledge graph is determined as the preset knowledge graph.

7. The method according to any one of claims 1 to 3, characterized in that: The acquiring the knowledge information of the target entity from the preset knowledge graph includes: Acquire all knowledge information related to the target entity from the preset knowledge graph; Obtaining the similarity between each piece of knowledge information of all knowledge information related to the target entity and the video feature or the local feature of the target entity; A preset amount of knowledge information is selected from all knowledge information related to the target entity according to the similarity, and is determined as the knowledge information of the target entity.

8. The method according to any one of claims 1 to 3, characterized in that: The feature extraction model, the entity recognition model, and the video description model are trained through the following process: A plurality of sample videos and corresponding description information are obtained, and the plurality of sample videos and corresponding description information are used as training data to jointly train the feature extraction model, the entity recognition model, and the video description model.

9. The method according to any one of claims 1 to 3, characterized in that: The video description model is trained through the following process: Get sample videos and corresponding description information; Extracting video features from the sample video using the feature extraction model; Using the entity recognition model to perform entity recognition according to the video features of the sample video, determine the sample entities included in the sample video, and obtain local features of the sample entities; Acquire knowledge information of the sample entity from the preset knowledge graph; The video features of the sample video, the local features of the sample entity, the knowledge information of the sample entity and the description information corresponding to the sample video are used as training data to train the video description model.

10. A device for generating video description information, characterized in that: include: A feature extraction unit, used for extracting video features from the video to be processed by using a preset feature extraction model; An entity recognition unit, configured to use a preset entity recognition model to perform entity recognition according to the video features, and determine a target entity included in the video to be processed; The feature extraction unit is also used to obtain local features of the target entity; A knowledge graph unit, used to obtain knowledge information of the target entity from a preset knowledge graph; The video description unit is used to generate description information of the video to be processed according to the video features, local features of the target entity, knowledge information of the target entity and a preset video description model.

11. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

Images