Core entity recognition model generation method, apparatus, and computer program product

Abstract

This disclosure provides a method, apparatus, electronic device, storage medium, and program product for generating a core entity recognition model, relating to the field of artificial intelligence technology. The specific implementation scheme includes: acquiring a training sample set; determining the entity feature sequences of entities in text data; determining the video frame feature sequences of video frames in video data; fusing the entity feature sequences and video frame feature sequences to obtain a fused feature sequence; and employing a machine learning method, using the fused feature sequence as input and the labels in the training samples corresponding to the fused feature sequence as the expected output, to obtain the trained core entity recognition model. This disclosure, by combining multimodal data from video, improves the recognition accuracy of the core entity recognition model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of deep learning and video understanding technology, and in particular to methods and devices for generating core entity recognition models, as well as methods, devices, electronic devices, storage media, and computer program products for recognizing core entity labels, which can be used in video understanding and recommendation scenarios. Background Technology

[0002] With the rapid development of software and hardware technologies, video has become one of the main ways to transmit information. In video distribution and recommendation scenarios, a deep understanding of the video is required, and tagging is the most important and fundamental part of video understanding. Existing core entity tagging methods often extract entities from text data, calculate the importance of the extracted entities through a tag coreity determination module, and select core entity tags based on the importance ranking. Existing core entity tagging methods are feasible for pure text scenarios, such as news text. However, in the video domain, video data includes multimodal data such as text and video frames. Core entity tags calculated solely based on text data are often biased due to the lack of supervision from video frame data and cannot serve as core entity tags for the entire video. Summary of the Invention

[0003] This disclosure provides a method and apparatus for generating a core entity recognition model, as well as a method, apparatus, electronic device, storage medium, and computer program product for recognizing core entity tags.

[0004] According to the first aspect, a method for generating a core entity recognition model is provided, comprising: acquiring a training sample set, wherein the training samples in the training sample set include corresponding text data and video data, and labels representing whether entities in the text data are core entities; determining entity feature sequences of entities in the text data; determining video frame feature sequences of video frames in the video data; fusing the entity feature sequences corresponding to the text data and the video frame feature sequences corresponding to the video data in the same training sample to obtain a fused feature sequence; and employing a machine learning method, taking the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the expected output, to obtain a trained core entity recognition model.

[0005] According to the second aspect, a method for identifying core entity labels is provided, including: determining text data and video data in the acquired target video; determining the entity feature sequence of entities in the text data; determining the video frame feature sequence of video frames in the video data; fusing the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence; and determining the core entity label corresponding to the target video based on the fused feature sequence through a core entity recognition model.

[0006] According to a third aspect, a core entity recognition model generation apparatus is provided, comprising: an acquisition unit configured to acquire a training sample set, wherein the training samples in the training sample set include corresponding text data and video data, and labels characterizing whether entities in the text data are core entities; a first determining unit configured to determine entity feature sequences of entities in the text data; a second determining unit configured to determine video frame feature sequences of video frames in the video data; a first fusion unit configured to fuse entity feature sequences corresponding to text data and video frame feature sequences corresponding to video data in the same training sample to obtain a fused feature sequence; and a training unit configured to employ a machine learning method, taking the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the desired output, to obtain a trained core entity recognition model.

[0007] According to the fourth aspect, a method for identifying core entity labels is provided, comprising: a third determining unit configured to determine text data and video data in an acquired target video; a fourth determining unit configured to determine entity feature sequences of entities in the text data; a fifth determining unit configured to determine video frame feature sequences of video frames in the video data; a second fusion unit configured to fuse the entity feature sequences and the video frame feature sequences to obtain a fused feature sequence; and a sixth determining unit configured to determine the core entity labels corresponding to the target video based on the fused feature sequence using a core entity recognition model.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first or second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first or second aspect.

[0010] According to the seventh aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0011] According to the technology disclosed herein, a method for generating a core entity recognition model is provided. During the training process, entity feature sequences corresponding to text data and video frame feature sequences corresponding to video data in the same training sample are fused. The model is trained based on the obtained fused feature sequences and the labels corresponding to the fused feature sequences, resulting in a core entity recognition model that includes an entity recognition module, a video encoding module, a multimodal data fusion module, and a core entity recognition module. This allows the core entity recognition model to fully consider the multimodal data (text data and video data) in the video, thereby improving the recognition accuracy of the core entity recognition model. Furthermore, the recognition of core entity labels in the video using the core entity recognition model improves the accuracy of the determined core entity labels.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure;

[0015] Figure 2 This is a flowchart of an embodiment of the method for generating the core entity recognition model according to this disclosure;

[0016] Figure 3 This is a schematic diagram of the entity recognition process of the entity recognition module according to this embodiment;

[0017] Figure 4 This is a schematic diagram of the video encoding process of the video encoding module according to this embodiment;

[0018] Figure 5 This is a schematic diagram of the structure of the multimodal data fusion module according to this embodiment;

[0019] Figure 6 This is a schematic diagram illustrating an application scenario of the core entity recognition model generation method according to this embodiment;

[0020] Figure 7 This is a flowchart of yet another embodiment of the method for generating the core entity recognition model according to this disclosure;

[0021] Figure 8 This is a flowchart of an embodiment of the core entity tag identification method according to this disclosure;

[0022] Figure 9This is a flowchart of yet another embodiment of the core entity tag identification method according to this disclosure;

[0023] Figure 10 This is a structural diagram of an embodiment of the apparatus for generating a core entity recognition model according to the present disclosure;

[0024] Figure 11 This is a structural diagram of one embodiment of the identification device for the core entity tag according to this disclosure;

[0025] Figure 12 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Figure 1 An exemplary architecture 100 is shown, which can be used to generate a core entity recognition model and to identify core entity labels.

[0029] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0030] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing with the server. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices supporting network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to video capture devices, smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0031] Server 105 can be a server providing various services, such as a backend processing server that trains a core entity recognition model using machine learning methods based on the training sample set provided by terminal devices 101, 102, and 103. Another example is a backend processing server that performs core entity label recognition on target videos provided by terminal devices 101, 102, and 103 using the trained core entity recognition model. As an example, server 105 can also be a cloud server.

[0032] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0033] It should also be noted that the core entity recognition model generation method and the core entity tag recognition method provided in the embodiments of this disclosure can be executed by a server, by a terminal device, or by a combination of both. Correspondingly, the various parts (e.g., units) included in the core entity recognition model generation device and the core entity tag recognition method can all be located in the server, all in the terminal device, or separately in the server and the terminal device.

[0034] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. When the electronic devices on which the core entity recognition model generation method and the core entity tag recognition method run do not require data transmission with other electronic devices, the system architecture may only include the electronic devices (e.g., servers or terminal devices) on which the core entity recognition model generation method and the core entity tag recognition method run.

[0035] Please refer to Figure 2 , Figure 2 A flowchart of a method for generating a core entity recognition model provided in this disclosure embodiment, wherein process 200 includes the following steps:

[0036] Step 201: Obtain the training sample set.

[0037] In this embodiment, the execution entity of the core entity recognition model generation method (e.g., Figure 1 The terminal device or server in the dataset can obtain the training dataset remotely or locally via a wired or wireless network connection. The training samples in the training sample set include corresponding text data and video data, as well as labels representing whether entities in the text data are core entities.

[0038] Text and video data within the same training sample correspond to each other, with the content represented by the text data matching the content represented by the video data. For example, the corresponding text and video data are the text and video data included in the same video; specifically, the video data consists of video frames from the video, and the text data is the text introducing or explaining the video data.

[0039] Text data can be obtained through text recognition of subtitles in video frames, as described above, or through speech recognition of audio in the video. Text data typically includes multiple entities; however, not every entity can express the main content of the video—that is, the core entity. Labels can be used to identify whether an entity in the text data is a core entity.

[0040] In this embodiment, distinctive characters, numbers, or other symbols can be used to label each entity in the text data to distinguish between core and non-core entities. For example, labels include 0 and 1. Label 0 represents a non-core entity, and label 1 represents a core entity.

[0041] Step 202: Determine the entity feature sequence of the entities in the text data.

[0042] In this embodiment, the aforementioned execution entity can determine the entity feature sequence of entities in the text data.

[0043] As an example, the aforementioned executive approach can use an entity recognition module to determine the sequence of entity features in text data. The entity recognition module is used to extract all entity words from the text data and determine the entity features of each entity. Entity words can serve as potential video tags.

[0044] The entity recognition module features named entity recognition (NER). It can identify entities in text data in the following ways:

[0045] Rule-based approaches rely on manually defined rules, typically designed based on patterns in syntax, grammar, and vocabulary, as well as domain-specific knowledge. When the dictionary size is limited, rule-based methods can achieve excellent results, usually characterized by high precision and low recall.

[0046] Unsupervised learning approach: Use semantic similarity to cluster text data, extract named entities from the clustered groups, and infer entity categories through statistical data.

[0047] Feature-based supervised learning can be represented as a multi-classification task or a sequence labeling task, enabling the entity recognition module to learn named entity recognition capabilities from the training data.

[0048] In each approach, after identifying all entities in the text data, a text encoder can be used to encode all entities to obtain the entity features corresponding to each entity, thereby obtaining an entity feature sequence.

[0049] Specifically, the aforementioned execution entity can first obtain a training sample set for training the initial entity recognition module. This training sample set includes text data and labels representing entities within the text data. Then, using machine learning methods, the initial entity recognition module is iteratively trained using the text data as input and the labels corresponding to the input text data as the expected output. Upon reaching a preset termination condition, the trained entity recognition module is obtained. The preset termination condition could be, for example, training time exceeding a preset time threshold, the number of training iterations exceeding a preset number of iterations threshold, or the training loss converging.

[0050] The entity recognition module can use any neural network model with named entity recognition capabilities. For example, the entity recognition module can be a BiLSTM (Bi-directional Long Short-Term Memory)-CRF (conditional random field) model, an IDCNN (Iterated Dilated Convolutional Neural Networks)-CRF model, or an ERNIE (Enhanced Representation through Knowledge Integration) model.

[0051] In some optional implementations of this embodiment, the execution entity can perform step 202 as follows: First, perform entity recognition on the text data to determine the entity sequence including the entities in the text data; then, for each entity in the entity sequence, use the entity representation vector corresponding to the entity output during the entity recognition process as the entity feature of the entity to obtain the entity feature sequence corresponding to the entity sequence.

[0052] Continue to refer to Figure 3 The diagram 300 illustrates the entity recognition process of the entity recognition module. The entity recognition module employs the ERNIE model. First, the text represented by the text data is segmented by character. Then, the segmented text is input into the entity recognition module. The ERNIE entity recognition module encodes the input text using a multi-head attention mechanism to obtain its deep semantics, and performs sequence labeling on the text at the output layer to extract entity words from the text data, thus obtaining the entity sequence.

[0053] For each entity in the entity sequence, take the token representation vector of each character at the corresponding position in the last layer of the ERNIE network, or the token representation vectors of its first and last characters at the corresponding positions in the last layer of the ERNIE network, and concatenate them to obtain the entity feature. As an example, let the entity feature sequence be T = [t1, t2, ..., t...]. m ], where t i This represents the entity feature of the i-th entity.

[0054] The ERNIE model, building upon the BERT (Bidirectional Encoder Representation from Transformers) model, employs a different masking strategy to enhance the model through knowledge graphs. In BERT, some words or characters are randomly masked, and language representation is achieved through context prediction. After obtaining a pre-trained model, it is fine-tuned to adapt to various NLP (Natural Language Processing) tasks. Because BERT's masking strategy is character-based, it is not conducive to learning knowledge information, especially for Chinese. BERT only masks individual characters during training, thus learning primarily the relationships between characters. In contrast, the ERNIE model masks consecutive characters, allowing it to learn not only relationships between individual characters but also relationships between masked words—the so-called knowledge information.

[0055] In this implementation, the entity representation vector used by the entity recognition module during the entity recognition process is adopted as the entity feature, enabling the entity feature to fully represent the entity and improving its expressiveness. In particular, when the entity recognition module uses the ERNIE model, the resulting entity feature sequence better represents the information between entities, increasing the information richness of the entity features.

[0056] Step 203: Determine the video frame feature sequence of the video frames in the video data.

[0057] In this embodiment, the aforementioned execution entity can determine the video frame feature sequence of video frames in the video data.

[0058] As an example, the aforementioned execution entity can determine the video frame feature sequence of video frames in video data through a video encoding module. The video encoding module can be an encoder that encodes video frames to generate video frame features. As an example, the video encoder employs the ViT (Vision Transformer) model.

[0059] Continue to refer to Figure 4The diagram 400 illustrates the video encoding process of the video encoding module. Specifically, the aforementioned execution entity can input all video frames from the video data into ViT. ViT divides the input image frames into multiple image patches, projects each image patch into a fixed-length image patch vector, and inputs the image patch vector into the Transformer module in the ViT model to encode the image patches of the video frames. The ViT model is essentially an image classification model; therefore, a learnable image classification token is added to the model's input sequence, and the output corresponding to this token is the category prediction for the video frame.

[0060] During the training of the ViT model, the aforementioned execution entity first obtains a training sample set for training the ViT model, which includes images and classification labels representing the images. Then, using machine learning methods, the image and classification labels are used as input to the initial ViT model to obtain the image representation vector of the image and the learned classification label. Next, the classification layer uses the learned classification label as input and the classification label corresponding to the input image as the desired output to train the ViT model.

[0061] In some alternative implementations of this example, the execution entity can perform step 203 as follows: First, determine the video frame sequence from the video data based on a preset frame rate; then, extract the video frame features of the video frames in the video sequence to obtain the video frame feature sequence corresponding to the video frame sequence.

[0062] As an example, firstly, the aforementioned execution entity segments the video data into frames at a certain frame rate, then the video can be converted into a video frame sequence Img = [img1, img2, ..., img...]. n ], where img i Let v represent the i-th video frame in the video frame sequence. Then, the trained ViT model is used as the video encoding module to extract the video frame features, denoted as v. i , where i represents the feature representation of the i-th video frame:

[0063] v i =f(img i )

[0064] Where f represents the ViT model.

[0065] In this implementation, a video frame sequence that can characterize the video is determined from the video data based on a preset frame rate. This reduces the amount of video frame data processed while ensuring the expressiveness of the video frame feature sequence.

[0066] Step 204: Fuse the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample to obtain the fused feature sequence.

[0067] In this embodiment, the aforementioned execution entity can fuse the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample to obtain a fused feature sequence.

[0068] As an example, the aforementioned execution entity can use a multimodal data fusion module to fuse entity feature sequences corresponding to text data and video frame feature sequences corresponding to video data from the same training sample to obtain a fused feature sequence. Specifically, the aforementioned execution entity can use a multimodal data fusion module to fuse video frame features from the video frame feature sequence to entity features from the entity feature sequence to obtain a fused feature sequence, denoted as H = [h1, h2, ..., h...]. n ], where h i This represents the feature vector after fusing the features of the i-th entity with the features of the video frame.

[0069] In some optional implementations of this embodiment, the execution entity can perform step 204 as follows: for the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample, a multi-head attention mechanism is used to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence.

[0070] As an example, for the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample, the entity feature sequence and the video frame feature sequence are input into the multimodal data fusion module with a multi-head attention mechanism to obtain the fused feature sequence.

[0071] Continue to refer to Figure 5 The diagram shows a schematic of the multimodal data fusion module 500. The multimodal data fusion module includes a multi-head attention module 501, a first fusion and normalization module 502, a feedback module 503, and a second fusion and normalization module 504. First, two video frame feature sequences K and V (K = V), and an entity feature sequence Q are input to the multi-head attention module 501. Then, the first fusion and normalization module 502 fuses and normalizes the output of the multi-head attention module 501 and the entity feature sequence Q. Next, the feedback module 503 feeds back the output of the first fusion and normalization module 502 to the second fusion and normalization module 504. The second fusion and normalization module 504 fuses the output of the feedback module 503 and the output of the first fusion and normalization module 502 to obtain a fused feature sequence.

[0072] Specifically, the aforementioned execution entity performs cross-modal attention calculation based on Q, K, and V, according to the following formula:

[0073]

[0074] Where, d k This represents the dimension of the video frame features in the video frame feature sequence.

[0075] This implementation method uses a multi-head attention mechanism to fuse entity feature sequences and video frame feature sequences, thereby improving the accuracy of the resulting fused feature sequence.

[0076] In some optional implementations of this embodiment, the aforementioned execution entity may further perform the feature fusion process as follows: employing a multi-layer attention network to perform multi-modal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence. Each layer of the multi-layer attention network employs a multi-head attention mechanism.

[0077] As an example, each layer of the above multi-layer attention network employs, for example, the attention network... Figure 5 The network structure shown employs a stacked attention network approach, where the input of one attention network is the output of the previous attention network, and the output of that attention network becomes the input of the next attention network. Specifically, the input of the first attention network consists of two video frame feature sequences K and V, and an entity feature sequence Q.

[0078] In this implementation, a multi-layer attention network is used to fuse entity feature sequences and video frame feature sequences at a deeper level, which improves the fusion degree of features from the two modalities and helps to improve the accuracy of the subsequent core entity recognition module.

[0079] In some implementations, for entity feature sequences corresponding to text data and video frame feature sequences corresponding to video data in the same training sample, the aforementioned execution entity can determine the correlation between entity feature sequences and video frame feature sequences, and then determine the degree of fusion between entity feature sequences and video frame feature sequences based on the correlation.

[0080] As an example, the aforementioned execution entity can map entity features from the entity feature sequence and video frame features from the video frame feature sequence to the same feature space, and determine the correlation based on the distance between the two types of features (e.g., Euclidean distance); furthermore, it can determine the number of layers in the attention network based on the correlation. Specifically, the correlation is positively correlated with the number of layers.

[0081] Step 205: Using machine learning methods, the core entity recognition model is obtained by taking the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the expected output.

[0082] In this embodiment, the aforementioned execution entity can employ machine learning methods, taking the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the desired output, to obtain the trained core entity recognition model.

[0083] As an example, for each training sample in the training sample set corresponding to the fused feature sequence, the aforementioned execution entity can input the fused feature sequence into the initial core entity recognition model to obtain the actual output characterizing whether the entity in the entity sequence is a core entity. Then, the binary cross-entropy loss between the actual output and the expected output is calculated using the following loss function:

[0084]

[0085] Among them, y i Indicates the expected output. This indicates the actual output.

[0086] Then, the gradient information corresponding to the initial core entity recognition model is determined based on the loss, and the parameters are updated using stochastic gradient descent based on the gradient information until the preset termination condition is reached, thus obtaining the trained core entity recognition model.

[0087] The core entity recognition model can be a fully connected layer. The fully connected layer fuses the feature sequence H = [h1, h2, ..., h...] according to the following formula. n h in ] i Perform mapping:

[0088]

[0089] Where w and b are the parameters of the fully connected layer.

[0090] In some optional implementations of this embodiment, the execution entity can use the model obtained in step 205 as the core entity recognition module, and then generate a core entity recognition model based on the entity recognition module, video encoding module, multimodal data fusion module and core entity recognition module.

[0091] As an example, the aforementioned execution entity can combine an entity recognition module, a video encoding module, a multimodal data fusion module, and a core entity recognition module to generate a core entity recognition model.

[0092] See also Figure 6 , Figure 6This is a schematic diagram 600 illustrating an application scenario of the core entity recognition model generation method according to this embodiment. Figure 6 In the application scenario, the server used to train the core entity recognition model first obtains a training sample set 601. The training samples in the training sample set include corresponding text data and video data, as well as labels representing whether entities in the text data are core entities. Then, the entity recognition module 602 determines the entity feature sequence of the entities in the text data; the video encoding module 603 determines the video frame feature sequence of the video frames in the video data; the multimodal data fusion module 604 fuses the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample to obtain a fused feature sequence; using a machine learning method, with the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the expected output, the trained core entity recognition module 605 is obtained; based on the entity recognition module 602, video encoding module 603, multimodal data fusion module 604, and core entity recognition module 605, the core entity recognition model is generated.

[0093] This embodiment provides a method for generating a core entity recognition model. During training, entity feature sequences corresponding to text data and video frame feature sequences corresponding to video data in the same training sample are fused. The model is trained based on the obtained fused feature sequences and the labels corresponding to the fused feature sequences, resulting in a core entity recognition model that includes an entity recognition module, a video encoding module, a multimodal data fusion module, and a core entity recognition module. This allows the core entity recognition model to fully consider the multimodal data (text data and video data) in the video, improving the recognition accuracy of the core entity recognition model. Furthermore, the core entity recognition model is used to identify the core entity labels in the video, improving the accuracy of the determined core entity labels.

[0094] Continue to refer to Figure 7 The illustration shows a schematic flow 700 of another embodiment of the method for generating a core entity recognition model according to the present disclosure, including the following steps:

[0095] Step 701: Obtain the training sample set.

[0096] The training samples in the training sample set include corresponding text data and video data, as well as labels that characterize whether the entities in the text data are core entities.

[0097] Step 702: Perform entity recognition on the text data to determine the entity sequence that includes the entities in the text data.

[0098] Step 703: For each entity in the entity sequence, the entity representation vector corresponding to that entity output during the entity recognition process is taken as the entity feature of that entity, and the entity feature sequence corresponding to the entity sequence is obtained.

[0099] Step 704: Determine the video frame sequence from the video data based on the preset frame rate.

[0100] Step 705: Extract video frame features from the video frames in the video sequence to obtain the video frame feature sequence corresponding to the video frame sequence.

[0101] Step 706: For the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample, a multi-layer attention network is used to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence.

[0102] In this multi-layer attention network, each layer employs a multi-head attention mechanism.

[0103] Step 707: Using machine learning methods, the core entity recognition model is obtained by taking the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the expected output.

[0104] As can be seen from this embodiment, with Figure 2 Compared with the corresponding embodiments, the process 700 of the core entity recognition model generation method in this embodiment specifically illustrates the determination process of entity feature sequence, the determination process of video frame feature sequence, and the fusion process of multimodal data, which enables the core entity recognition model to better consider multimodal data (text data, video data) in the video, and further improves the recognition accuracy of the core entity recognition model.

[0105] Continue to refer to Figure 8 The illustration shows a schematic flow 800 of an embodiment of a core entity tag identification method according to the present disclosure, including the following steps:

[0106] Step 801: Determine the text data and video data in the acquired target video.

[0107] In this embodiment, the entity executing the core entity label identification method (e.g., Figure 1 The terminal device or server in the process can acquire the target video and determine the text data and video data within the acquired target video. Specifically, the video data consists of the video frames in the target video, and the text data consists of the text describing or explaining the video frames.

[0108] The text data can be obtained by the aforementioned execution entity through text recognition of subtitles in video frames, or through speech recognition of audio in the video.

[0109] Step 802: Determine the entity feature sequence of the entities in the text data.

[0110] As an example, the aforementioned execution entity can use the entity recognition module in the core entity recognition model to determine the entity feature sequence of entities in text data. The entity recognition module is used to extract all entity words from the text data and determine the entity features of each entity. Entity words can serve as potential video tags.

[0111] Specifically, step 802 in this embodiment can be performed with reference to step 202 in the above embodiment 200, and will not be repeated here.

[0112] In some optional implementations of this embodiment, the execution entity can perform step 802 as follows: First, perform entity recognition on the text data to determine the entity sequence including the entities in the text data; then, for each entity in the entity sequence, use the entity representation vector corresponding to the entity output during the entity recognition process as the entity feature of the entity to obtain the entity feature sequence corresponding to the entity sequence.

[0113] Specifically, this implementation method can be executed with reference to the corresponding implementation method in the above embodiment 200, and will not be repeated here.

[0114] Step 803: Determine the video frame feature sequence of the video frames in the video data.

[0115] As an example, the aforementioned execution entity can determine the video frame feature sequence of video frames in video data through the video encoding module in the core entity recognition model. The video encoding module can be an encoder that encodes video frames to generate video frame features. As an example, the video encoder adopts the ViT (Vision Transformer) model.

[0116] Specifically, step 803 in this embodiment can be performed with reference to step 203 in the above embodiment 200, and will not be repeated here.

[0117] In some optional implementations of this embodiment, the execution entity can perform step 803 as follows: First, determine the video frame sequence from the video data based on a preset frame rate; then, extract the video frame features of the video frames in the video sequence to obtain the video frame feature sequence corresponding to the video frame sequence.

[0118] Specifically, this implementation method can be executed with reference to the corresponding implementation method in the above embodiment 200, and will not be repeated here.

[0119] Step 804: Fuse the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence.

[0120] As an example, the aforementioned execution entity can use the multimodal data fusion module in the core entity recognition model to fuse entity feature sequences and video frame feature sequences to obtain a fused feature sequence. Specifically, step 804 in this embodiment can be executed with reference to step 204 in embodiment 200 above, and will not be repeated here.

[0121] Step 805: Using the core entity recognition model, determine the core entity labels corresponding to the target video based on the fused feature sequence.

[0122] As an example, the aforementioned execution entity can determine the core entity label corresponding to the target video based on the fused feature sequence through the core entity recognition module in the core entity recognition model.

[0123] As an example, the core entity recognition module outputs the coreity value of entities in the entity sequence. If the coreity value is greater than a preset threshold, the entity is identified as a core entity label; otherwise, the entity is identified as a non-core entity label.

[0124] After determining the core entity tags corresponding to the target video, the aforementioned execution entity can store or transmit the core entity tags to downstream tasks (e.g., video recommendation tasks).

[0125] In this embodiment, the core entity recognition model fuses the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data, and determines the core entity label based on the obtained fused feature sequence. This allows the core entity recognition model to fully consider the multimodal data (text data and video data) in the video, thereby improving the accuracy of the determined core entity label.

[0126] Continue to refer to Figure 9 The illustration shows a schematic flow 900 of another embodiment of the core entity tag identification method according to the present disclosure, including the following steps:

[0127] Step 901: Determine the text data and video data in the acquired target video.

[0128] Step 902: Perform entity recognition on the text data to determine the entity sequence that includes the entities in the text data.

[0129] Step 903: For each entity in the entity sequence, the entity representation vector corresponding to the entity output during the entity recognition process is taken as the entity feature of the entity, and the entity feature sequence corresponding to the entity sequence is obtained.

[0130] Step 904: Determine the video frame sequence from the video data based on the preset frame rate.

[0131] Step 905: Extract the video frame features from the video frames in the video sequence to obtain the video frame feature sequence corresponding to the video frame sequence.

[0132] Step 906: A multi-layer attention network is used to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence.

[0133] In this multi-layer attention network, each layer employs a multi-head attention mechanism.

[0134] Step 907: Using the core entity recognition model, determine the core entity label corresponding to the target video based on the fused feature sequence.

[0135] As can be seen from this embodiment, with Figure 8 Compared with the corresponding embodiments, the process 900 of the core entity label recognition method in this embodiment specifically illustrates the determination process of entity feature sequence, the determination process of video frame feature sequence, and the fusion process of multimodal data, so that the core entity recognition model can more fully consider the multimodal data (text data, video data) in the video, and further improve the accuracy of the determined core entity labels.

[0136] Continue to refer to Figure 10 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a core entity recognition model generation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0137] like Figure 10 As shown, the core entity recognition model generation device 1000 includes: an acquisition unit 1001 configured to acquire a training sample set, wherein the training samples in the training sample set include corresponding text data and video data, and labels representing whether entities in the text data are core entities; a first determination unit 1002 configured to determine the entity feature sequence of entities in the text data; a second determination unit 1003 configured to determine the video frame feature sequence of video frames in the video data; a first fusion unit 1004 configured to fuse the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample to obtain a fused feature sequence; and a training unit 1005 configured to use a machine learning method, taking the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the expected output, to obtain a trained core entity recognition model.

[0138] In some optional implementations of this embodiment, the first fusion unit 1004 is further configured to: for the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample, adopt a multi-head attention mechanism to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence.

[0139] In some optional implementations of this embodiment, the first fusion unit 1004 is further configured to: employ a multi-layer attention network to perform feature fusion of multimodal data on entity feature sequences and video frame feature sequences to obtain a fused feature sequence, wherein each layer of the multi-layer attention network employs a multi-head attention mechanism.

[0140] In some optional implementations of this embodiment, the first determining unit 1002 is further configured to: perform entity recognition on the text data to determine an entity sequence including entities in the text data; for each entity in the entity sequence, take the entity representation vector corresponding to the entity output during the entity recognition process as the entity feature of the entity to obtain the entity feature sequence corresponding to the entity sequence.

[0141] In some optional implementations of this embodiment, the second determining unit 1003 is further configured to: determine a video frame sequence from the video data based on a preset frame rate; extract video frame features from the video frames in the video sequence to obtain a video frame feature sequence corresponding to the video frame sequence.

[0142] This embodiment provides a method for generating a core entity recognition model. During training, entity feature sequences corresponding to text data and video frame feature sequences corresponding to video data in the same training sample are fused. The model is trained based on the obtained fused feature sequences and the labels corresponding to the fused feature sequences, resulting in a core entity recognition model that includes an entity recognition module, a video encoding module, a multimodal data fusion module, and a core entity recognition module. This allows the core entity recognition model to fully consider the multimodal data (text data and video data) in the video, improving the recognition accuracy of the core entity recognition model. Furthermore, the core entity recognition model is used to identify the core entity labels in the video, improving the accuracy of the determined core entity labels.

[0143] Continue to refer to Figure 11 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a core entity label recognition method, which is similar to... Figure 8 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0144] like Figure 11As shown, the core entity label recognition method 1100 includes: a third determining unit 1101, configured to determine text data and video data in the acquired target video; a fourth determining unit 1102, configured to determine the entity feature sequence of entities in the text data; a fifth determining unit 1103, configured to determine the video frame feature sequence of video frames in the video data; a second fusion unit 1104, configured to fuse the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence; and a sixth determining unit 1105, configured to determine the core entity label corresponding to the target video based on the fused feature sequence using a core entity recognition model.

[0145] In some optional implementations of this embodiment, the second fusion unit 1104 is further configured to: employ a multi-head attention mechanism to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence.

[0146] In some optional implementations of this embodiment, the second fusion unit 1104 is further configured to: employ a multi-layer attention network to perform feature fusion of multimodal data on entity feature sequences and video frame feature sequences to obtain a fused feature sequence, wherein each layer of the multi-layer attention network employs a multi-head attention mechanism.

[0147] In some optional implementations of this embodiment, the fourth determining unit 1102 is further configured to: perform entity recognition on the text data to determine an entity sequence including entities in the text data; for each entity in the entity sequence, take the entity representation vector corresponding to the entity output during the entity recognition process as the entity feature of the entity to obtain the entity feature sequence corresponding to the entity sequence.

[0148] In some optional implementations of this embodiment, the fifth determining unit 1103 is further configured to: determine a video frame sequence from the video data based on a preset frame rate; extract video frame features from the video frames in the video sequence to obtain a video frame feature sequence corresponding to the video frame sequence.

[0149] In this embodiment, the core entity recognition model integrates the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data, and determines the core entity label based on the obtained fused feature sequence. This enables the core entity recognition model to better consider the multimodal data (text data and video data) in the video, thereby improving the accuracy of the determined core entity label.

[0150] According to embodiments of this disclosure, this disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the method for generating a core entity recognition model and the method for recognizing a core entity label as described in any of the above embodiments.

[0151] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the core entity recognition model generation method and the core entity label recognition method described in any of the above embodiments when executed.

[0152] This disclosure provides a computer program product that, when executed by a processor, can implement the method for generating a core entity recognition model and the method for recognizing core entity tags described in any of the above embodiments.

[0153] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0154] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0155] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0156] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the method for generating a core entity recognition model. For example, in some embodiments, the method for generating a core entity recognition model may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the method for generating a core entity recognition model described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured by any other suitable means (e.g., by means of firmware) to perform a method for generating a core entity recognition model.

[0157] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0159] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0161] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0162] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers integrated with blockchain technology.

[0163] According to the technical solution of this disclosure, a method for generating a core entity recognition model is provided. During the training process, entity feature sequences corresponding to text data and video frame feature sequences corresponding to video data in the same training sample are fused. The model is trained based on the obtained fused feature sequences and the labels corresponding to the fused feature sequences, resulting in a core entity recognition model including an entity recognition module, a video encoding module, a multimodal data fusion module, and a core entity recognition module. This allows the core entity recognition model to fully consider the multimodal data (text data and video data) in the video, improving the recognition accuracy of the core entity recognition model. Furthermore, the recognition of core entity labels in the video using the core entity recognition model improves the accuracy of the determined core entity labels.

[0164] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0165] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a core entity recognition model, comprising: Obtain a training sample set, wherein the training samples in the training sample set include corresponding text data and video data, as well as labels characterizing whether the entities in the text data are core entities; Determine the entity feature sequence of the entities in the text data; Determine the video frame feature sequence of the video frames in the video data; By fusing the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample, a fused feature sequence is obtained. The fused feature in the fused feature sequence corresponds one-to-one with the entity feature in the entity feature sequence. The fused feature is obtained by fusing the corresponding entity feature and all video features in the video frame feature sequence. A machine learning approach is used, with the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the expected output, to obtain the trained core entity recognition model.

2. The method of claim 1, wherein, The fused feature sequence is obtained by fusing the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample, including: For the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample, a multi-head attention mechanism is used to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence.

3. The method of claim 2, wherein, The multi-head attention mechanism is used to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence, including: A multi-layer attention network is used to perform feature fusion of multimodal data on the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence. Each layer of the multi-layer attention network adopts a multi-head attention mechanism.

4. The method of claim 1, wherein, Determining the entity feature sequence of entities in the text data includes: Entity recognition is performed on the text data to determine an entity sequence that includes the entities in the text data; For each entity in the entity sequence, the entity representation vector corresponding to that entity output during the entity recognition process is taken as the entity feature of that entity, and the entity feature sequence corresponding to the entity sequence is obtained.

5. The method of claim 1, wherein, Determining the video frame feature sequence of the video frames in the video data includes: The video frame sequence is determined from the video data based on a preset frame rate; Extract the video frame features from the video frame sequence to obtain the video frame feature sequence corresponding to the video frame sequence.

6. A method for identifying core entity tags, comprising: Identify the text and video data in the acquired target video; Determine the entity feature sequence of the entities in the text data; Determine the video frame feature sequence of the video frames in the video data; The entity feature sequence and the video frame feature sequence are fused to obtain a fused feature sequence, wherein the fused features in the fused feature sequence correspond one-to-one with the entity features in the entity feature sequence, and the fused features are obtained by fusing the corresponding entity features and all video features in the video frame feature sequence; The core entity labels corresponding to the target video are determined by using the core entity recognition model and based on the fused feature sequence.

7. The method of claim 6, wherein, The process of fusing the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence includes: A multi-head attention mechanism is used to perform feature fusion of multimodal data on the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence.

8. The method of claim 7, wherein, The multi-head attention mechanism is employed to perform multimodal data feature fusion on the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence, including: A multi-layer attention network is used to perform feature fusion of multimodal data on the entity feature sequence and the video frame feature sequence to obtain the fused feature sequence. Each layer of the multi-layer attention network adopts a multi-head attention mechanism.

9. The method of claim 6, wherein, Determining the entity feature sequence of entities in the text data includes: Entity recognition is performed on the text data to determine an entity sequence that includes the entities in the text data; For each entity in the entity sequence, the entity representation vector corresponding to that entity output during the entity recognition process is taken as the entity feature of that entity, and the entity feature sequence corresponding to the entity sequence is obtained.

10. The method of claim 6, wherein, Determining the video frame feature sequence of the video frames in the video data includes: The video frame sequence is determined from the video data based on a preset frame rate; Extract the video frame features from the video frame sequence to obtain the video frame feature sequence corresponding to the video frame sequence.

11. A device for generating a core entity recognition model, comprising: The acquisition unit is configured to acquire a training sample set, wherein the training samples in the training sample set include corresponding text data and video data, as well as labels characterizing whether the entities in the text data are core entities. The first determining unit is configured to determine the entity feature sequence of entities in the text data; The second determining unit is configured to determine the video frame feature sequence of the video frames in the video data; The first fusion unit is configured to fuse the entity feature sequence corresponding to the text data and the video frame feature sequence corresponding to the video data in the same training sample to obtain a fused feature sequence. The fused features in the fused feature sequence correspond one-to-one with the entity features in the entity feature sequence. The fused features are obtained by fusing the corresponding entity features and all video features in the video frame feature sequence. The training unit is configured to use machine learning methods, taking the fused feature sequence as input and the labels in the training samples corresponding to the input fused feature sequence as the expected output, to obtain the trained core entity recognition model.

12. A device for identifying a core entity tag, comprising: The third determining unit is configured to determine the text data and video data in the acquired target video; The fourth determining unit is configured to determine the entity feature sequence of entities in the text data; The fifth determining unit is configured to determine the video frame feature sequence of the video frames in the video data; The second fusion unit is configured to fuse the entity feature sequence and the video frame feature sequence to obtain a fused feature sequence, wherein the fused features in the fused feature sequence correspond one-to-one with the entity features in the entity feature sequence, and the fused features are obtained by fusing the corresponding entity features and all video features in the video frame feature sequence; The sixth determining unit is configured to determine the core entity label corresponding to the target video based on the fused feature sequence using a core entity recognition model.

13. An electronic device, comprising: include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

14. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

15. A computer program product, comprising: A computer program that, when executed by a processor, implements the method of any one of claims 1-10.

Images