Methods, apparatus, equipment and media for generating descriptive information of multimedia data

Abstract

This application provides a method, apparatus, device, and medium for generating descriptive information of multimedia data. The method includes: extracting feature information from multimedia data to be processed, wherein the multimedia data includes video or images; and generating a textual description of the multimedia data based on the extracted feature information. The method provided by this application can effectively improve the accuracy of the generated textual description of multimedia data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, electronic device, and storage medium for generating descriptive information of multimedia data. Background Technology

[0002] In computer vision technology, video captioning or image captioning refers to outputting a textual description of a given video or image. For example, for a video of a child cleaning the floor, video captioning can automatically output the textual description "a child is cleaning the floor". Video captioning is an intersection of computer vision and natural language processing.

[0003] Existing video description methods generally involve selecting several frames from a video, extracting full-image features from these frames, decoding them using these features, and generating a text description of the video based on maximum likelihood probability. Image description follows a similar principle. As can be seen, current video description models primarily employ an encoder-decoder structure, where the encoder extracts features from video frames, and the decoder decodes these features to generate text descriptions. Although various methods for generating video description information exist, the accuracy of the generated video descriptions still needs improvement. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for generating video description information, so as to improve the accuracy of the generated video description information. The specific solutions provided by this application are as follows:

[0005] In a first aspect, embodiments of this application provide a method for generating descriptive information of multimedia data, the method comprising:

[0006] Extract feature information from the multimedia data to be processed, whereby the multimedia data includes video or images;

[0007] Based on the extracted feature information, a textual description of the multimedia data is generated.

[0008] Secondly, embodiments of this application provide an apparatus for generating descriptive information of multimedia data, the apparatus comprising:

[0009] The feature information extraction module is used to extract feature information from the multimedia data to be processed, wherein the multimedia data includes video or images;

[0010] The description information generation module is used to generate textual descriptions of multimedia data based on the extracted feature information.

[0011] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor; wherein the memory stores a computer program; and the processor is used to execute the method provided in embodiments of this application when running the computer program.

[0012] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run by a processor, executes the methods provided in embodiments of this application.

[0013] The beneficial effects of the technical solutions provided in this application will be described in detail in the following description of the specific implementation methods in conjunction with various optional embodiments, and will not be described again here. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0015] Figure 1 This is a schematic diagram illustrating an example of image description.

[0016] Figure 2 This is a schematic diagram illustrating an example of video description.

[0017] Figure 3 This is a schematic diagram illustrating the principle of an existing video description algorithm;

[0018] Figure 4 This is a schematic diagram of the training process of an existing supervised learning-based video description algorithm;

[0019] Figure 5 A flowchart illustrating a method for generating description information of multimedia data provided in an embodiment of this application;

[0020] Figure 6 This is a schematic diagram illustrating the principle of obtaining semantic features through a semantic prediction network, as provided in one example of this application.

[0021] Figure 7a This is a schematic diagram of a spatial scene in one example of this application;

[0022] Figure 7b This is a schematic diagram of a spatial scene in another example of this application;

[0023] Figure 8 This is a schematic diagram illustrating the principle of obtaining relation features through a relation prediction network in one example of this application;

[0024] Figure 9This is a schematic diagram illustrating the principle of obtaining attribute features through an attribute prediction network in one example of this application;

[0025] Figure 10 This is a schematic diagram of a space-time scene in one example of this application;

[0026] Figure 11 This is a schematic diagram of a space-time scene diagram in another example of this application.

[0027] Figure 12 This application provides a schematic diagram of the principle of a feature selection network in one example;

[0028] Figure 13a This is a schematic diagram of the structure of a self-attention-based encoder-decoder in one example of this application;

[0029] Figure 13b This is a schematic diagram of the structure of a self-attention-based encoder-decoder in one example of this application;

[0030] Figure 14 , Figure 15 and Figure 16 These are schematic diagrams illustrating the methods for generating video description information provided in the three examples of this application;

[0031] Figure 17a and Figure 17b This is a schematic diagram illustrating the principle of obtaining video description information in two optional examples of this application;

[0032] Figure 18 and Figure 19 The flowcharts for the methods of generating video description information in two other examples of this application are shown.

[0033] Figure 20 This is a schematic diagram illustrating the principle of obtaining video description information in one example of this application;

[0034] Figure 21 A flowchart illustrating a method for generating image description information provided in an embodiment of this application;

[0035] Figure 22 and Figure 23 These are schematic diagrams of the codecs provided in two optional examples of this application;

[0036] Figure 24 A flowchart illustrating a training method for a multimedia data description model provided in an embodiment of this application;

[0037] Figure 25 This is a schematic diagram of a sample video with video description annotations (i.e., original description annotations) in one example of this application;

[0038] Figure 26 This is a schematic diagram illustrating the principle of a training method for a video description model in one example of this application;

[0039] Figure 27 A flowchart illustrating a method for obtaining enhanced multimedia data description information provided in an embodiment of this application;

[0040] Figure 28a and Figure 28b These are schematic diagrams illustrating the structures of the two codecs provided in the two optional examples of this application;

[0041] Figure 29 A flowchart illustrating a training method for an image description model provided in an embodiment of this application;

[0042] Figure 30 and Figure 31 This is a schematic diagram illustrating the principle of the video description information generation method provided in the two examples of this application;

[0043] Figure 32 A schematic diagram of a device for generating descriptive information of multimedia data provided in an embodiment of this application;

[0044] Figure 33 This is a schematic diagram of the structure of an electronic device to which this application applies. Detailed Implementation

[0045] The embodiments of this application are described in detail below. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application to aid in a comprehensive understanding of the embodiments of this application as defined by the claims and their equivalents. Various specific details are included to aid understanding, but these examples and details are considered exemplary only and should not be construed as limiting this application. Therefore, those skilled in the art will recognize that changes and modifications can be made to the described embodiments without departing from the scope and spirit of this application. Furthermore, for clarity and brevity, some descriptions of well-known functions and structures may be omitted in the following description.

[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0047] First, it should be noted that the method for generating description information of multimedia data provided in this application embodiment can be used to generate description information of a video containing multiple frames of images, or it can be used to generate description information of a single image. The source of this image is not limited in this application embodiment; it can be an image that has been captured, downloaded, or received, or it can be an image from a video, such as a keyframe image or a specified frame image. In other words, the generation method in this application embodiment can be a method for generating description information of a video or a method for generating description information of an image.

[0048] To better illustrate and understand the solutions provided in the embodiments of this application, the technologies related to the embodiments of this application will be described first below.

[0049] In computer vision, video / image description refers to outputting a textual description of a given video / image. It represents the intersection of computer vision and natural language processing. Compared to other computer vision tasks such as object detection and image segmentation, video / image description is a more challenging task. It requires not only a more comprehensive understanding of the video or image but also the ability to express its content in natural language. Figure 1 As shown, when given as Figure 1 When the image shown is displayed, the system can automatically output a text description of the image: "A boy is playing tennis." For example... Figure 2 As shown, when given Figure 2 When a video containing multiple frames is shown, a text description of the video, such as "A child is cleaning the floor," can be automatically output.

[0050] Currently, most existing image description models employ an encoder-decoder structure. The encoder is typically designed based on Convolutional Neural Networks (CNNs), which is responsible for extracting image features, while the decoder is typically designed based on Recurrent Neural Networks (RNNs), which is responsible for decoding image features and generating text descriptions.

[0051] Similarly, existing video description models generally select several frames from the video, use a CNN to extract full-image features from these selected frames, and then use an RNN to decode the features of all frames, generating a text description of the video based on maximum likelihood probability. It is evident that existing video description algorithms basically adopt an encoder-decoder structure. The CNN mentioned above is used to encode video frames and is responsible for extracting features from them; therefore, it can also be called an encoder or CNN encoder. The RNN mentioned above decodes video frames, is responsible for decoding the features of the video frames and generating text descriptions; therefore, it can also be called a decoder or RNN decoder. The RNN can be a Long Short-Term Memory (LSTM) network, in which case it can be called an LSTM decoder.

[0052] As an example Figure 3 The diagram illustrates the principle of an existing video description model, such as... Figure 3 As shown, select from the video Figure 3 The several frames shown (the ellipses in the figure indicate frames that are omitted or not shown) are processed by a CNN encoder to extract features from each selected video frame. The extracted features are then decoded by an LSTM decoder to generate the corresponding text description "A man is putting pizza in the oven".

[0053] Although existing technologies can generate text descriptions of videos or images, the inventors of this application have found through analysis and research that the existing technologies still have at least the following technical problems:

[0054] (1) Existing decoders, such as RNNs, are recurrent structures that require step-by-step training. Therefore, existing decoders suffer from slow training speed, low training efficiency, difficulty in learning long-distance correlations, and insufficient expressive power.

[0055] (2) In the datasets commonly used in the field of video / image description, the descriptive information of training samples (i.e., sample videos or sample images) is relatively limited. For example, the descriptive annotations of sample images are usually only 5. It is usually difficult to fully express the information in the image using only 5 descriptive annotations. Moreover, due to the diversity of natural language, the same semantic meaning can be expressed in multiple ways. Therefore, the poor diversity of descriptive information in training samples is also a problem that hinders the further development of this field.

[0056] (3) For videos containing multiple frames, the existing technology does not take into account intra-frame information. However, this information is of great significance for generating more accurate video descriptions. Therefore, it is necessary to solve the problem of how to make full use of intra-frame information.

[0057] (4) Existing technologies do not take into account the semantic information of videos or images, which is of great significance for generating more accurate video descriptions.

[0058] (5) Existing video or image description models are generally based on supervised learning methods. For example, in video description algorithms, each training video corresponds to one or more labeled video descriptions, and then reinforcement learning methods are used to train the video description model, such as... Figure 4 As shown, for data with labeled video descriptions, the video data P in this data is input into the video description model K, which analyzes and processes the video data P to generate corresponding video descriptions. Then, the loss function T is calculated based on the video descriptions Q in the labeled data and the generated video descriptions. mark The value of (α) is determined by the loss function T. mark (α) guides the learning of the video description model K. However, annotating videos requires a lot of manpower and time, which not only limits the number of samples in existing video description datasets, but also leads to poor accuracy and precision of video description models trained on such datasets.

[0059] (6) In the existing methods of generating video or image description information, the length of the generated description information is uncontrollable and cannot meet the application needs of users for different lengths of description information required in different application scenarios. For example, when users publish images or videos, they want a long description information in order to share more detailed information. For another example, when users are driving, they need a short description information, but the existing technology cannot meet this need.

[0060] In order to solve at least one of the above-mentioned technical problems existing in the prior art, embodiments of this application provide a method, apparatus, electronic device and storage medium for generating descriptive information of multimedia data, wherein the multimedia data may be video or image.

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following will describe in detail, with reference to specific embodiments and accompanying drawings, the various optional implementation methods of this application and how the technical solutions of the embodiments of this application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0062] Figure 5 The figure shows a flowchart illustrating a method for generating descriptive information of multimedia data according to an embodiment of this application. As shown in the figure, the method mainly includes the following steps:

[0063] Step S101: Extract feature information from the multimedia data to be processed;

[0064] Step S102: Based on the extracted feature information, generate a text description of the multimedia data.

[0065] In this embodiment of the application, the multimedia data includes video or images. Based on the method provided in this embodiment of the application, it is possible to generate video description information or image description information.

[0066] In an optional embodiment of this application, the feature information extracted from the multimedia data to be processed includes at least one of the following:

[0067] Extract local visual features of the target contained in each target region of each image in the multimedia data;

[0068] Extracting semantic features from multimedia data;

[0069] If the multimedia data is video, extract the spatiotemporal visual features of the multimedia data;

[0070] Extracting global visual features from multimedia data;

[0071] Extract the attribute features of the target contained in each target region of each image in the multimedia data;

[0072] Extract global attribute features for each image in the multimedia data.

[0073] In other words, the feature information of multimedia data can include one or more (including two) of the following: local visual features, semantic features, spatiotemporal visual features, global visual features, local attribute features (i.e., the attribute features of the target), and global attribute features. Among them, "local" in the above-mentioned local visual features refers to the target region relative to the image to which it belongs; local visual features are the visual features of a target region in the image.

[0074] In this context, for images, each image in the aforementioned multimedia data refers to that image. For videos, each image in the multimedia data can be a frame from the video, or it can be a selection of frames from the video. For example, for a given video, several frames can be selected at equal intervals, or a keyframe algorithm can be used to select several image frames (i.e., keyframes) from the given video, or a neural network can be used for frame selection. In the following description, for videos, the images in the multimedia data will be illustrated using frames selected from the video as an example.

[0075] Visual features are those that reflect pixel information in an image, while attribute features reflect the attribute information of each target in an image. Therefore, visual features and attribute features can be used to generate descriptive information for videos or images. Local visual features can more accurately and in more detail reflect the information of each target region in each image. Thus, local visual features make fuller use of the intra-frame information of each image, thereby generating more accurate textual descriptions of videos or images.

[0076] In practical applications, in addition to visual and attribute features of images, other features are also helpful in describing multimedia data. For example, the spatiotemporal visual features of a video can effectively reflect the dynamic changes of a video in space and time, while the semantic features of a video or image reflect the semantic information of the content contained in a video or image. Therefore, when generating text descriptions of videos or images, the spatiotemporal visual features of the video and the semantic features of the video or image can be extracted through neural networks to integrate more diverse features into the generation of video description information, thereby enhancing the expressive power of the generated video description and improving the accuracy of the description information.

[0077] Optionally, for each image, a feature extraction network can be used to obtain several target regions and the regional features of each target region (i.e., the aforementioned local visual features, also known as target features). Attribute features can be obtained through an attribute prediction network. For example, a Faster Region-CNN (Faster R-CNN) can be applied to each image to extract several target regions and the local visual features of each target region (i.e., local features / region features). Faster R-CNN can be pre-trained using sample datasets (such as ImageNet or Visual Genome datasets).

[0078] It should be noted that the method of extracting each target region and its local features using Faster Region-CNN is merely exemplary, and this application is not limited to this. The feature extraction network can also be implemented using any other available neural network.

[0079] For spatiotemporal visual features of videos, spatiotemporal feature extraction networks can be used. These can be obtained using models based on ECO (Efficient Convolutional Network for Online Video Understanding) or 3D-CNN, or other spatiotemporal visual feature extraction networks. Semantic features, on the other hand, can be extracted using semantic prediction networks.

[0080] Specifically, a trained semantic prediction network can be applied to obtain the semantic features of the entire video or image. As an example, Figure 6 The figure shows a schematic diagram of a semantic prediction network structure. As shown, this semantic prediction network can include CNN and a multi-classification structure (multi-classification shown in the figure) to... Figure 6 Taking the video shown as an example, when using this network, several frames of the selected video are input into the semantic prediction network. The CNN extracts the video features of each frame, and the extracted video features are then subjected to multi-classification operations using a multi-classification structure. This allows for the acquisition of probabilities corresponding to various predefined semantic features of the video. Finally, one or more predefined semantic features are output based on these probabilities. For example... Figure 6As shown in the figure, based on several input frames (the images containing people and dogs shown in the figure), the probability of various semantic features corresponding to people, dogs, roads, etc. can be obtained through the semantic prediction network. For example, the probability of containing a person is 0.95 and the probability of containing a dog is 0.8. Based on the predicted probability and the pre-configured semantic feature filtering rules, semantic features with a probability greater than a set threshold can be used as output, or a set number of semantic features with a high probability can be used as output.

[0081] In an optional embodiment of this application, the feature information of the multimedia data includes the local visual features of the target contained in each target region of each image of the multimedia data. Based on the extracted feature information, a text description of the multimedia data is generated, including:

[0082] For each image, the relationship features between the targets are obtained based on the local visual features of each target in the image, and a scene graph of the image is constructed based on the local visual features and the relationship features between the targets.

[0083] For each image, obtain the graph convolutional features of the image based on the scene graph of that image;

[0084] Textual descriptions of multimedia data are obtained based on the graph convolution features of each image in the multimedia data.

[0085] A scene graph is a graph structure that represents the local visual features, attribute features (which will be explained in more detail later), and relational features of each target region in an image. A scene graph can include multiple nodes and multiple edges. Each node specifically represents the target features (i.e., the aforementioned local visual features) or attribute features of the target contained in the target region (i.e., the object). Each edge represents the relational features between the nodes.

[0086] As an example, Figure 7a This shows a scene diagram corresponding to a single frame of an image, such as... Figure 7a As shown, certain nodes in the scene graph represent the local visual features of each target in the frame image, for example... Figure 7a The nodes "person," "dog," and "skateboard" in the diagram represent the feature vectors of "person," "dog," and "skateboard," respectively. Some nodes represent the attribute features of the target; for example, the node "wearing blue clothes" represents the attribute feature of the target "person," while "black" represents the attribute feature of the "dog." The edges between nodes in the scene graph represent the relationship between the two connected nodes, that is, the relationship between two targets. For example, the relationship between nodes "person" and "dog" is "leaning," and the relationship between "person" and "skateboard" is "skating," etc. Figure 7aAs can be seen, this scene graph can reflect the spatial relationships between various objects within an image, and therefore it can also be called a spatial scene graph.

[0087] In practical applications, for each image, a feature extraction network can be used to obtain several target regions and their regional features (i.e., the aforementioned local visual features), and a relationship prediction network can be used to obtain the relationship features between the target regions. Specifically, after obtaining the local features of each target region of the image through the feature extraction network, a pre-trained relationship prediction network can be used to obtain the relationship features between the target regions from the extracted target region features, and the regional features of each target region and their relationship features are represented graphically to obtain the scene map for each frame.

[0088] The relationship prediction network is a classification network used to predict the relationships between target regions. The specific network structure can be selected based on actual needs. As an optional approach, the relationship prediction network can include fully connected layers, feature concatenation layers, and softmax layers. Fully connected layers can not only extract local visual features of the target regions but also perform dimensionality reduction to accommodate subsequent feature concatenation and softmax layer processing. The relationship prediction network can be trained on a sample dataset, such as Visual Genome. Visual Genome is a commonly used dataset for relationship and attribute learning, containing a large number of object attribute and relationship annotations; therefore, it can be used to train the relationship prediction network.

[0089] Specifically, when obtaining the relationship features between various target regions through the aforementioned relationship prediction network, for the regional features of at least two target regions in each target region, a fully connected layer can be used to extract features. The features corresponding to each target region extracted by the fully connected layer are input into the feature concatenation layer for feature concatenation and then input into the softmax layer. Based on the probabilities corresponding to each relationship output by the softmax layer, the relationship features between at least two target regions are obtained.

[0090] As an example, Figure 8 The diagram illustrates a method for predicting relation features using a relation prediction network, such as... Figure 8As shown, the regional features of the target person and the dog in an image frame are input into the fully connected layer of the relationship prediction network. The features output from the fully connected layer are concatenated and then input into the softmax layer to obtain the probabilities corresponding to various predefined relationship features between the person and the dog. Finally, one or more predefined relationship features are output based on the probabilities. For example, the probability of the relationship feature "above" is 0.12, the probability of the relationship feature "on the left" is 0.20, and the probability of the relationship feature "leading" is 0.40, etc. Based on the probabilities of each relationship feature, the final output relationship feature is "leading". Of course, in practical applications, for at least two target regions, the corresponding relationship feature can be one, such as selecting the relationship feature with the highest probability in the example above, or multiple, such as outputting relationship features with probabilities higher than a set threshold or outputting the highest probability set number (such as two) of relationship features.

[0091] The method provided in this application embodiment obtains text descriptions of videos or images based on the local visual features (or simply region features, local features) of each target region (also called target candidate region) of each image. Since the local visual features of each target region can more accurately and in more detail reflect the information of each local region in each image, the method in this application embodiment makes full use of the intra-frame information of each image, thereby generating more accurate text descriptions of videos or images.

[0092] Furthermore, the method in this embodiment can determine the relationship features between target regions based on local features, construct a spatial scene map for each image (which is essentially one image for each image) based on these relationship features, and obtain graph convolution features for each image based on the spatial scene map. This allows for the acquisition of textual descriptions of videos or images based on the graph convolution features of each image. Since spatial scene maps effectively reflect the targets in an image and the relationships between them, and these relationships are very helpful for understanding and describing image content, graph convolution features based on spatial scene maps can further improve the accuracy of textual descriptions of videos or images.

[0093] For ease of description, in the following description, for a target region, the node corresponding to the local visual features of the target region can be simply referred to as the target node, and the local visual features represented by the target node can also be called the target feature. The node representing the attribute features of the target region can be simply referred to as the attribute node.

[0094] In an optional embodiment of this application, the feature information of the multimedia data may include the attribute features of the target contained in each target region of each image in the multimedia data;

[0095] The scene graph of the image, constructed based on the local visual features of each target and the relationship features between them, may include:

[0096] Based on the local visual features of each target, the relationship features between each target, and the attribute features of each target, a scene graph of the image is constructed. Each node in the scene graph represents the local visual features or attribute features of the target corresponding to the target region.

[0097] In this alternative approach, the local visual features, attribute features, and relational features corresponding to each target region are used to construct a scene graph. That is, the scene graph can contain both target nodes and attribute nodes. In this case, the nodes in the scene graph represent the local visual features or attribute features of each target region, reflecting the target features and attributes of the targets contained within each region. The corresponding edges can include edges representing the relationships between targets (e.g., ...). Figure 7a The image shows the edges connecting "people" and "dogs" in the spatial scene diagram, as well as the edges representing the relationships between targets and attributes (such as...). Figure 7a (The edge connecting the "person" and the "person in blue clothes" in the spatial scene diagram shown).

[0098] In practical applications, to reduce connection redundancy, attribute nodes and target nodes for the same target region can be merged. That is, for a target region, nodes representing its local visual features and attribute features can be the same node. For example, for... Figure 7a The scene diagram shown can be obtained by merging target nodes and attribute nodes of the same objective, as shown below. Figure 7b The scene diagram shown is as follows: Figure 7a In the image, "person" and "wearing blue clothes" represent the target feature and attribute feature of the person, respectively, and therefore can be merged into... Figure 7b The node shown, representing a person wearing blue clothes, simultaneously reflects both the category and attribute characteristics of the person in the corresponding target area. Similarly... Figure 7a In the image, "dog" and "black" represent the target feature and attribute feature of the dog, respectively, and therefore can be combined as follows: Figure 7b The "dog black" node is shown.

[0099] The attribute features can be obtained based on the local visual features of the target region. This alternative approach also considers the attribute features of each target region in the image when constructing the scene graph. Since the attributes of each target in the image are also very helpful in describing the image content, a scene graph that incorporates attribute features can more accurately describe the video.

[0100] Optionally, the attribute features corresponding to each target region can be obtained from the extracted local visual features using an attribute prediction network. The attribute prediction network is a multi-classification network that can be trained based on a sample dataset (such as the Visual Genome dataset). In practical applications, the specific network structure of the attribute prediction network can be selected according to actual needs.

[0101] In an optional embodiment of this application, the attribute prediction network may include multiple attribute classifiers, wherein each classifier corresponds to a type of attribute prediction.

[0102] The specific classification method for attribute types can be configured according to actual needs. As an optional method, the attribute type can specifically refer to the part of speech corresponding to the attribute, such as nouns, verbs, adjectives, and other relatively uncommon attribute types. Attribute features are predicted by employing a classifier that includes classes corresponding to multiple attribute types.

[0103] When constructing a spatial scene map based on the predicted attribute features of this scheme, compared with traditional methods, the traditional method does not distinguish the attributes of objects (targets) and classifies various attributes using a single classifier, resulting in low attribute accuracy. However, the scheme in this embodiment can obtain more specific target attribute information, including noun attributes (such as clothing, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, miniature horse, etc.). Furthermore, different classifiers can be used for different types of attributes, resulting in more accurate and diverse attributes. This allows for the generation of more accurate descriptive information based on the predicted attribute features. In addition, to reduce redundancy, attribute nodes and target nodes can be merged to improve data processing efficiency.

[0104] As an optional approach, the attribute prediction network structure can include fully connected layers and multi-classification layers. The fully connected layers can not only extract attribute features of the target region but also perform dimensionality reduction to adapt the features for subsequent multi-classification processing. Optionally, the multi-classification layer can be implemented using multiple sigmoid functions. When using the attribute prediction network to obtain attribute features between different target regions, the network input is the local visual features of a target region, and the output is one or more of a set of predefined attribute features.

[0105] As an example Figure 9 This diagram illustrates the principle of using an attribute prediction network to predict the attribute features of a target region. Figure 9As shown, the local visual features of a target "person" in an image frame (the local features shown in the figure) are input into the fully connected layer of the attribute prediction network. The features output from the fully connected layer are then input into a multi-classification layer for multi-classification, thereby obtaining the probabilities corresponding to various predefined attribute features of this person. Finally, based on the probabilities, some predefined attribute features are output. For example, the attribute feature "blue" with a probability of 0.92 and the attribute feature "high" with a probability of 0.54 are output. Specifically, in practical applications, for a target region, its attribute features can be one or more. For example, the attribute feature with the highest probability can be output as the attribute feature of the target region, or the attribute features with a probability higher than a set threshold or the top set number of attribute features with the highest probabilities can be output as the attribute features of the target region.

[0106] It should be noted that, as an optional solution in this application, the steps for obtaining the attribute features corresponding to each target region described above can be used or omitted. That is, when constructing the scene graph, local visual features, relational features, and attribute features can be used, or it can be constructed based solely on local visual features and relational features without using attribute features. When the step of obtaining attribute features is omitted, the nodes in the constructed scene graph represent the target features corresponding to the target regions, i.e., local attribute features. When constructing the scene graph based on relational features and attribute features, each node in the scene graph represents the target features and / or attributes corresponding to the target regions, and each edge represents the relationship between the targets corresponding to the nodes.

[0107] Still with Figure 7a The example shown is used for illustration, such as Figure 7a As shown, after extracting target features, attribute features, and relationship features between people and dogs from the image, a scene graph for that image can be constructed using a graph approach. Figure 7a In the scene diagram shown, each box represents a target feature or attribute feature of a target area. The lines connecting the boxes representing target features indicate the relationships between the targets, i.e., correspondence features. The elliptical boxes within the lines show the relationship features between the various target areas. The arrows (i.e., the direction of the connecting edges) indicate the subject-object relationship between the two. In the scene diagram shown in Figure 7, in the relationship between "human" and "dog," "human" is the subject and "dog" is the object. The direction of the line connecting "human" and "dog" indicates the relationship between the nodes representing target features and the nodes representing attribute features, representing the attribution relationship between them. Figure 7a The image shows a person wearing blue clothes. Wearing blue clothes is an attribute of the person, and the direction of the arrow indicates the attribution relationship, meaning that the attribute of wearing blue clothes belongs to the person. Figure 7aThe scene diagram clearly shows the various targets included in the diagram, their relative positions, attributes, and behavioral relationships.

[0108] In an optional embodiment of this application, if the multimedia data is video, and each image of the multimedia data is a number of frames selected from the video, if the target area in two adjacent frames contains the same target, then there is a time connection between the nodes (target nodes) corresponding to the same target in the scene graph of the two adjacent frames, that is, multiple connections also include time connections.

[0109] To better utilize temporal information, this alternative approach, for video, can also consider the temporal information between adjacent frames within the selected frames, adding this temporal information to the scene graph corresponding to each frame to obtain a spatiotemporal scene graph. Specifically, if the target regions in adjacent frames correspond to the same target, a temporal edge is added between the target nodes in the target regions containing the same target in the scene graphs of the adjacent frames. The scene graph after adding the temporal edge reflects both the spatial and temporal relationships between targets, and therefore can be called a spatiotemporal scene graph.

[0110] As an example, Figure 10 The diagram illustrates a temporal-spatial scene graph. Temporal edges are added to the scene graph corresponding to each frame. In the scene graphs corresponding to two adjacent frames, if the target categories of target regions in the scene graphs of the two adjacent frames are the same, a temporal edge is added between these two target regions. For example, Figure 10 In the scene graphs corresponding to the first and second frames shown, the targets such as people, ovens, and pizzas have the same target category in the two frames. Therefore, time-related edges can be added between the corresponding target regions in the scene graphs of these two frames, as shown in the figure, to add time-related edges between people, between ovens, and between pizzas in the scene graphs of the two frames.

[0111] Compared to spatial scene graphs, temporal-spatial scene graphs add the relationships between objects (i.e., targets) in the temporal dimension, which can better depict the spatiotemporal information of the video. In addition, temporal-spatial scene graphs can also include the action information of the targets corresponding to the temporal edges (which will be described in detail later) to improve the accuracy of action description.

[0112] In this optional scheme of the embodiment of the present application, the temporal information between frames is also considered. That is, when the scene graph is obtained, a spatial-temporal scene graph is obtained by incorporating temporal connections. In this scheme, by fusing temporal information, the correlation between images between frames is fully considered. Since the temporal connections are established between the same targets in adjacent frames, when extracting graph convolutional features based on the scene graph, the continuity information of the target in different images can be better learned. Based on making full use of the intra-frame information and inter-frame information of each frame, a better video description is obtained.

[0113] In an optional embodiment of this application, obtaining the graph convolutional features of the image based on the scene graph of the image includes:

[0114] The nodes and edges in the scene graph are encoded to obtain feature vectors with the same dimension as the target. Graph convolutional networks are then used to obtain graph convolutional features based on the obtained feature vectors.

[0115] It should be noted that in practical applications, if the dimensions of each feature in the acquired local visual features, attribute features, and relation features are the same, this step can be performed or not.

[0116] Specifically, when nodes in the constructed scene graph represent target features or attribute features corresponding to a target region, the obtained relational and attribute features of the target region can be encoded into feature vectors with the same dimension as the target. Then, a graph convolutional network is applied to the encoded feature vectors to learn the relationships between adjacent nodes and edges in the scene graph, thereby obtaining the graph convolutional features (i.e., graph convolutional features) of each node contained in the scene graph. The features learned based on the graph convolutional network are based on the graph structure (scene graph), so the graph convolutional features can include target features, attributes, and relational information.

[0117] When the nodes in the constructed scene graph only represent target features, the features after graph convolution can include target features and relational information. In this case, attribute features of each target region can be obtained from the extracted target region features without using an attribute prediction network.

[0118] As an example, a fully connected matrix can be used to encode all or some nodes (such as target nodes) and edges in a scene graph into feature vectors with the same dimension as the target dimension (a fixed dimension related to the feature dimension of the input vector of the subsequent decoder). For example, when the dimension of the relation features in the scene graph is 512 and the target dimension is 1024, a 512*1024 matrix can be applied to the relation features of the scene graph to obtain the same dimension as the target dimension of 1024.

[0119] After obtaining the feature vector with the same dimension as the target, the graph convolution feature of each node in the obtained feature vector can be obtained by using the graph convolution formula. The simplest unweighted graph convolution formula is shown in the following equation (1):

[0120]

[0121] Among them, v i N(v) represents the feature vector of node i in the scene graph, which is the target feature or attribute feature vector. i ) represents the set of nodes adjacent to node i in the scene graph (i.e., within the same frame of the image), v j Let W be the feature vector of node j adjacent to node i in the same frame image (i.e., the same scene graph). The adjacent nodes of node i generally do not include node i itself. W is the network weight parameter of the graph convolutional network to be learned. Let σ represent the graph convolution features of node i, where σ represents the nonlinear activation function.

[0122] In practical applications, since the importance of the relationships between different nodes in the scene graph varies in the image, that is, the importance of the relationships between different targets to the descriptive information of the final multimedia data varies, the edges in the scene graph can be weighted edges. Therefore, as another alternative, the following equation (2) can be used to obtain the graph convolution features of each node:

[0123]

[0124] Among them, v i N(v) i The meaning of ) is the same as that in formula (1), N(v i ) can include v i The text describes the graph convolutional network's weights and biases, where W and b represent the parameters to be learned. The final line, dir(v), is a direct representation of the graph convolutional network. i ,v j ) indicates the direction of the connection, where dir(v i ,v j There are two possible values, representing the change from v. i to v j (That is, the direction of the connection is from node i to node j), or from v j to v i , Therefore, there are two possible results, each with dir(v i ,v j ) corresponds to one result, label(v) i ,v j ) represents v i and vj The relationship between them can have different bias values ​​for different relationships, where σ represents the nonlinear activation function. This represents the graph convolution feature corresponding to node i.

[0125] As another alternative, equation (2) can be extended to the following form, as shown in equation (3) below. In this alternative, one weight can be used for the node itself, that is, the importance of different nodes is also different, and two other weights are used for adjacent nodes according to the subordinate relationship of the relationship, that is, two weights are used for the relationship features:

[0126]

[0127]

[0128] in, σ, v i v j N(v) i The meaning of ) is the same as that of the corresponding parameters mentioned above, and will not be repeated here; W s W (sub,obj) W (in,out) W a These are all parameter weight matrices that need to be learned; specifically, W s For v i The parameter weight matrix, W (sub,obj) The features of adjacent nodes, i.e., v j The parameter weight matrix, (sub, obj), represents the subordination relationship and has two possible values, representing v and obj respectively. i Is it the subject or the object of the relationship, such as... Figure 7a In the scene diagram shown, in the relationship between "human" and "dog," "human" is the subject and "dog" is the object. If v j It is the main body, v i If it is an object, then use W. obj (e.g. for) Figure 7a (Regarding the "dog" in the text, "human" is the subject; otherwise, use W.) sub W (in,out) The parameter weight matrix represents the relation, where (in, out) represents the direction of the edge, with two possible values: one indicating whether the edge is output from node i or input from node i. For example, for... Figure 7a The edges between "human" and "dog" shown in the diagram represent the output edges for the "human" node and the input edges for the "dog" node. If v i It is the main body, v j If it is an object, then use W. (in) Conversely, use W. (out) ;er This represents the feature vector indicating the relationship between two corresponding nodes. It can have different values ​​depending on the direction of the edge, or it can have the same value. This represents the relational feature vector corresponding to node i and node j. Depending on the direction of the connection, that is, the different subject-object relationships, this feature vector can be the same or different. For example, in the above example, the relationship between "person" and "dog" is "leading" for the person and "being led" for the dog. This represents the attention layer, specifically the attention parameter matrix, W. a This represents the weights of the attention layer.

[0129] In another alternative approach, for video, if the constructed scene graph is a spatial-temporal scene graph, for each frame, in addition to considering intra-frame information, inter-frame information can also be considered. That is, the graph convolutional features of a frame can be obtained based on the scene graph of that frame and the scene graphs of its neighboring frames. Specifically, all or part of the nodes and edges in the temporal-spatial scene graph can be encoded to obtain feature vectors with the same dimension as the target. Then, a graph convolutional network is used to train the obtained feature vectors to obtain graph convolutional features.

[0130] Optionally, graph convolution features can be obtained using the following expression (4):

[0131]

[0132] In this expression, the same parameter has the same meaning as that in expression (3). b (v i () represents the set of nodes of the same category as node i in the adjacent frames of the current frame image, that is, the set of the same target in adjacent frames, such as Figure 10 In the example shown, for the "person" node in the second frame image shown in the figure, if the node is v i Then, the set consisting of the "person" node in the first frame and / or the third frame shown in the figure is N. b (v i W (pre,aft) Indicates the relationship between the current frame to which node i belongs and N. b (v i The order of the frames to which node j belongs in the video, i.e., the parameter weight matrix W of the same target in adjacent frames. (pre,aft) There are two possible values, representing whether the current frame is the frame preceding or following the adjacent frame. In other words, it indicates whether the frame containing node i is the preceding or following frame in terms of temporal sequence, relative to the adjacent frames. If v j If it's the previous frame, then use W. (pre) Otherwise use W(aft) .

[0133] In an optional embodiment of this application, the method may further include the following when constructing a time-space scene graph:

[0134] Determine the action characteristics of the target corresponding to the time-connected edge;

[0135] At this point, for each frame of image, a time-space scene graph can be constructed based on the local visual features, attribute features (optional), relational features, and action features of the target corresponding to each target region in that frame of image.

[0136] In other words, action features of each target node corresponding to a temporal connection can be added to the scene graph. Optionally, object tracking methods can be used to identify common targets between adjacent frames. For such common targets, a pre-trained action classifier (action detector) can be used to identify the action category (also known as action relationship) of the target in the image, and the feature vector of the action category can be used as the action feature of the target.

[0137] like Figure 11 In the space-time scene diagram shown, the common targets in each frame are "person," "pizza," and "oven." The action corresponding to "person" is "open," meaning that in the space-time scene diagram, the temporal connection value of the common target in adjacent frames is "open." The action corresponding to "pizza" is "being held," and the action corresponding to "oven" is "being opened." Therefore, this scene diagram... Figure 10 Compared to the scene graph in the previous one, it can further include action information corresponding to common targets contained in adjacent frames. When generating description information based on this scene graph, more image detail information can be used, thereby further improving the accuracy of the generated description information.

[0138] Corresponding to this sustainable approach, the graph convolutional features of each node in the scene graph can also be calculated using the following formula:

[0139]

[0140] The explanation of the same parameters as in formula (4) above can be found in the preceding text, W. T The parameter weight matrix represents the action relationship (i.e., action category) of the same object in adjacent frames. This represents the action category (specifically, a feature vector representing the action category), i.e., the action features of the same target in adjacent frames, such as... Figure 11In the example shown, for the common target "person" in adjacent frames, the action relationship is "open"; for the scene graph of the first frame, the action category corresponding to the node "oven" in that frame (node ​​i at this time) and the node "oven" in the scene graph of the second frame (node ​​j in the adjacent frame at this time) is the feature vector of the action "opened"; W T It is also a weight matrix that needs to be learned, and different weight values ​​can be applied to different action categories. It is an attention parameter matrix, which can assign different weights to different targets, such as Figure 7b In the example shown, when updating the features of the "dog" node, the "human" node is more closely related to the "dog" node than the "skateboard" node, so the "human" node is given a higher weight.

[0141] To understand formula (5) more intuitively, it means that for different object (i.e., target node) features v i ,like Figure 11 The node "person" shown, after passing through a graph convolutional network, has updated features, namely graph convolutional features, including the feature v of the target itself. i And features of some connected objects of the target, such as ovens, pizzas, and features of the target node in adjacent frames.

[0142] The graph convolution feature extraction scheme provided in this application differs from existing graph convolution feature extraction schemes in that it adds attention (i.e., attention weights) when updating node features. In other words, when updating features, adjacent nodes can be assigned different weights. For example, in the above example, when updating the features of the "dog" node, the "person" node has a closer relationship with the "dog" node than the "skateboard" node, so the "person" node is given a higher weight. Another difference is that for two adjacent objects, different weight parameter matrices can be used to update features depending on the subject and object, and the time frame. For example, in the relationship between the "person" node and the "skateboard" node, "person" is the subject, and "skateboard" is the object. When updating the "person" node, the weight parameter matrix is ​​W. sub When updating the "microphone" node, the weight parameter matrix is ​​W. obj .

[0143] In practical applications, nodes representing the attribute features and target features of the same object in the scene graph can be merged. That is, nodes in the scene graph can represent both target features and attribute features. Through the above-mentioned alternative methods, the graph convolutional features of each node in the scene graph of each image can be obtained using a graph convolutional network. The convolutional features of an image are the convolutional features of all nodes contained in the scene graph of that image. Furthermore, nodes representing the attribute features and target features of the same object in the scene graph can also be merged. That is, nodes in the scene graph can contain both target nodes and attribute nodes. In this case, the graph convolutional features of each target node and each attribute node can be obtained. When obtaining the graph convolutional features of each feature node through the above alternative methods, if one or more parameters in the above expression do not exist for a certain node, pre-configured values ​​can be used, such as a zero vector or other pre-configured feature vectors.

[0144] In an optional embodiment of this application, if the feature information of the multimedia data includes at least two of local visual features, semantic features, spatiotemporal visual features, and global features, a textual description of the multimedia data is generated based on the extracted feature information, including:

[0145] Determine the weight of each feature;

[0146] Based on the weights of each feature information, a weighted processing is performed on each feature information;

[0147] Based on the weighted feature information, a text description of the multimedia data is generated.

[0148] In practical applications, since the importance of feature information of different categories may be different for different multimedia data (such as different videos and different images), different feature information of different categories can be assigned different weights, so that different features play different roles. This makes the solution of the present application embodiment adaptive for the generation of description information for different videos, that is, for different videos, multiple feature information can play different roles.

[0149] Optionally, when determining the weight of each feature information, a feature selection network can be used. By training the feature selection network, the network can select specific feature information used to generate the descriptive information of different multimedia data. In other words, for a given multimedia data, the feature selection network can determine the weights of different feature information.

[0150] As an example, Figure 12 The diagram illustrates the principle of a feature selection network, as shown below. Figure 12 As shown in the example, the feature information is the graph convolution feature V.GCN Spatiotemporal visual features V 3d and semantic features V SF In this example, the specific implementation of the feature selection network can be expressed by the following formula:

[0151]

[0152] Among them, a t E represents a set of weight values ​​output by the feature selection network at time t, i.e., the weights of each feature information at time t. 1:t-1 This represents the word embedding from time 1 to time t-1, which is the feature vector of the first t-1 words that have been decoded when the t-th word of the video description information is obtained. W 3d W GCN W SF W e This is the network's parameter weight matrix. Each parameter weight matrix transforms various features into the same dimension and then sums them. After passing through a non-linear layer (tanh), it is then... The parameter weight matrix is ​​transformed into a 3x1 vector, and then normalized using softmax. Each dimension represents the weight of a different feature, and they all sum to 1. Intuitively, this formula performs an attention operation on each feature to obtain the attention weights for each feature.

[0153] like Figure 12 In the example shown, the weights of spatiotemporal visual features, graph convolutional features, and semantic features are 0.3, 0.2, and 0.5, respectively.

[0154] It is understandable that time 1, time t-1, and time t are all relative time concepts, representing the relative decoding time of the first, (t-1), and t-th words in the video description information when the decoder decodes the output video description information. The weights of each feature information at each time other than time 1 can be obtained based on each feature information and the words already decoded before the current time.

[0155] The scheme provided in this application uses a variety of different features to represent video information. In addition to temporal-spatial scene graph features (i.e., graph convolutional features), it also uses spatiotemporal visual features and semantic features, meaning it can employ three types of features. Graph convolutional features focus more on the relationships and attributes between objects, spatiotemporal visual features focus more on temporal information, and semantic features focus more on the overall semantic information contained in the video. A feature selection gate is used to select different features based on different videos. The output of the feature selection gate is a set of weight values ​​'a'. t, representing the weights of different features. For example, some videos are longer, so temporal information is more important, and therefore the weight of spatiotemporal visual features is higher. Some videos are shorter, but contain more objects, so the relationships between objects and the attributes of objects are more important, and therefore the weight of graph convolution features is higher.

[0156] After obtaining the weights of each feature, as an option, in subsequent processing based on these features, one approach is to weight each feature using its respective weight and then use the weighted features for subsequent processing. Alternatively, one approach is to weight and fuse the features based on their respective weights to obtain fused features, and then use these fused features for subsequent processing. For example... Figure 12 In the example shown, the fused features (i.e., 0.3 * spatiotemporal visual features + 0.2 * graph convolutional features + 0.5 * semantic features as shown in the figure) can be used for subsequent processing. Alternatively, these weighted features can be processed separately to obtain the weighted features: 0.3 * spatiotemporal visual features, 0.2 * graph convolutional features, and 0.5 * semantic features. By adaptively assigning different weights to different features, different categories of features can play different roles in generating text descriptions, based on the characteristics of the multimedia data itself.

[0157] In an optional embodiment of this application, generating a textual description of the multimedia data based on the extracted feature information may include:

[0158] The obtained feature information is encoded using a self-attention-based encoder;

[0159] The encoded feature information is input into the decoder to generate a text description of the multimedia data;

[0160] Wherein, if the multimedia data is an image, the self-attention-based encoder is a self-attention-based intra-frame encoder; if the multimedia data is video, the self-attention-based encoder includes a self-attention-based intra-frame encoder and / or a self-attention-based inter-frame encoder.

[0161] In other words, a self-attention-based intra-frame encoder can be used to encode each obtained feature (if weighted, the weighted features; if weighted fusion, the weighted fused features) separately to obtain deeper and higher-level feature information. The encoded features are then input into the decoder to generate the corresponding text description. For video, a self-attention-based inter-frame encoder can also be used to encode the obtained video features, and the encoded features are then input into the decoder to generate the video's text description. The decoder can also be a self-attention-based decoder. For images, an attention-based intra-frame decoder can be used to better learn intra-frame information during decoding. For videos, a self-attention-based intra-frame decoder and / or a self-attention-based inter-frame decoder can be used to learn better intra-frame and / or inter-frame information during decoding, resulting in a more accurate video description.

[0162] Taking the text description of a video based on graph convolution features (which can be understood as spatiotemporal visual features and / or semantic features, as well as graph convolution features) as an example, a vector of graph convolution features can be obtained for each selected frame. These feature vectors are input into a self-attention-based decoder to learn and obtain inter-frame information between frames.

[0163] When generating text descriptions for videos based on graph convolutional features, the decoder outputs the possible words and their probabilities at each time step, based on the decoder input and the graph convolutional features. As an example, a self-attention-based decoder can be implemented using a transformer decoder.

[0164] In the process of generating text descriptions for a video, assuming that at the first time step, the decoder input is a global feature and a start symbol, and the output of the self-attention-based decoder is a set of probability values, each representing a word, the word with the highest probability is selected as the output at the first time step. At the second time step, the input is the global feature + the start symbol + the output from the first time step, and again, the word with the highest probability is selected as the output. At the third time step, the input is the global feature + the start symbol + the outputs from the first and second time steps, and so on, until at a certain time step, the word with the highest probability is the stop symbol, at which point the loop ends, thus obtaining a sentence sequence. In other words, the output of the self-attention-based decoder is the final sentence sequence describing the video.

[0165] For example, the transformer decoder can output words a and b that might be output at a first time step, along with the output probabilities of word a (e.g., 60%) and word b (e.g., 40%); words c, d, and e that might be output at a second time step, along with the output probabilities of word c (e.g., 60%), word d (e.g., 20%), and word e (e.g., 20%); and so on for subsequent time steps. In this case, according to an exemplary embodiment of this application, a video description sentence can be generated using a greedy decoding method, that is, by combining the words with the highest output probabilities that might be output at each time step in chronological order. However, this disclosure is not limited to this, and other decoding methods can also be used to generate video description sentences.

[0166] According to an exemplary embodiment of this application, a video description sentence can be obtained by combining the words with the highest output probability at each possible output time step in chronological order, until the probability of the output terminator is maximized. The self-attention-based decoder learns inter-frame information and has a structure based on a self-attention mechanism, including a multi-head attention layer, a layer normalization layer, and a feedforward network layer. Compared with decoders with RNN structures, the self-attention-based decoder has the advantages of faster training speed, fewer parameters, and easier learning of long-range correlations.

[0167] As an example Figure 13a The diagram shows a structural schematic of a self-attention-based encoder-decoder model provided in an embodiment of this application. Figure 13a As shown, the self-attention-based encoder-decoder model of this application consists of two parts: a self-attention-based encoder and a self-attention-based decoder. Optionally, this self-attention-based encoder-decoder can be implemented using a transformer encoder-decoder. In this example, the video feature information is still illustrated using graph convolution features. Figure 13a As shown, the self-attention-based encoder in this example consists of a multi-head attention layer, a feedforward network, and a layer normalization layer. The self-attention-based decoder can consist of a masked multi-head attention layer, a multi-head attention layer, a layer normalization layer, and a feedforward network layer.

[0168] Figure 13a The encoder shown, with its self-attention mechanism-based structure, can be a multi-block structure. Each block can have the same or different structure, and the blocks can be cascaded sequentially, meaning the output of the current block becomes the input of the next block. As an alternative, the encoder could contain, for example, six blocks with identical structures. Figure 13aThe diagram shows a block, each block mainly consists of two parts: a multi-head attention layer and a position-wise fully connected feedforward network. The feedforward network can be implemented by two linear prediction layers, with a ReLU activation operation between the two linear prediction layers. Each block's multi-head attention layer and feedforward network part can correspond to a layer normalization layer. Specifically, as shown... Figure 13a As shown, each block can be composed of a multi-head attention layer, a layer normalization layer, a feedforward network layer, and another layer normalization layer in sequence, and the blocks can be stacked to form the encoder. The encoder input is graph convolutional features.

[0169] When encoding graph convolutional features using a self-attention-based encoder, encoder embedding can be performed on the graph convolutional features to change the dimensionality of the feature information so that it is suitable for subsequent encoder processing. The feature information output after embedding is input into the encoder for encoding processing.

[0170] The following explanation uses the processing of the first block in the encoder as an example: The graph convolutional features are first processed by a multi-head attention layer for self-attention. The output of this layer is then fused with the encoder's embedded output (e.g., by addition), followed by layer normalization. The normalized result is processed by a feedforward network, then fused with the output of the previous normalization layer (e.g., by addition), and then normalized again to obtain the output of the first block. The output of the first block serves as the input for the second block, and the encoding processes are performed sequentially to obtain the encoder's output (i.e., the encoder output in Figure 13).

[0171] Figure 13aThe decoder structure of the self-attention mechanism shown can also be a multi-block structure. Each block can have the same or different structure, and these blocks can be cascaded sequentially. For example, it can contain six blocks with the same structure. Each block can mainly consist of three parts: masked multi-head attention, feature-corresponding multi-head self-attention, and a feedforward network. The multi-head attention and feedforward network parts in each block can correspond to a normalization layer. Specifically, each block's structure can be composed of a masked multi-head attention layer, a normalization layer, another multi-head attention layer, a normalization layer, a feedforward network layer, and another normalization layer. These blocks can be stacked to form the decoder. The decoder input in the figure is global feature information and word vectors. The feature vector of the target region extracted by the feature extraction network from each frame can be called local features or region features. Therefore, for a given frame, multiple region features of the target region can be obtained. After averaging these region features, the corresponding global features for that frame can be obtained. Alternatively, global features can be obtained through other methods (such as weighted processing). In addition, the start symbol and word vectors predicted during the iterative prediction process can be obtained (if it is the first prediction in the iterative prediction, only the start symbol is obtained; when training the model, all word vectors can be input). For the above decoder input (i.e., global features, start symbols, and word vectors predicted during the iterative prediction process), decoder embedding processing can be performed. The purpose is to change the dimensionality of the feature information to suit subsequent decoder processing. The global feature information, start symbols, and word vectors output after embedding processing can be input into the decoder for decoding.

[0172] The following explanation uses the processing of the first block as an example: The global feature information, start symbol, and word vector output after embedding are first processed by a masked multi-head attention layer. The processing result is fused with the decoder's embedded output (e.g., addition), and then subjected to layer normalization. The normalized output, along with the encoder output (if the encoder includes an inter-frame encoder, the encoder output is the inter-frame encoder output; if the encoder only has an intra-frame encoder, the encoder output can be obtained by fusing the outputs of each intra-frame encoder, such as concatenating the outputs of each intra-frame encoder), is processed by the multi-head attention layer. Then, it is fused with the output of the previous normalization layer (e.g., addition), and then subjected to layer normalization. The normalized output is processed by the feedforward network, then fused with the output of the previous normalization layer, and then subjected to layer normalization. The result is the output of the first block. The output of the first block is used as the input of the second block, and the decoding process is performed sequentially to obtain the decoder's output.

[0173] The decoder's output undergoes a linear transformation by a linear layer, followed by processing by a softmax layer. This softmax layer outputs the possible word vectors and their corresponding probabilities for the current iteration (i.e., the predicted word at this moment), such as words a and b, and their respective output probabilities. The decoder, linear layer, and softmax layer repeat this iterative prediction process until the probability of the output terminator is maximized. Based on the word vectors obtained in each iteration, the descriptive information corresponding to the video can be obtained.

[0174] It is understood that the examples above are illustrative using graph convolutional features as an example. In practical applications, in addition to graph convolutional features, spatiotemporal visual features and / or semantic features of the video can also be included. In this case, the encoding process described above can encode each feature information separately, and the decoder can decode the fused features. Alternatively, the feature selection network described above can be used to determine the weights of each encoded feature, and the fused features based on these weights can be used as the encoder's output. The decoder then obtains the text description of the video based on this fused feature. Alternatively, the weights can be input separately into different cross-attention layers of the decoder for processing, and the decoder obtains the text description of the video.

[0175] In an optional embodiment of this application, a textual description of the multimedia data is generated based on the extracted feature information, including:

[0176] The extracted feature information is input into multiple decoders respectively;

[0177] A textual description of the multimedia data is obtained based on the decoding results of each decoder.

[0178] To provide decoding capabilities and improve the representation of descriptive information, the scheme of this application embodiment can employ a decoder bank, including multiple decoders, to decode the encoded results separately, thereby enhancing the decoding capabilities of the decoders. The final textual description information is obtained based on the decoding results of each decoder; for example, the final output can be obtained by averaging the decoding results of each decoder. The decoder bank can contain two or more encoders. The types of decoders included in the decoder bank are not limited in this application embodiment; for example, it can include LSTM-based decoders, gated recurrent unit-based decoders, self-attention-based decoders, transformer decoders, etc. The final output result is obtained by averaging the outputs of each decoder.

[0179] The tests revealed that increasing the number of decoders in the decoder group from 2 onwards resulted in increasingly better performance. However, once the number exceeded 4, the performance improvement plateaued. Furthermore, increasing the number of decoders also increased system complexity. Therefore, in practical applications, a trade-off between performance and complexity must be made when selecting the number of decoders. Generally, 2 or 3 decoders are a good choice. For example, 2 decoders are suitable for on-device systems, while 3 or more decoders can be selected for cloud-based systems.

[0180] Regarding the selection of each decoder in the decoder group, many decoder groups can be pre-trained. Different decoder groups may contain different decoders, and the number of decoders in a decoder group may vary. In practical applications, the decoder group that performs best on the validation set or test set can be selected from among multiple decoder groups. When selecting, both the decoding efficiency and decoding performance of the decoder group can be considered.

[0181] When using multiple decoders for decoding, during the training of the decoder group, a consistency loss needs to be added to the output of each decoder to constrain it and prevent significant performance differences between different decoders in the decoder group, which could lead to the decoder group performing worse than a single decoder. Assume the decoder group has two decoders, with outputs of two probability distributions p1 and p2. The consistency loss can be defined as follows:

[0182] loss = D KL (p1||p2)

[0183] Among them, D KL This represents the KL divergence.

[0184] In the various optional solutions of this application's embodiments, attention-based neural networks can be used for encoding or decoding. This is because attention-based neural networks can simultaneously draw global dependencies between different input and target positions, thus enabling better learning of long-term dependencies. Furthermore, this type of neural network allows for more efficient parallel computation during data processing. In addition, in self-attention-based encoders or decoders, especially in self-attention-based decoders, multiple cross-attention layers can be stacked. Since self-attention-based neural networks are good at learning the correlation information between elements of the same feature vector, and cross-attention (with multi-head attention as its core) is good at learning the correlation information between different feature vectors, by adding cross-attention layers to a self-attention-based decoder, the decoder can learn both the correlation features between elements of the feature vector itself and the correlation features between different feature vectors, thereby better handling various types of features and obtaining better descriptive information.

[0185] As an example, Figure 13b The diagram illustrates a self-attention-based decoder structure provided in an embodiment of this application. In this example, the encoder portion may include a spatiotemporal visual encoder (spatiotemporal feature extraction network) and a semantic encoder (semantic prediction network). These two encoders can include the encoder outputs of spatiotemporal visual features and semantic features. When employing… Figure 13b During decoding, the output of the semantic encoder is input to the semantic cross-attention layer, and the output of the spatiotemporal visual feature encoder is input to the spatiotemporal visual cross-attention layer. The masked self-attention layer ensures that information from later time steps is not received during decoding, masking the input of later time steps. The input of this masked self-attention layer can then be represented as... Figure 13a The decoder input shown includes start symbols, etc., and may be a feature vector processed by the decoder embedding layer.

[0186] In an optional embodiment of this application, generating a text description of multimedia data based on the extracted feature information includes:

[0187] Obtain the length information of the text description to be generated;

[0188] Based on the length information and the extracted feature information, a text description of the video is generated.

[0189] To address the problem of existing technologies being unable to generate video or image descriptions of varying lengths for users, the solution provided in this application generates text descriptions of appropriate lengths by obtaining the desired length information, thus meeting the needs of different application scenarios. The length information can be relative, such as "long" (e.g., more than 20 words), "moderate" (e.g., 10-20 words), or "short" (e.g., less than 10 words). The length information can be obtained from the user, such as by prompting the user whether they want a long or short description, allowing the user to provide the corresponding instruction; alternatively, it can be obtained through video analysis. In the case of real-time video capture, analysis can determine the current application scenario, allowing for the determination of different length information for different scenarios.

[0190] In this scheme, unlike existing technologies, the start symbol of the decoder can be a start symbol containing length information when training the decoder. For example, for each training sample, the start symbol can contain a start symbol that indicates that a longer description needs to be generated or a start symbol that indicates that a shorter description needs to be generated. Different start symbols correspond to different sample description annotation information. When training the decoder based on the training samples, the decoder can learn the mapping relationship between start symbols with different length information and description information of corresponding length. Thus, when decoding based on the trained decoder, the start symbol corresponding to the length information can be used as the start symbol for decoding based on the acquired length information, thereby generating video descriptions or image descriptions that meet the length requirements.

[0191] In other words, the solution in this embodiment replaces the "BOS" (Begin Of Sentence) in the existing method with length information, such as "short," "moderate," or "long," during training to control the length of the output description information. In actual training, different length identifiers can be used for different length information. Specifically, when training to output a short description, the start symbol is "short," a moderate description corresponds to a "moderate" start symbol, and a long description corresponds to a "long" start symbol. This way, during training, sentence length is associated with "short," "moderate," and "long." When used online, users can input "short," "moderate," or "long" to obtain description information of different lengths according to their different needs.

[0192] Based on the methods in the various optional embodiments of this application, it is possible to analyze in detail the intra-frame information of each frame or image in the video (such as the target, attributes, and relationships of the image, as well as the semantic information and spatiotemporal visual features of the video or image), and to fully utilize the information of the image to generate more accurate text descriptions. As can be seen from the above description, based on the video description information generation method provided by the embodiments of this application, in practical applications, various different specific implementation methods can be selected according to actual application needs.

[0193] Furthermore, the solution provided in this application, when extracting feature information from multimedia data, not only utilizes a feature extraction network to extract features from each region of the image, but also adds an encoder (i.e., a relationship prediction network) for learning the relationships between features of each region. This encoder can be implemented by a self-attention-based encoder (e.g., a transformer encoder), thereby improving the performance of obtaining video or image descriptive information by enhancing the performance of feature encoding. Moreover, when obtaining descriptive information, the embodiments of this application may not use a conventional RNN-structured decoder, but instead employ a self-attention-based decoder (e.g., a transformer decoder). Compared to conventional RNNs, self-attention-based decoders have the advantages of faster training speed, fewer parameters, and easier learning of long-range correlations.

[0194] The following uses video as an example to illustrate the method for generating description information of multimedia data provided in the embodiments of this application, in conjunction with several optional embodiments.

[0195] Example 1

[0196] Figure 14 The diagram shows a flowchart of a video description information generation method according to an optional embodiment of this application. Figure 14 As shown, the generation method may include the following steps:

[0197] Step S301: Select several frames from the video;

[0198] Step S302: Construct scene graphs for each of the several frames;

[0199] Step S303: For each frame, use a graph convolutional network based on the constructed scene graph to obtain the graph convolutional features of each frame;

[0200] Step S304: Generate a text description of the video based on the obtained graph convolution features.

[0201] Optionally, after obtaining the graph convolutional features of each frame, a textual description of the video can be obtained based on these graph convolutional features. For example, the obtained graph convolutional features can be input into a decoder, and a textual description of the given video can be obtained by decoding the obtained graph convolutional features. As an alternative approach, a self-attention-based decoder can be used to generate a textual description of the video based on the graph convolutional features of each frame in several frames; however, this application is not limited to this.

[0202] Example 2

[0203] Figure 15 The diagram shows a flowchart illustrating one possible method for generating video description information as given in this example. Figure 15 As shown, this optional implementation may include the following steps:

[0204] Step S1201: Select several frames from a given video, such as... Figure 17a 501 and Figure 17b 1001 in the middle;

[0205] Step S1202: For each frame in the selected set of frames, use a feature extraction network to obtain several target regions and the features (i.e., region features or local features) of each target region, such as... Figure 17a 502 and Figure 17b In step 1002, for each image frame, the Faster R-CNN algorithm can be used to extract each target region and its features in each frame.

[0206] Step S1203: Apply a relationship prediction network to the extracted regional features of each target region to obtain the relationship features between the target regions, such as... Figure 8 The example shown.

[0207] Step S1204: Construct a scene graph for each image frame based on the obtained relationship features between various target regions, such as... Figure 17a 503 in the middle.

[0208] Step S1205: Based on the nodes and edges in the scene graph of each frame, use a graph convolutional network to obtain the graph convolutional features of each frame, such as... Figure 17a 504 in the middle;

[0209] Step S1206: Generate video text descriptions based on graph convolution features. Optionally, a self-attention-based decoder can be used to learn inter-frame information of selected frames based on the obtained graph convolution features to generate text descriptions for the given video. For example, a vector of graph convolution features can be obtained for each selected frame. These feature vectors are input into the self-attention-based decoder to learn inter-frame information between frames, such as... Figure 17a As shown in 505, the vectors of graph convolution features of each frame can be input into the inter-frame transformer decoder, and the text description of the video can be obtained based on the decoder output: that is, "a person is putting pizza into the oven".

[0210] Example 3

[0211] Figure 16 The diagram shows a flowchart of one possible method for generating video description information given in this example. By comparison... Figure 15 and Figure 16 As can be seen, the first three steps in this example are the same as those in Example 2 above, and will not be repeated here. The difference from Example 1 is that this example adds step S1304, which is to apply an attribute prediction network to the extracted features of each target region to obtain the attribute features of each target region. For example, a relation prediction network can be trained based on the Visual Genome dataset, and then the attribute prediction network can be applied to obtain the attribute features of each target region. Figure 9 The example shown.

[0212] Accordingly, in step S1305, when constructing the scene graph for each frame, it can be constructed based on the obtained attribute features of each target region and the relationship features between each target region. Then, in step S1306, the graph convolution features of each frame are obtained based on the scene graph constructed based on the attribute features and relationship features.

[0213] It should be noted that the execution order of steps S1303 and S1304 can be interchanged or performed simultaneously.

[0214] After obtaining the graph convolutional features of each frame, a text description of the video can be generated based on the obtained graph convolutional features in step S1307. A self-attention-based decoder can be used to learn the inter-frame information of several selected frames to generate a text description of the given video.

[0215] For example, such as Figure 17a As shown in Figure 505, inter-frame information can be learned using an attention-based decoder (such as the inter-frame transformer decoder shown in the figure) based on graph convolutional features to generate video text descriptions.

[0216] For example, such as Figure 17b As shown in 1005, 1006, and 1007, the obtained graph convolutional features can be encoded, and the encoded features can be processed to obtain a feature vector with the same dimension as the target. Figure 17bThe cross connection between the scene graph and graph convolutional features indicates that the scene graph used can be a spatial-temporal scene graph, meaning that inter-frame information can be considered when constructing the scene graph; alternatively, a spatial scene graph can also be used. Specifically, a self-attention-based intra-frame encoder (such as...) can be used. Figure 17b The intra-frame transformer encoder shown performs encoding operations on the graph convolutional features of each frame. The role of this self-attention-based intra-frame encoder is to learn intra-frame information; that is, it can utilize the self-attention mechanism to further learn the relevant information between objects within the frame. Optionally, the structure of this self-attention-based intra-frame encoder is based on the self-attention mechanism, including a multi-head attention layer, a layer normalization layer, and a feedforward network layer. Next, the output from the self-attention-based intra-frame encoder is processed to obtain a feature vector with the same dimension as the target for each frame.

[0217] For example, suppose the output sequence of a self-attention-based intra-encoder has a dimension of T*C, where T represents the number of nodes in the scene graph and C is the feature dimension of the feature vector corresponding to each node. The self-attention-based intra-encoder uses the self-attention mechanism to learn information such as the relationships between the output sequences and outputs the learned sequence. Here, the length of the output sequence is the same as the length of the input sequence, which is T*C. Averaging this output yields a feature vector with a dimension of 1*C. Thus, each frame produces a 1*C feature vector.

[0218] For each selected frame, an encoded feature vector can be obtained. These encoded feature vectors are then input into a self-attention-based inter-frame encoder (e.g., ...). Figure 17b The inter-frame transformer encoder shown in the figure encodes again to obtain a feature vector with the same dimension as the target.

[0219] Then, based on the encoded features, a self-attention-based decoder (such as...) is used. Figure 17b The inter-frame transformer decoder shown learns inter-frame information to generate a textual description of a given video. Encoded features are input into a self-attention-based inter-frame decoder for learning, obtaining inter-frame information between frames. Textual descriptions of a given video are generated by learning the input features.

[0220] As another example, a self-attention-based intra-frame encoder can be used to encode the graph convolutional features of each of the obtained frames separately, and then the encoded features are input into the decoder to generate the text description of the video. Alternatively, a self-attention-based inter-frame encoder can be used to encode the graph convolutional features of each of the obtained frames, and then the encoded features are input into the decoder to generate the text description of the video. In other words, only a self-attention-based inter-frame encoder can be used to encode the graph convolutional features of each of the obtained frames, and then the encoded features are input into the decoder to generate the text description of the video. Figure 17b Operation 1005, or simply execute Figure 17b Operation 1006 in the middle can also be executed together. Figure 17b Operation 1005 in the middle and Figure 17b Operation 1006 in the middle.

[0221] After performing a series of processing steps on a given video, it is possible to create a textual description of the video, such as... Figure 17b As shown, a text description of "a man is putting pizza into the oven" can be generated from a selection of frames.

[0222] Example 4

[0223] Figure 18 The diagram shows a flowchart illustrating one possible method for generating video description information as given in this example. Figure 18 As shown, by comparison Figure 18 and Figure 16 As can be seen, the difference between this example and Example 3 lies in step S1505. In constructing the scene graph for each frame, this example also considers temporal information. Specifically, based on the obtained attribute features of each target region and the relationship features between each target region, a spatial scene graph for each image frame is constructed. Temporal information is added between the scene graphs for each frame to obtain a spatial-temporal scene graph, such as... Figure 10 The example shown.

[0224] It should be noted that this example can also be implemented based on Example 1, that is, step S1504 can be omitted and the attribute characteristics of the target area can be ignored.

[0225] After obtaining the spatial-temporal scene graphs for each frame, in step S1506, a graph convolutional network is used to obtain the graph convolutional features for each frame based on the nodes and edges in the spatial-temporal scene graphs for each frame. Then, in step S1507, a text description of the given video is generated based on the obtained graph convolutional features. For example, a self-attention-based encoder can be used to encode the graph convolutional features, and a general self-attention-based decoder can be used to learn inter-frame information to generate the text description of the video.

[0226] Example 5

[0227] Figure 19 The diagram shows a flowchart of one possible method for generating video description information given in this example. By comparison... Figure 19 and Figure 16 As can be seen, the differences between this example and Example 3 above are as follows:

[0228] Step S1602: For each of the selected frames, use a feature extraction network to obtain several target regions and the features of each target region (i.e., region features or local features), as well as the spatiotemporal visual features of the video.

[0229] Compared to the steps for extracting region features in the examples above, the features extracted in this step of this example may also include spatiotemporal visual features of the video. Optionally, such as Figure 20 As shown, spatiotemporal visual features can be obtained through a spatiotemporal feature extraction network.

[0230] Step S1603: Based on the selected frames, extract semantic features from the video using a semantic feature extraction network. For example... Figure 20 As shown, semantic features can be obtained through a semantic prediction network based on several frames.

[0231] Steps S1604 to S1607 correspond to the steps in the previous example of obtaining relation features, attribute features, constructing scene graphs (spatial scene graphs or space-time scene graphs), and extracting graph convolution features, and will not be repeated here.

[0232] Step S1608: Generate a text description of the video based on the graph convolutional features of each frame, the aforementioned spatiotemporal visual features, and semantic features. Specifically, multiple decoders can be used based on the spatiotemporal visual features, semantic features, and graph convolutional features. Figure 20 The decoder group shown in the figure learns inter-frame information to generate video text descriptions.

[0233] Each decoder can be based on self-attention or RNN. Specifically, spatiotemporal visual features, semantic features, and graph convolutional features can be input into the decoder group to learn inter-frame information, and then the results of each decoder are averaged to obtain the final decoding result. Textual descriptions of a given video are generated by learning the input features.

[0234] The video description information generation method provided in this application solves the problem of insufficient accuracy caused by neglecting intra-frame information in existing video description algorithms, and proposes an improved video description scheme. In implementation, the method provided in this application can obtain text descriptions of videos based on graph convolutional networks (GCNNs) and decode outputs based on self-attention structures. Specifically, after obtaining graph convolutional features, these features can be directly input into a self-attention-based decoder for decoding to output text descriptions of the given video. Alternatively, the obtained graph convolutional features can be encoded, and then the encoded features can be input into a self-attention-based codec for inter-frame encoding and decoding, thereby outputting text descriptions of the given video.

[0235] It should be noted that the accuracy of the descriptive information generated in this application can be further improved by employing self-attention-based intra-frame encoders and self-attention-based inter-frame encoders. Optionally, a self-attention-based intra-frame encoder can be used to encode the graph convolutional features of each frame separately, and the encoded features are then fused and input into the decoder to generate the text description of the video. Alternatively, a self-attention-based inter-frame encoder can be used to encode the graph convolutional features of each frame, and the encoded features are then input into the decoder to generate the text description of the video. In other words, intra-frame encoders and inter-frame encoders can be used selectively. The solution provided by the embodiments of this application can fully utilize intra-frame information and / or inter-frame information to generate a more accurate text description for a given video.

[0236] The following section uses an image as an example to illustrate some optional implementation methods for generating image description information.

[0237] Example 5

[0238] Figure 21 The figure shows a flowchart illustrating a method for generating image description information according to an embodiment of this application. As shown in the figure, the method includes:

[0239] Step S10: Extract the feature information corresponding to the image;

[0240] Step S20: Obtain the descriptive information corresponding to the image based on the extracted feature information.

[0241] The images can be acquired from local storage or a local database as needed, or received from an external data source (e.g., the Internet, a server, a database, etc.) via an input device or transmission medium.

[0242] Specifically, feature extraction networks can be used to extract the corresponding feature information of an image. As an alternative, a trained Faster R-CNN can be used to extract local features of each target region. For example, the feature vector obtained by average pooling the feature map of the Pool5 (Region of Interest) layer can be selected as the feature.

[0243] After obtaining the feature information of the image, a decoder can be used to obtain the corresponding descriptive information of the image based on the extracted feature information. The specific structure of the decoder is not limited in this embodiment. For example, the decoder can be implemented by a self-attention-based decoder (e.g., a transformer decoder). Specifically, the decoder can output the possible words and their output probabilities (which can be normalized probabilities) at each time step based on the extracted feature information and the input word vectors (which may include start symbols and word vectors predicted during iterative prediction). As an optional solution, the self-attention-based decoder can be composed of a masked multi-head attention layer, a multi-head attention layer, a layer normalization layer, and a feedforward network layer.

[0244] For example, the decoder can output words a and b that may be output at a first time step, along with the output probabilities of word a (e.g., 60%) and word b (e.g., 40%); words c, d, and e that may be output at a second time step, along with the output probabilities of word c (e.g., 60%), word d (e.g., 20%), and word e (e.g., 20%); and so on for subsequent time steps. In this case, according to an exemplary embodiment of this application, an image description sentence can be generated using a greedy decoding method, i.e., by combining the words with the highest output probabilities that may be output at each time step in chronological order. Alternatively, according to another exemplary embodiment of this application, an image description sentence can be generated using a Monte Carlo sampling method, i.e., by performing Monte Carlo sampling based on the output probabilities of the words that may be output at each time step.

[0245] Correspondingly, when generating image description information, the image description sentence can be obtained by combining the words with the highest output probability at each time step in chronological order, until the probability of the output terminator is the highest.

[0246] Optionally, step S10 may further include: obtaining global features corresponding to the image;

[0247] Accordingly, in step S20 above, obtaining the text description of the image can be based on the obtained local and global features.

[0248] To obtain a more accurate image description, after acquiring the local features of each target region of the image, the global features of the image can be obtained based on each local feature, thereby obtaining more accurate image description information based on local and global feature information.

[0249] Optionally, obtaining the global features corresponding to the image may include: obtaining the global features of the image based on the local features of the image, or extracting the global features based on the image through a feature extraction network.

[0250] Specifically, a feature extraction network can be used to extract local feature information corresponding to each candidate region of the image, and global feature information can be obtained based on the local feature information. Correspondingly, a decoder can be used to obtain descriptive information of the image based on the local and global feature information. As an alternative, global feature information can be obtained by averaging the local feature information corresponding to each candidate region of the image. Alternatively, global feature information can also be obtained by applying a feature extraction network (such as a CNN) to the image, for example, by extracting feature maps from each layer (i.e., each channel) of the image using ResNet and then performing average pooling.

[0251] In optional embodiments of this application, the aforementioned local features may include local image features and / or local attribute features, and the global features may include global image features and / or global attribute features; correspondingly, global image features corresponding to an image may be obtained based on local image features; and / or, global attribute features corresponding to an image may be obtained based on local attribute features.

[0252] In other words, the obtained local feature information can include not only local image features but also local text attribute information. Therefore, when extracting local features through a feature extraction network, the feature extraction network can also include an attribute prediction network. The attribute prediction network can be a multi-label classification network; optionally, it can be trained using a weakly supervised method like noisy-OR. In practical applications, attributes can be finely categorized into nouns, verbs, adjectives, relatively rare words, and topics. Each attribute is obtained based on a specific attribute prediction network (such as a Multiple Instance Learning (MIL) network). Finally, the various attribute features can be concatenated to obtain the final text attribute features.

[0253] Optionally, when the obtained local feature information includes local image feature information and local text attribute information, the obtained global feature information can also include global image feature information and global text attribute information. Similarly, global image feature information and global text attribute information can be obtained based on their respective local feature information; that is, global image feature information can be obtained based on local image feature information, and global text attribute information can be obtained based on local text attribute information. Alternatively, global image feature information and global text attribute information can be extracted from the image using a neural network. Of course, the local feature information and global feature information can also be configured such that one contains both image feature information and text attribute information, and the other contains only image feature information or text attribute information, depending on application requirements.

[0254] In an optional embodiment of this application, obtaining a textual description of the image based on local features and the global features includes:

[0255] Based on all the extracted local features, each local feature is encoded to obtain encoded local features. Based on the encoded local features and global features, a text description of the image is obtained.

[0256] In other words, after obtaining the local features of the image, an encoder can be used to encode each local feature based on all the extracted local feature information, resulting in encoded local feature information. The encoder can be used to learn the relationships between the local features of each target candidate region based on all the extracted local feature information.

[0257] As an optional approach, the encoder can be implemented using a self-attention-based encoder. This encoder encodes each of the extracted local feature information separately to obtain encoded local feature information. Correspondingly, when the decoder obtains the image description information based on the local and global feature information, the decoder can output the possible words and their probabilities (normalized probabilities) at each time step based on the input word vectors (which can be start symbols and word vectors predicted during iterative prediction) and the encoded local and global feature information. The image description sentence can be obtained by combining the words with the highest possible output probabilities at each time step in chronological order until the probability of the output terminator is maximized.

[0258] As an alternative, the self-attention-based encoder described above may include a multi-head attention layer, a layer normalization layer, and a feedforward network layer cascaded in sequence.

[0259] As an example Figure 22The figure shows a schematic diagram of an encoder-decoder structure provided in an embodiment of this application. As shown in the figure, in order to obtain the input information of the encoder, for the image to be processed (the image shown in the lower right corner of the figure), local features of each target region of the image can be extracted by a feature extraction network (Faster R-CNN shown in the figure). Specifically, the feature extraction network can divide the input image into multiple target candidate regions (i.e., target regions) and obtain a feature vector (i.e., local features) from each target candidate region, thereby obtaining multiple feature vectors, such as... Figure 22 The {V} shown j},{V j Each vector in} represents the local feature information of a target candidate region (the region corresponding to the rectangle marked in the lower left figure).

[0260] The local feature information extracted by the feature extraction network can be processed by region feature embedding. The purpose is to change the dimension of the feature information so that it is suitable for subsequent encoder processing. The local feature information output after embedding is input into the encoder for encoding processing.

[0261] Optional, Figure 22 The encoder shown can be a single or multi-block structure. In a multi-block structure, each block can have the same or different structures. As an example, suppose the encoder contains six blocks with identical structures, cascaded sequentially. Each block can consist of a multi-head attention layer and a position-wise fully connected feedforward network. Optionally, the feedforward network can be implemented using two linear prediction layers, with a ReLU activation operation between them. Each block's multi-head attention layer and feedforward network portion can correspond to a layer normalization layer, such as... Figure 22 As shown in the example, each block can be composed of a multi-head attention layer, a layer normalization layer, a feedforward network layer, and another layer normalization layer in sequence, and the blocks can be stacked to form an encoder.

[0262] The following explanation uses the processing of the first block as an example: Each local feature information is first processed by a multi-head attention layer using self-attention. Its output is then fused with the output of the region feature embedding (e.g., by addition), followed by layer normalization. The normalized result is processed by a feedforward network, then fused with the output of the previous normalization layer (e.g., by addition), and then normalized again to obtain the output of the first block. The output of the first block serves as the input to the second block, and the encoding processes are performed sequentially to obtain the encoder's output (i.e.,...). Figure 25 (encoder output in the image).

[0263] Optional, such as Figure 22As shown, the local feature information extracted by the feature extraction network can be further used to obtain global feature information. For example, the local feature information can be averaged to obtain global feature information. Figure 22 eigenvectors in In addition, the start symbol and the word vectors predicted during the iterative prediction process can be obtained (if it is the first prediction in the iterative prediction, only the start symbol is obtained). The start symbol and the word vectors predicted are as follows: Figure 22 As shown in w, when training the model, all word vectors corresponding to the sample can be input.

[0264] For global feature information The start symbol and the word vector w predicted during the iterative prediction process can be processed by the decoder embedding. The purpose is to change the dimension of the feature information so that it is suitable for subsequent decoder processing. The global feature information, start symbol and word vector output after embedding can be input into the decoder for decoding processing.

[0265] Optional, Figure 22 The decoder can be a single or multiple blocks, each with the same structure. For example, it can contain six blocks with the same structure. As an optional structure, each block can mainly consist of three parts: a masked multi-head attention layer, a feature-corresponding multi-head self-attention layer, and a feedforward network. The multi-head attention layer and the feedforward network in each block can correspond to a layer normalization layer. Therefore, the specific structure of each block can be composed of a masked multi-head attention layer, a layer normalization layer, another multi-head attention layer, a layer normalization layer, a feedforward network layer, and a layer normalization layer in sequence. These blocks can be stacked to form the decoder.

[0266] The following explanation uses the processing of the first block as an example: global feature information The start symbol and word vector w are first processed by a masked multi-head attention layer. The result is then fused with the decoder's embedded output (e.g., by addition), followed by layer normalization. The normalized output, along with the encoder's output, is processed by the multi-head attention layer, then fused with the output of the previous normalization layer (e.g., by addition), and then normalized again. The normalized output is processed by a feedforward network, then fused with the output of the previous normalization layer, and finally normalized again. The result is the output of the first block. The output of the first block is used as the input of the second block, and the decoding process continues to the next block to obtain the decoder's output.

[0267] The decoder's output is processed by a linear layer and then by a softmax layer, which outputs the possible word vectors and their corresponding probabilities for the current time step (i.e., this iteration's prediction), such as words a and b, and the output probabilities of word a and word b. The decoder, linear layer, and softmax layer repeat this iterative prediction process until the probability of the output terminator is maximized. Based on the word vectors obtained in each iteration, the descriptive information corresponding to the input image can be obtained.

[0268] When local features include both local image features and local text attribute information, the encoder used can include an image feature encoder and an attribute feature encoder. These two encoders are used to encode the local image feature information and the local text attribute information, respectively. Specifically, the image feature encoder encodes each extracted local image feature information to obtain encoded local image feature information, and the attribute feature encoder encodes each extracted local text attribute information to obtain encoded local text attribute information. Correspondingly, the decoder can then obtain the image's corresponding descriptive information based on the encoded local image feature information, the encoded local text attribute information, the global image feature information, and the global text attribute information.

[0269] As another example, Figure 23 The diagram shows a structural schematic of a codec provided in an embodiment of this application, which is composed of... Figure 22 and Figure 23 It can be seen that, Figure 23 The decoder structure can adopt the same... Figure 22 The encoder in this example has a similar structure to the decoder. It can contain an image feature encoder part and an attribute feature encoder part, which may have the same or different structures.

[0270] like Figure 23 As shown, for the image to be processed (the image shown in the lower left corner of the figure), local image feature vectors of multiple target candidate regions can be obtained through a feature extraction network (such as Faster R-CNN), such as... Figure 23 The {v shown j},{v j Each vector in} represents a target candidate region ( Figure 26 The local image feature vector of the region corresponding to the rectangle marked in the image processed by Faster R-CNN shown can also yield multiple local text attribute vectors, such as... Figure 23 The {a} shown j}, {aj Each vector in} represents a local text attribute vector of a target candidate region (the region corresponding to the rectangle marked in the lower left figure).

[0271] The extracted local image features can be processed by region image feature embedding. The local image features output after embedding are then input into an image feature encoder for encoding. Similarly, the extracted local text attribute information can be processed by region attribute feature embedding. The local text attribute information output after embedding is then input into an attribute feature encoder for encoding.

[0272] In this example, Figure 23 The structures of the image feature encoder and attribute feature encoder shown are both based on... Figure 22 The encoder structure shown is used as an example for illustration. For instance, each encoder can contain 6 identical blocks, and the structure of each block is as follows: Figure 23 As shown, the blocks can be stacked to form the encoder. The processing flow for each block can be found in the previous section. Figure 22 The description of the encoder block structure in the text shows that the difference between image feature encoders and attribute feature encoders lies only in that the input of the image feature encoder is the feature after region image feature embedding processing of local image feature information, while the input of the attribute feature encoder is the feature after region attribute feature embedding processing of local text attributes. After the encoding process, the output result of the encoder is obtained (i.e., Figure 23 Image feature encoder output and attribute feature encoder output (in the image feature encoder output).

[0273] Furthermore, the global image feature information can be obtained by averaging the feature information of each local image, such as... Figure 23 eigenvectors in The global text attribute information can be obtained by averaging the local text attribute information, such as... Figure 23 eigenvectors in In addition, the start symbol and the word vectors predicted during the iterative prediction process can be obtained (if it is the first prediction in the iterative prediction, only the start symbol is obtained). The start symbol and the word vectors predicted are as follows: Figure 23 As shown in w, when training the model, all word vectors of the sample can be input.

[0274] For global image feature information Global text attribute information The start symbol and the word vector w predicted during the iterative prediction process can be processed by the decoder embedding. The purpose is to change the dimension of the feature information so that it is suitable for subsequent decoder processing. The global image feature information, global text attribute information, start symbol and word vector output after embedding can be input into the decoder for decoding processing.

[0275] As an optional structure, Figure 23 The decoder structure shown can be a single or multi-block structure. In a multi-block structure, each block can have the same or different structures. For example, it can contain six blocks with the same structure. Each block can consist of four parts: a masked multi-head attention layer, a multi-head self-attention layer corresponding to image features, a multi-head self-attention layer corresponding to attribute features, and a feedforward network. The multi-head attention and feedforward network parts in each block can correspond to a normalization layer. Specifically, as shown... Figure 23 As shown, the structure of each block can be composed of a masked multi-head attention layer, a layer normalization layer, a multi-head attention layer, a layer normalization layer, a multi-head attention layer, a layer normalization layer, a forward network layer, and a layer normalization layer in sequence. The blocks can be stacked to obtain the decoder.

[0276] The following explanation uses the processing of the first part as an example: Regarding global image feature information... Global text attribute information The start symbol and the word vector w predicted during iterative prediction are first processed by a masked multi-head attention layer. The processing result is fused with the decoder's embedded output (e.g., by addition). Then, layer normalization is performed. The normalized output, along with the output of the image feature encoder, is processed by the multi-head attention layer, then fused with the output of the previous normalization layer (e.g., by addition). This process is repeated, and the normalized output, along with the output of the attribute feature encoder, is processed by the multi-head attention layer, then fused with the output of the previous normalization layer (e.g., by addition). This process is repeated, and the normalized output is processed by the feedforward network, then fused with the output of the previous normalization layer, and finally processed by a layer normalization layer. The result is the output of the first block. The output of the first block is used as the input of the second block, and the decoding process is performed sequentially to obtain the decoder's output.

[0277] The decoder's output is processed by a linear layer, then by a softmax layer, which outputs the possible word vectors and their corresponding probabilities for the current iteration (i.e., the word vectors for each iteration), such as the output probabilities of words a and b. The decoder, linear layer, and softmax layer repeat this iterative prediction process until the probability of the output terminator is maximized. Based on the word vectors obtained in each iteration, the descriptive information corresponding to the input image can be obtained.

[0278] As can be seen from the descriptions of the image description information generation methods in the above optional embodiments, this generation method can be specifically implemented through an image description model. That is, an image can be input into the image description model, and a textual description of the image can be obtained based on the model's output. The specific neural network structure of the image description model is not limited in this embodiment; however, it may include, but is not limited to, models based on… Figure 22 or Figure 23 The diagram shows the encoder-decoder network structure based on self-attention and the decoder based on self-attention.

[0279] It is understood that the solutions in the above examples are merely illustrative examples of some optional methods of this application and are not intended to limit the solutions of this application. In addition, the above examples are applicable to the generation of image description information and also to the generation of video description information. The only difference from the generation of video description information is that when generating image description information, since there is only one image, there is no need to consider the information between frames, i.e., between adjacent images, such as the aforementioned temporal connections, inter-frame encoders, etc.

[0280] As can be seen from the above description of the methods for generating multimedia data description information provided in the various optional embodiments of this application, the generation of multimedia data description information can be specifically implemented through a multimedia data description model. Specifically, for video, this multimedia data description model is a video description model; for images, it is an image description model. The video description model and the image description model can be different models, or they can be the same model. That is, it can be a model suitable for generating both image and video descriptions. Optionally, the video description model can be an RNN-based model or a model based on other network structures, such as a Transformer-based model. In practical applications, the specific structure of the model can be set according to actual needs, and this application embodiment does not impose any restrictions on this.

[0281] For a video for which video description information needs to be obtained, the video itself or several frames selected from the video can be input into a video description model. Based on the input to the video description model, a textual description of the video can be obtained. The specific model structure of the video description model is not limited in this embodiment; the original video description model can be trained using video samples, and the trained video description model can be used to generate video description information.

[0282] Specifically, in an optional embodiment of this application, the textual description of the multimedia data is obtained through a multimedia data description model, which is trained in the following manner:

[0283] Obtain training samples, wherein the training samples contain multimedia data of the first sample with descriptive annotations;

[0284] The initial description model is trained based on the first sample of multimedia data until the model loss function converges. The trained description model is then used as the multimedia data description model.

[0285] It is understandable that for video description models, sample multimedia data refers to sample videos, and description annotations refer to the video descriptions. For image description models, sample multimedia data refers to sample images, and description annotations refer to the sample image descriptions. The specific form of the model loss function can be configured according to actual needs. For example, loss functions commonly used in training video or image description models can be selected. During training, the value of the model loss function represents the difference between the predicted multimedia data description information and the description annotation information, or whether the predicted description information meets other preset termination conditions. Through continuous training, the model can make the predicted multimedia information approximate the description annotation information, or meet other preset conditions.

[0286] In order to improve the accuracy of the descriptive information in the generated multimedia data, Figure 24 The figure illustrates a training method for a multimedia data description model provided in an optional embodiment of this application. As shown in the figure, the training samples in this method also include second sample multimedia data without descriptive labels. The model loss function includes a first loss function and a second loss function. When training the initial description model based on the first sample multimedia data, the method may include the following steps S201 to S203.

[0287] Step S201: Train the preset description model based on the first sample multimedia data to obtain the value of the first loss function, and train the description model based on the second sample multimedia data to obtain the value of the second loss function;

[0288] Specifically, in this application embodiment, a preset video description model can be trained using both first sample multimedia data with descriptive annotations and second sample multimedia data without descriptive annotations.

[0289] The sources of the first and second sample multimedia data are not limited in this embodiment. Taking video as an example, the original video description corresponding to the first sample video data can be manually annotated by a technician, such as... Figure 25 The video shown can be labeled with a description such as "a child is cleaning the floor" by a technician. The second sample video data can be any video without a description, such as a video obtained from a video website or a video shot by the user. The specific forms of the first and second loss functions are not limited in this embodiment and can be configured according to actual application needs.

[0290] Step S202: Based on the values ​​of the first loss function and the second loss function, obtain the value of the final loss function;

[0291] Optionally, different loss functions can each have their own weights, allowing for varying importance of different loss functions during training. For instance, since the first multimedia data sample has original descriptive annotations while the second sample multimedia data sample does not, the annotation information (original descriptive annotations) of the first sample multimedia data is highly accurate, and the weight of the first loss function can be greater than that of the second loss function. When different loss functions have their own weights, the final loss function of the multimedia data description model can be determined based on the respective weights of each loss function. For example, the final loss function could be a weighted sum of all loss functions.

[0292] In other words, the steps to obtain the value of the model loss function (also known as the final loss function) based on the values ​​of the first and second loss functions can include:

[0293] Based on the preset weights of the first loss function, the value of the corresponding target first loss function is obtained, and based on the preset weights of the second loss function, the value of the corresponding target second loss function is obtained;

[0294] The sum of the values ​​of the first target loss function and the second target loss function is used as the value of the final loss function.

[0295] Specifically, the value of the final loss function can be calculated using the following formula:

[0296] min θ J = J label (θ)+∈J unlabel(θ)

[0297] Where ∈ is a hyperparameter. In this example, J label (θ) is the first loss function, J unlabel (θ) represents the second loss function. The weights of the first loss function can be set to 1, and the weights of the second loss function are ∈. Thus, the product of the first loss function and its corresponding weights is the target first loss function, and the product of the second loss function and its corresponding weights is the target second loss function. The sum of the target first loss function and the target second loss function is the final loss function.

[0298] Step S203: Train the description model based on the value of the final loss function until the final loss function converges, and obtain the trained multimedia data description model.

[0299] Specifically, after obtaining the final loss function of the video description model, the model parameters are updated based on the final loss function until the final loss function converges to its minimum, resulting in the trained video description model. The final loss function of the video description model is determined by a first loss function and a second loss function. The convergence of the final loss function to its minimum can mean that the function itself converges to its minimum, or that both the first and second loss functions converge to their minimum values ​​simultaneously.

[0300] In this embodiment, when a first sample multimedia data with descriptive annotations is received, a preset multimedia data description model is trained based on the first sample multimedia data and the descriptive annotations to obtain the value of a first loss function. When a second sample multimedia data without descriptive annotations is received, the description model is trained based on the second sample multimedia data to obtain the value of a second loss function. Then, based on the first and second loss functions, the value of the final loss function of the multimedia data description model is obtained, and the multimedia data description model is trained based on the final loss function until the final loss function converges based on the minimum value to obtain the trained multimedia data description model.

[0301] Through the above methods, this optional embodiment of the present application can train a multimedia data description model using sample video data with descriptive annotations, and it can also train a video description model simultaneously using sample multimedia data without descriptive annotations. This not only significantly reduces the manpower and time costs required to annotate descriptive information for sample multimedia data, especially when the number of sample multimedia data is large, but also improves the accuracy and precision of the multimedia data description model because of the increased number of sample multimedia data. Furthermore, the algorithm of the embodiments of this application is applicable to different models, such as the aforementioned RNN-based model or Transformer-based model; this method is a universal training method.

[0302] In an optional embodiment of this application, step S201 above, obtaining the value of the first loss function based on the first sample multimedia data, may include:

[0303] Input the first sample multimedia data into the video description model to obtain the predicted target description information;

[0304] The value of the first loss function is obtained based on the target description information and the corresponding description annotation.

[0305] The value of the first loss function represents the difference between the target description information obtained based on the model output and the corresponding labeled description information.

[0306] As an example, Figure 26 The diagram illustrates the principle of a training method for a multimedia data description model provided in this application. The example uses video as a case study. The labeled data shown in the diagram corresponds to the video data in the first sample video data, and the unlabeled data shows the video data in the second sample video data. The following is a detailed explanation... Figure 26 This training method is explained.

[0307] like Figure 26 As shown, specifically, labeled video data V can be input into the video description model M. The video description model analyzes and processes the video data to generate the corresponding target video description. Then, based on the original video description in the first sample video data (corresponding to...), Figure 26 The value of the first loss function can be calculated from the annotation y) and the target video description. In this example, the first loss function can be the cross-entropy loss function, as shown in formula (1):

[0308]

[0309] Among them, J label(θ) represents the cross-entropy loss, θ represents the model parameters of the video description model, t represents the current time, T represents the maximum time, and y t y represents the truth value at the current moment. 1:t-1 Let V represent the truth value from time 1 to time t-1, and let p represent the video. θ This represents the probability that the output word is true. Specifically, (p θ (y t |y 1:t-1 ,V) represents the probability that the word predicted by the model at the current time step is the corresponding labeled word. The meaning of this loss function is to maximize the probability that the output at the current time step is also a correct word when the input at the current time step is the correct word from all previous time steps.

[0310] For example, using video description models to... Figure 25 The video shown is analyzed and processed. Assuming the current time t=2, the word y0 output at the initial time t=0 is "a", and the word y1 output at time t=1 is "child". Then, when the word y1 "child" output at the current time t=2 and the word output at time t=1 is the correct word, the probability of outputting "is" for y2 is maximized.

[0311] Assuming that the video description model is used to describe... Figure 25 The video description obtained by analyzing and processing the video shown is "a child is sweeping the floor". Then, the video description model is trained based on "a child is sweeping the floor" and the original video description "a child is cleaning the floor".

[0312] In practical applications, embodiments of this application can pre-set a dictionary, and the words output at each time step are determined from this dictionary. The word y0 output at the initial time t=0 can be determined based on the start symbol of the video. For example, for Figure 25 In the video shown, the word y0 "one" output at time t=0 is determined based on the video's start character. Of course, in practical applications, other methods can be used to determine the first word of the video description, and this application embodiment does not limit this.

[0313] In an optional embodiment of this application, training the description model based on the second sample multimedia data to obtain the value of the second loss function includes:

[0314] Perform at least one data augmentation on the second sample multimedia data to obtain the third sample multimedia data;

[0315] Input the second sample multimedia data into the description model to obtain at least one multimedia description;

[0316] Based on the second and third sample multimedia data, the scores for each multimedia description are determined.

[0317] The value of the second loss function is obtained based on the scores of each multimedia description.

[0318] In other words, when training the model based on the second sample multimedia data without descriptive annotations, the multimedia data can be augmented. Based on the augmented third sample multimedia data and the second sample video data, the scores of each descriptive information obtained by the model based on the second sample multimedia data are determined, and the value of the second loss function is obtained based on each score.

[0319] Optionally, for example, the second sample multimedia data can be input into the description model to obtain first description information and second description information. Based on the second sample multimedia data and the third sample multimedia data, the first score value of the first description information and the second score value of the second description information can be determined. Based on the first score value and the second score value, the value of the second loss function can be obtained.

[0320] In this embodiment of the application, when training a multimedia data description model using sample multimedia data without descriptive annotations, the second sample multimedia data is augmented K times (K≥1) to obtain the third sample multimedia data. The description model is then trained based on the second and third sample multimedia data. Since the second and third sample multimedia data are the same or similar, their descriptive information should also be the same or similar. This allows the calculation of the value of the second loss function, which is then used to train the description model, thereby further improving the accuracy and precision of the description model.

[0321] In an optional embodiment of this application, the second sample multimedia data is input into the multimedia data description model to obtain corresponding first description information and second description information, which may specifically include:

[0322] The second sample multimedia data is input into the description model, and the first description information is determined based on the output of the description model using a greedy algorithm.

[0323] The second sample of multimedia data is input into the description model, and the video description is determined based on probability sampling of the output of the description model.

[0324] Greedy algorithms (also known as greedy search algorithms) refer to algorithms that always make the best choice at the moment when solving a problem. In other words, they do not consider the overall optimal solution, but rather make a locally optimal solution in some sense.

[0325] Optionally, taking video as an example, the first description information mentioned above is the first video description, and the second description information is the second video description. The first video description c is obtained based on a greedy algorithm. g The formula can be shown in formula (2):

[0326] c g ={c g (1), c g (2), ..., c g Formula (2);

[0327] Among them, c g (t)(t=1,2,......,T) represents the word output at time t, which is optional. g (f) = (argmax) y∈Y (p θ (c g，1:t-1 ,V))), where V represents the second sample video data, c g,1:t-1 This represents the sequence of words output from the initial time to time t-1, that is, all the words output before the current time. At this point, c... g (t) means selecting the word with the highest probability at the current time t as the word to be output at the current time. The output probability of each candidate word at the current time is determined based on the words output at previous times and the video V. The final output word at this time is the word with the highest probability among all candidate words.

[0328] After obtaining the words output at each time step, sorting the words output at each time step according to the output order will yield the first video description c. g .

[0329] Regarding the second video description information, probability sampling refers to sampling in which every unit in the survey population sample has an equal chance of being selected. Also known as random sampling or probability sampling, probability sampling is based on probability theory and random principles to select a sample. It ensures that every unit in the population has a pre-known non-zero probability of being selected. The probability of a population unit being selected can be specified through sample design and implemented through some form of randomization, although a random sample generally will not be completely identical to the population.

[0330] Optionally, the second video description c is obtained based on probability sampling. s The formula can be shown in formula (3):

[0331] c s ={c s (1), c s (2), ..., c s (T)} Formula (3);

[0332] Among them, c s (t)(t=1,2,......,T) represents the word output at time t, which is optional. s (t)=(multinomial y∈Y (p θ (c s，1：t-1 ,V))), where, c s，1：t-1 This represents the sequence of words output from the initial time to time t-1, that is, all the words output before the current time. V represents the second sample video data. At this time, c s (t) means sampling according to the output probability of each word at the current time and taking the sampling result as the output. The output probability of each candidate word at the current time is determined based on the output words and video V at previous times. The final output word at this time is obtained by randomly sampling each output probability.

[0333] Still with Figure 26 Taking the example in the text as an illustration, for the second sample video data, that is... Figure 26 The unlabeled data shown can be augmented K times to obtain third sample video data (such as...). Figure 26 The data enhancement method can be random removal of several frames from the video, or transformation processing such as rotation and cropping of each frame in the video, or other data enhancement methods. In practical applications, any data enhancement method for video data is applicable to the embodiments of this application, and the embodiments of this application do not limit it. The second sample video data is input into the video description model M to obtain the corresponding first video description c. g Second video description c s For example, the first video description c can be obtained through the above formula (2). g The first video description c can be obtained through the above formula (3). s

[0334] Specifically, for example, using video description models to describe... Figure 25 Analyzing videos with identical content but lacking the original descriptive information, we can obtain the words corresponding to the five time points (output results): "one (corresponding to c)". g (1)) "Child (corresponding to c)" g (2)) "Currently (corresponding to c)" g (3)) "Clean (corresponding to c)" g (4)) "Cleaning (corresponding to c)" g (4)) "Sorting (corresponding to c)" g (4)) "Ground (corresponding to c)"g (5))” multiple words, among which, c g (4) The output probabilities of the three candidate words at time t are 0.5 (clean), 0.2 (clean), and 0.3 (tidy). Without considering the output probabilities of other words, based on the greedy algorithm, we choose "clean" as the word with the highest output probability, c. g (4) The output word at time (4), therefore, the final c generated based on the greedy algorithm g The image depicts "a child cleaning the floor."

[0335] For example, continuing from the previous example, c s (4) At any given moment, three candidate words are determined: "sweep", "tidy up", and "clean". The output probability of "sweep" is 0.2, that of "tidy up" is 0.3, and that of "clean" is 0.5. Without considering the output probabilities of other words, three video descriptions can be generated: "A child is sweeping the floor," with an output probability of 0.2; "A child is tidying the floor," with an output probability of 0.3; and "A child is cleaning the floor," with an output probability of 0.5. Therefore, the final c generated based on probability sampling... s It can be any one of the three video descriptions.

[0336] In other words, in response to the above situation, the second sample video data is output into the video description model to obtain the output result. Assuming that the output result is generated 10 times based on the greedy algorithm, then all 10 video descriptions may be "a child is cleaning the floor". Assuming that the output result is generated 10 times based on probability sampling, then the video descriptions that may be obtained 2 times are "a child is sweeping the floor", 3 times are "a child is tidying up the floor", and 5 times are "a child is cleaning the floor".

[0337] In an optional embodiment of this application, a first score value for the first video information and a second score value for the second descriptive information are obtained based on the second sample multimedia data and the third sample multimedia data. Specifically, this may include:

[0338] The first descriptive information is input into the description model along with the second sample multimedia data and the third sample multimedia data, respectively, to obtain the first output probability distribution of the second sample multimedia data and the second output probability distribution of the third sample multimedia data, and the first score of the first descriptive information is obtained based on the first output probability distribution and the second output probability distribution.

[0339] The second descriptive information is input into the description model along with the second sample multimedia data and the third sample multimedia data, respectively, to obtain the third output probability distribution of the second sample multimedia data and the fourth output probability distribution of the third sample multimedia data. Based on the third output probability distribution and the fourth output probability distribution, the second score of the second descriptive information is obtained.

[0340] Specifically, taking video as an example, for the first video description, the first video description is used as the ground truth and input into the video description model along with the second sample video data to obtain the first output probability distribution at each time step of the first video description. Simultaneously, the first video description is used as the ground truth and input into the video description model along with the third sample video data to obtain the second output probability distribution at each time step of the first video description. Then, the KL divergence between the first and second output probability distributions can be calculated, and the first score r can be obtained based on this KL divergence. g .

[0341] As an alternative, the first score r of the first video description can be obtained by multiplying the temporal weights by the KL divergence and then taking the negative of the result. g Specifically, as shown in formula (4):

[0342]

[0343] Among them, W t =T / t is the temporal weight, which assigns higher weights to words at the beginning of the first video description and lower weights to words at the end, to balance the problem of error accumulation. Since the enhanced video V′ can include K data points, K r ​​values ​​can be obtained. g At this point, we can base our decisions on these k r values. g The final first score value r′ is obtained. g For example, it is possible to have K r g The first score r′ is obtained by averaging. g Of course, it can also be obtained in other ways, such as weighted average, assigning different weights to the video V′ obtained by different enhancement processing methods.

[0344] Similarly, for the second video description, the second video description is used as the ground truth and input into the video description model along with the second sample video data to obtain the third output probability distribution of the second video description at each time step. Simultaneously, the second video description is also used as the ground truth and input into the video description model along with the third sample video data to obtain the fourth output probability distribution of the second video description at each time step. Then, the KL divergence between the third and fourth output probability distributions is calculated, and this KL divergence is multiplied by the temporal weights. The negative of this multiplication result is then used to obtain the second component value r of the second video description. s Specifically, as shown in formula (5):

[0345]

[0346] Since V′ includes K data points, we can use methods such as applying K r to... s The first score r′ is obtained by averaging. s .

[0347] In obtaining r′ g and r′ s Then, the second loss function of the video description model can be calculated using both methods.

[0348] like Figure 26 In the example shown, after obtaining the first video description c g After that, c g The second sample video data V is input into the video description model M. Based on the model output, the third output probability distribution of the second video description at each time step is obtained. For each enhanced video V′ ( Figure 21 The diagram shows a V′, which can be used to convert c g By combining the video V′ with the video description model M, we obtain the fourth output probability distribution for each moment of the second video description corresponding to each V′. The third output probability distribution and the r corresponding to each fourth output probability distribution can be calculated using the above formula (3). g r′ can be obtained by averaging or other methods. g (c in the figure) g (The KL divergence of a given value). Using the same calculation principle, r′ can be obtained based on formula (4). s (The KL divergence of s in the figure).

[0349] In an optional embodiment of this application, the step of obtaining the value of the second loss function based on the first score and the second score may include:

[0350] The difference between the first score and the second score will be used as the reward value.

[0351] The second loss function describing the model is obtained based on the reward and the second descriptive information.

[0352] Specifically, the second loss function can be the policy gradient loss function, as shown in Equation (6):

[0353]

[0354] Among them, (r′ s -r′ g The score is the reward value, which is the difference between the first score and the second score. To calculate the gradient of θ, after obtaining the second loss function, the descriptive model is trained using the policy gradient. From the above description, we can see that if the sampled words are more correct, the KL divergence between the third and fourth output probability distributions will be smaller, the reward will be larger, and the model will have a higher probability of outputting that word after the update. Conversely, if the sampled words are less correct, the KL divergence between the third and fourth output probability distributions will be larger, the reward will be smaller, and the model will have a lower probability of outputting that word after the update.

[0355] Taking video as an example, such as Figure 26 The example shown can be based on r′ g (c in the figure) g The corresponding KL divergence and r′ s (c in the figure) s The reward value (shown in the figure) is obtained by taking the difference between the corresponding KL divergence values. Based on this reward, the policy gradient loss value can be calculated using the above formula (6), thereby obtaining the value of the first loss function (i.e., Figure 26 The values ​​of the cross-entropy loss and the second loss function shown in the figure) Figure 26 The value of the policy gradient loss shown in the figure is used to obtain the value of the final loss function.

[0356] Furthermore, as described above, based on the second sample multimedia data, only one descriptive information can be generated. In this case, the value of the second loss function can be obtained based solely on this descriptive information. Taking the above formula (6) as an example, r′ in formula (6) g Removing it is also feasible, so formula (6) can be rewritten as follows:

[0357]

[0358] In other words, the corresponding score can be obtained based solely on the second descriptive information (such as the second score in the example above), and the value of the second loss function can be obtained based on the second descriptive information and the score.

[0359] Currently, in commonly used datasets for video or image description, the descriptive annotations for videos or images are generally limited. For example, a training sample image typically has only five descriptive annotations, which is usually insufficient to fully express the information in the image. To improve the diversity of training sample descriptions, this application also provides a method for obtaining multimedia data descriptions. Based on this method, data augmentation can be performed on the descriptive annotations of sample multimedia data to obtain enhanced descriptive information, thereby increasing the number of descriptions in the sample data. This allows for the training of a better multimedia data description model based on sample data with enhanced descriptive information.

[0360] Accordingly, in an optional embodiment of this application, the description annotation of the first sample multimedia data may include at least one original description annotation of the first sample multimedia data, and enhanced description annotations corresponding to each original description annotation.

[0361] Figure 27 The figure shows a flowchart illustrating a method for obtaining multimedia data description according to an embodiment of this application. As shown in the figure, the method may include:

[0362] Step S2501: Obtain at least one original descriptive annotation corresponding to the multimedia data;

[0363] For the aforementioned first sample multimedia data, this means obtaining the original descriptive annotations of the first sample multimedia data.

[0364] The multimedia data can be sample data from a training image database or training video database obtained as needed from local storage or a local database, or it can be training samples from a training image database or training video database received from an external data source via an input device or transmission medium. Taking images as an example, a training image can include a predetermined number of N image description annotations, where N can be a positive integer not less than 1. For example, the images in this scheme can be training images from training image databases commonly used in the field of image description (e.g., the MS-COCO dataset). Images in commonly used training image databases typically have 5 image description annotations, and the 5 image description annotations for the same training image are different from each other but have similar semantics.

[0365] Step S2502: Based on the original description annotations corresponding to the multimedia data, generate enhanced description information corresponding to each original description annotation, i.e., enhanced description information.

[0366] Specifically, a generator can be used to generate enhanced descriptive information corresponding to each original descriptive annotation based on the multimedia data. The generator can also generate descriptive sentences with similar semantics but different from the original descriptive annotations. In other words, when a sentence with an original descriptive annotation is input into the generator, the generator can generate a descriptive sentence with similar semantics but different from the original descriptive annotation.

[0367] The process of generating sentences is a temporal process. As an optional method, a greedy decoding method can usually be used to generate sentences. That is, at the first time step, the input word vector is the start symbol, and the output is the first word with the highest predicted output probability. At the second time step, the input is the start symbol and the output from the first time step, and the output is the second word with the highest predicted output probability. This process continues until the output word is the end symbol.

[0368] The specific network structure of the generator is not limited in this embodiment. As an optional solution, the generator can be implemented using a self-attention-based encoder and a self-attention-based decoder.

[0369] As two examples Figure 28a and Figure 28b The diagram illustrates a network structure of a generator according to an embodiment of this application. This generator may consist of a self-attention-based encoder (e.g., a transformer encoder) and a self-attention-based decoder (e.g., a transformer decoder), as shown below. Figure 28a and Figure 28b As shown, the encoder in the generator can consist of a multi-head attention layer, a layer normalization layer, and a feedforward network layer, used to encode the original input descriptive annotations. The decoder in the generator can consist of a masked multi-head attention layer, a multi-head attention layer, a layer normalization layer, and a feedforward network layer, used to decode the encoded image or video descriptive annotations to obtain enhanced image or video descriptive information. For Figure 28a and Figure 28b For a detailed description of the structure of each part of the encoder and decoder shown, please refer to the previous description of Figure 13. Figure 22 or Figure 23 The corresponding descriptions of the encoders and decoders shown are provided.

[0370] It should be noted that the network structure of the generator in the embodiments of this application may include, but is not limited to, the structure shown in the above examples, and may also be implemented using any other available encoders and decoders.

[0371] To ensure the accuracy of the enhanced image description information generated by the generator, the generator also needs to be trained. As an alternative, the generator can be trained in the following way:

[0372] Obtain the training database, which includes multiple training sample data, each training sample data includes N original description labels, where N is a positive integer not less than 1;

[0373] The generator is trained based on the original description annotations of multiple training sample data in the training database. The generator is used to generate descriptive information with similar semantics that is different from the original description annotations.

[0374] In addition, to improve the generator's performance, as an optional approach, a discriminator can be introduced during generator training, employing adversarial training. Specifically, the steps for training the generator can include:

[0375] The generator and discriminator are trained alternately until the similarity value of the description information generated by the generator for each original description label of each training sample data meets the preset condition. Specifically, the discriminator can be used to determine the probability that the description information generated by the generator is the real original description label.

[0376] The specific network structure of the discriminator can be configured according to actual needs. Understandably, the discriminator also needs to be trained. When the trained discriminator determines that the description sentence generated by the generator is a true original description annotation with a relatively high probability (e.g., exceeding a predetermined threshold), it means that the description sentence generated by the generator is close to the description of the real sample (i.e., the true original description annotation), and it can "fool" the trained discriminator. In this case, such a description sentence can be used as enhanced descriptive information during the training process to improve sample diversity.

[0377] Specifically, during training, the generator and discriminator can be trained alternately until the similarity value of the description information generated by the generator for each original description annotation of each training sample data meets the preset conditions. The specific calculation method for the similarity value is not limited in this embodiment.

[0378] As an optional approach, the similarity value can be the CIDEr value. CIDEr is a commonly used evaluation metric for assessing descriptive performance; a higher CIDEr value indicates a greater similarity between the generated descriptive sentence and the original descriptive annotation. The CIDEr metric treats each sentence as a "document," representing it as a TF-IDF vector. It calculates the cosine similarity between the reference (i.e., the real) descriptive sentence and the generated descriptive sentence, using this as a score to generate the CIDEr value. Therefore, according to an exemplary embodiment of the present invention, the CIDEr value of the generated descriptive sentence (image descriptive sentence or video descriptive sentence) can be calculated based on the similarity between the descriptive sentence and the N original descriptive annotations in the training sample data to which the original descriptive annotations used to generate the descriptive sentence belong. For example, taking images as an example, when the CIDEr value of the image descriptive sentence generated by the generator for each original image descriptive annotation of each training image meets a preset condition, it indicates that the generator has been able to generate image descriptive sentences that are semantically very similar to the real image descriptive annotations, thus completing the training of the generator and discriminator.

[0379] The preset conditions may include that the similarity value of the generated descriptive information for each original descriptive annotation of each training sample data reaches a predetermined threshold, or that the average similarity value of the generated image descriptive information for each original descriptive annotation of each training sample data reaches a predetermined threshold. These preset conditions may be system defaults or can be set by the user according to their needs or experience. Furthermore, the completion of the generator and discriminator training can also be determined based on the user's needs or experience. For example, when the generator and discriminator have been trained to a certain extent, the user can use a batch of training sample data to test the generator and observe whether the generator's output is satisfactory. When the generator's output is satisfactory, the training of the generator and discriminator is complete.

[0380] In an optional embodiment of this application, the step of alternately training the generator and the discriminator includes:

[0381] The discriminator is trained with the generator parameters fixed.

[0382] The generator is trained with the parameters of the trained discriminator fixed.

[0383] In other words, when training the generator and discriminator alternately, the discriminator can be trained first with the generator's parameters fixed, and then the generator can be trained with the trained discriminator's parameters fixed. This training process can be repeated for different training datasets. For example, taking images as an example, for the first training image set, the discriminator and generator are trained once based on their original parameters (i.e., network structure parameters). Then, for the second training image set, the discriminator and generator are trained again based on the parameters trained on the first training image set. Then, for the third training image set, the discriminator and generator are trained again based on the parameters trained on the second training image set. This continues until the similarity values ​​of the image descriptions generated by the generator for each training image's image description annotation meet a preset condition, or the user's test results of the generator are satisfactory.

[0384] In an optional embodiment of this application, the discriminator can be trained using the following operations:

[0385] Perform the following operations for each original descriptive annotation of each training sample data (in this operation, the number of original descriptive annotations of the sample data is greater than 1, i.e., N is greater than 1):

[0386] The original descriptive label is paired with the other N-1 original descriptive labels in the training sample data to generate N-1 first pairs. The original descriptive label is then input into a generator to generate descriptive information, which is then paired with the original descriptive label to generate second pairs. Based on the N-1 first and second pairs, the discriminator can be trained using the cross-entropy loss function, where the output of the discriminator is the probability value that each pair is two true original descriptive labels.

[0387] In other words, based on an original descriptive label (referred to as the baseline label), N-1 reference pairs, or sample pairs, are obtained by configuring it with N-1 other original descriptive labels. Based on the baseline label, the generator can generate N-1 descriptive information. The baseline label is then paired with the N-1 generated descriptive information to obtain predicted pairs. The loss function is calculated based on each corresponding sample pair and predicted pair. The network parameters of the discriminator are adjusted based on the loss function value until a preset condition is met. For example, for images, the discriminator outputs a probability value greater than a set threshold for each predicted pair to be two real image descriptive labels (i.e., reference pairs).

[0388] In an optional embodiment of this application (which may be referred to as Scheme 1), the step of training the generator while keeping the parameters of the trained discriminator fixed may include:

[0389] Perform the following operations for each original description annotation of each training sample data:

[0390] The original description annotation is input into the generator, which then generates description information using a greedy decoding method. Based on the generated description information, the following operations are performed:

[0391] The similarity value between the generated description information and the N original description labels of the corresponding training image is calculated. The generated description information is then paired with the original description labels to generate a second pair. The probability value of the second pair being two original image description labels is obtained using a trained discriminator. The calculated similarity value and the obtained probability value are weighted and summed to obtain a reward. The parameters of the generator are then adjusted based on the obtained reward.

[0392] In another optional embodiment of this application (which may be referred to as Scheme Two), the step of training the generator while keeping the parameters of the trained discriminator fixed may include:

[0393] Perform the following operations for each original description annotation of each training sample data:

[0394] The original description annotation is input into the generator, and the generator generates the first description information according to the greedy decoding method.

[0395] The original description annotation is input into the generator, and the generator is used to generate a second description information according to the Monte Carlo sampling method;

[0396] For the generated first description information, perform the following operations:

[0397] The first similarity value corresponding to the first description information generated is calculated based on the first description information generated and the N original description labels of the corresponding training image. The first description information generated is paired with the original description label to generate a second pairing. The first probability value of the second pairing being two real original description labels is obtained using a trained discriminator. The first similarity value calculated and the first probability value obtained are weighted and summed to obtain the first reward.

[0398] For the generated second description information, perform the following operations:

[0399] The second similarity value corresponding to the generated second description information and the N original description labels of the corresponding training image is calculated. The generated second description information is paired with the original description label to generate a second pairing. The trained discriminator is used to obtain the second probability value that the second pairing is two real original description labels. The calculated second similarity value and the obtained second probability value are weighted and summed to obtain the second reward.

[0400] The generator parameters are adjusted based on the difference between the first reward and the second reward, which is used as the final reward.

[0401] In practical applications, due to the discrete nature of text data, it is difficult to backpropagate the discriminator's gradient to the generator. To address this issue, as an alternative approach, a policy gradient method can be employed. Based on the descriptive sentences generated by the generator, a reward is calculated. A higher reward indicates a better generated descriptive sentence, and the generator's parameters are adjusted more accordingly. Traditional methods only award the discriminator's output as a reward, while the reward in the alternative embodiment provided in this application can include two parts: the discriminator's output and a similarity value (e.g., the CIDEr value). The weighted sum of these two parts yields the final reward. By using more diverse data to determine the reward for generator parameter adjustment, the generator can effectively learn more information and generate enhanced descriptive information that is more similar to but different from the original descriptive annotations. This allows for better enhanced image descriptive information based on a trained generator, providing a more and better data foundation for training multimedia data description models based on sample data containing this enhanced descriptive information.

[0402] To better understand and illustrate the training scheme for the generator provided in the embodiments of this application, the following will be combined with... Figure 28a and Figure 28b The training scheme described above will be explained in further detail. This example uses images as multimedia data; however, it is understood that the principle behind this example also applies to video.

[0403] As an optional example, corresponding to the first scheme above, such as Figure 28a As shown, when training the generator, the following operations can be performed for each image description annotation of each training image: the image description annotation (X1:T as shown in the figure) is input into the generator, and the generator generates image description information according to the greedy decoding method; for the generated image description information (such as Y1:T), the following operations are performed: the similarity value corresponding to the generated image description information is calculated based on the generated image description information and the N image description annotations of the corresponding training image; the generated image description information is paired with the image description annotation to generate a second pair (e.g., X1:T, Y1:T); the probability value of the second pair being two real image description annotations is obtained using the trained discriminator; the calculated similarity value and the obtained probability value are weighted and summed to obtain a reward; and the parameters of the generator are adjusted according to the obtained reward.

[0404] Specifically, such as Figure 28a As shown, the image description sentence y, generated using a generator and obtained through a greedy decoding method, is analyzed. b The calculated CIDEr value and the discriminator's analysis of the image description sentence y generated by the generator using a greedy decoding method. b The obtained probability values ​​are weighted and summed to obtain the reward r(y). b The formula for weighted summation is as follows:

[0405]

[0406] Where r (corresponding to) Figure 28a r(y b )) represents the reward, and τ represents the weighting coefficient. C represents the probability value output by the discriminator, where C is the CIDEr value (corresponding to...). Figure 28a (CIDEr score in the data).

[0407] Figure 28aIn the example shown, the discriminator structure can be based on a CNN, such as including convolutional layers and max-pooling layers as shown in the figure. Specifically, for each pair of image description labels and the corresponding generated image description information, i.e., each second pair, the pair can be processed by embedding to obtain the corresponding feature vector. Based on this feature vector, convolution processing can be performed using convolutional layers with various convolution processing parameters. After pooling each convolution result by a max-pooling layer, the pooling results are concatenated, and the probability value corresponding to each second pair is predicted based on the concatenated vector.

[0408] In addition, according to the other alternative scheme, namely Scheme 2 above, in order to further improve the effect, a self-evaluation mechanism can also be used to train the generator. That is, the reward difference between the image description sentence obtained by Monte Carlo sampling and the image description sentence obtained by greedy decoding is used as the final reward, and the parameters of the generator are adjusted according to the final reward obtained.

[0409] like Figure 28b In the example shown, the image description sentence y, generated using a generator and obtained through a greedy decoding method, is analyzed. b The calculated CIDEr value and the discriminator's analysis of the image description sentence y generated by the generator using a greedy decoding method. b The obtained probability values ​​are weighted and summed to obtain the reward r(y). b (This refers to the image description sentence y generated by the generator based on the Monte Carlo sampling method.) s The calculated CIDEr value and the discriminator's description of the image y generated by the generator based on the Monte Carlo sampling method. s The obtained probability values ​​are weighted and summed to obtain the reward r(y). s ). r(y s )-r(y b The generator's parameters are adjusted as the final reward. The image description sentence y, obtained according to the Monte Carlo sampling method, is used as the reward. s The calculated CIDEr value and the image description sentence y obtained according to the Monte Carlo sampling method. s Obtain the probability values ​​and perform a weighted summation to obtain the reward r(y). s For specific details, please refer to the above. Figure 28a We get r(y) b The description of ) can be based on the same principle, so it will not be repeated here.

[0410] In an optional embodiment of this application, to avoid duplicate information in the generated enhanced descriptive information, the method may further include:

[0411] When duplicates exist in the enhanced description information, the original description annotations corresponding to the duplicate enhanced description information are re-input into the generator, and the enhanced description information is regenerated by adjusting the bundle value based on the bundle search method.

[0412] Taking images as an example, once the generator and discriminator are trained, the trained generator can be used to generate enhanced image descriptions corresponding to each image description annotation, based on the image description annotations for each image. These enhanced image descriptions differ from the real image description annotations, but there may be overlaps between them.

[0413] To address the potential for redundancy among enhanced descriptive information, a beam search method can be employed as an alternative to regenerate the enhanced descriptive information. The generator produces enhanced descriptive information based on maximum probability, meaning it outputs the word with the highest predicted probability at each time step (equivalent to a beam value of 1). The beam search method can adjust the generator's output by changing the beam value (e.g., 2, 3, etc.). For example, when two identical enhanced descriptions exist, the ground truth description (i.e., the original description label) corresponding to one of the enhanced descriptions can be input into the generator, the beam value set to 2, and the generator can produce a different enhanced descriptive information. For instance, the generator can output the two words with the highest probabilities at the first time step, let's say {a} and {b}; at the next time step, it can output the two words with the highest probabilities based on the two words {a} and {b} from the first time step, let's say {a,c}, {a,d}, {b,e}, {b,f}; then, it can select the two words with the highest probabilities from these four sequences, let's say {a,c} and {b,e}; and so on for subsequent time steps. For example, when three identical enhanced descriptions exist, the original description tag corresponding to one of the enhanced descriptions can be input into the generator, the bundle value set to 2, and the generator can produce a different enhanced description. Alternatively, the original description corresponding to one of the enhanced descriptions can be input into the generator, the bundle value set to 3, and the generator can produce another different enhanced description. This process can be repeated. In this way, enhanced description sentences can be generated using different bundle sizes, changing the generated results and solving the duplication problem.

[0414] After obtaining the enhanced descriptive information using the method provided in the application embodiment, the original descriptive annotations and corresponding enhanced descriptive information of the multimedia data can be used as annotation information for the multimedia data. Based on multimedia data samples containing more annotation information, the initial multimedia data description model is trained to obtain a better description model. Specifically, taking an image as an example, the image description model can be trained in the following way:

[0415] Obtain training samples, each sample image in the training samples corresponds to labeled information, the labeled information includes at least one image description label of the sample image, and enhanced image description information corresponding to each image description label;

[0416] The initial image description model is trained based on each sample image until the preset training termination condition is met, and a trained image description model is obtained.

[0417] For each sample image, the enhanced image description information corresponding to that sample image is obtained through the image description acquisition method provided in any optional embodiment of this application. The specific network structure of the image description model is not limited in this embodiment; for example, it can be an image description model based on an encoder and decoder, or it can be based on... Figure 22 or Figure 23 The network image description model of the encoder-decoder shown is illustrated.

[0418] As an example Figure 29 The figure shows a flowchart illustrating a training method for an image description model provided in an embodiment of this application. In this example, the image description model is an encoder-decoder based model. As shown in the figure, the method may include the following steps:

[0419] Step S2701: For each training image (i.e. sample image) in the training image database, perform the first training on the encoder and decoder using the cross-entropy loss function;

[0420] Optionally, the training images in this step can be training images from a training image database. The training images may include a predetermined number of N image description annotations, where N can be a positive integer greater than or equal to 1. For example, the training images may be training images from a commonly used training image database in the field of image description (e.g., the MS-COCO dataset), which typically has 5 image description annotations.

[0421] Specifically, it can be based on training images and use reference Figure 24 or based on Figure 24 The various optional methods can be used to obtain the descriptive information corresponding to the training images, and can be used according to the reference. Figure 27 The method shown obtains augmented image description information corresponding to the training images, and based on the obtained description information, the image description annotations of the training images, and the augmented image description information, the encoder and decoder are trained using the cross-entropy loss function. For example, training can be performed using the formula for the cross-entropy loss function below.

[0422]

[0423] Among them, J xe(θ) represents the loss, θ represents the parameters of transformer encoder 302 and transformer decoder 303, t represents the current time, T represents the maximum time, and y t This represents the word output at the current moment, y. 1:t-1 The word representing the ground truth of the previous time step, I represents the current image, p θ This represents the probability that the output word is true. Here, the first image description sentence is composed of the words with the highest output probabilities at each time step. Therefore, the probability of outputting each word at each time step can be obtained from the first image description sentence. t Furthermore, ground-truth words for each time step can be obtained from each of the image description annotations and augmented image descriptions of the training images.

[0424] Step S2702: When the encoder and decoder are trained based on the first training pair, a second training can be performed on the encoder and decoder trained by the first training pair for each training image in the training image database, using policy gradient and / or self-evaluation mechanism.

[0425] Specifically, policy gradient is used because the optimization objective of cross-entropy loss and the metrics used for evaluation (such as the CIDEr value) are different. To solve this problem, policy gradient is used to directly optimize the CIDEr value.

[0426] The formula is as follows:

[0427]

[0428] Where J(θ) is the loss, θ represents the parameters of the encoder and decoder, E represents the expectation, and y s The sampled image description sentence, r(y) s ) represents the CIDEr value, i.e., the reward, y s ~p θ This represents the set of image description sentences sampled using existing network parameters.

[0429] The self-evaluation mechanism refers to setting the reward as the difference between the CIDEr value of the image description sentence obtained through Monte Carlo sampling and the CIDEr value obtained through greedy decoding. In other words, using greedy decoding to constrain the reward yields better results. The formula is as follows:

[0430]

[0431] in, Let y be the image description sentence obtained using a greedy decoding method. s The image description sentence obtained from Monte Carlo sampling, where r is the calculated CIDEr value. Let p be the gradient of the loss. θ (y s ) is the sample y s The probability corresponding to the time.

[0432] Optionally, when performing the second training described above, you can refer to Figure 24 The method or based on Figure 24 The method shown utilizes a greedy decoding method to obtain a first image description sentence, which is then used to determine the image description sentence based on a reference. Figure 24 The method or based on Figure 24 The method shown utilizes a second image description sentence obtained through Monte Carlo sampling, and employs policy gradients and a self-evaluation mechanism to perform a second training on the encoder and decoder trained in the first stage. Specifically, the CIDEr value of the first image description sentence can be calculated based on the similarity between the first image description sentence and N image description annotations of the corresponding training image; the CIDEr value of the second image description sentence can be calculated based on the similarity between the second image description sentence and the N image description annotations of the corresponding training image; the difference between the CIDEr values ​​of the first and second image description sentences is calculated to obtain a reward; and the parameters of the encoder and decoder trained in the first stage are adjusted based on the obtained reward.

[0433] It is understood that the training method for the image description model provided in this application embodiment is only an optional training method. As long as the enhanced image description obtained by adopting the enhanced image description acquisition scheme provided in this application embodiment is also used as the annotation information of the sample image, the quantity and diversity of the annotation data of the sample image are increased. Training the image description model based on the training data containing the sample image can effectively improve the model's performance. In addition, it will be clear to those skilled in the art that the above-described scheme applicable to images is also applicable to video processing, and the principle is the same.

[0434] Furthermore, to enhance the expressive power of the descriptive annotations on the training samples, in an optional embodiment of this application, generative adversarial networks can be used for descriptive data augmentation, and the augmented samples can be applied to the training descriptive model, thereby increasing sample diversity and improving the performance of the video or image descriptive model of this application.

[0435] The following two diagrams will be used to provide an overall explanation of the method for generating descriptive information of multimedia data provided in this application.

[0436] Taking video as an example, Figure 30The figure shows a flowchart of a method for generating video description information according to this application. As shown in the figure, for a given video, several frames of the video can be selected through a frame selection step. The region encoder shown in the figure is a region encoder used to extract the local visual features of each target region in each frame image of the selected several frames. Optionally, the region encoder may include a region feature extraction network, a relationship detector (i.e., a relationship prediction network), an attribute detector (i.e., an attribute prediction network), and may also include an action detector (i.e., an action classifier).

[0437] To obtain a well-trained video description model, before using the model with the codec structure shown in this example for video description, it can be trained using semi-supervised learning, i.e., semi-supervised training, as shown in Figure 30. Annotated videos (i.e., videos with description annotations) and unannotated videos (i.e., videos without description annotations) can be used for training. During training, for each video, several frames can be used. Unannotated videos can be augmented to obtain augmented videos, and at least one video description can be obtained from the unannotated videos. Based on the augmented videos, the scores of each video description are obtained, thus yielding the value of the second loss function. For annotated videos, the value of the first loss function can be obtained based on the target description information and corresponding annotation information output by the model. The total loss function of the model is then obtained based on the values ​​of the first and second loss functions, guiding the model training until the total loss function converges to its minimum value.

[0438] Furthermore, regarding the part showing the acquisition of enhanced descriptive information, in order to obtain a greater quantity and diversity of annotation information during model training, this part can generate enhanced descriptive information corresponding to the original descriptive annotations (i.e., the real descriptive information) of the labeled video through a generator. Both the original descriptive annotations and the enhanced descriptive annotations are used as annotation information for the sample video data during training, thereby increasing the quantity and diversity of annotation information. By using a greater quantity of annotation information and the descriptive information predicted by the model's decoder to guide model training, the stability of the model and the accuracy of the generated descriptive information can be further improved.

[0439] like Figure 30As shown in the example, when processing video based on a trained video description model, the region encoder can extract local visual features, relational features, and attribute features of each target region in each frame of the video. It can then construct a scene graph for each frame based on these extracted features. In this example, the scene graph can be a spatial-temporal scene graph incorporating temporal information. The updated features, i.e., graph convolutional features, can be obtained through a graph convolutional network. Correspondingly, for the decoder part of the model, this example can employ a self-attention-based intra-frame decoder and a self-attention-based inter-frame decoder. Furthermore, when decoding and generating descriptive information, it can also obtain information about the user's desired descriptive information to generate a more user-friendly textual description of the video. For example, while a user is driving, real-time video of the area in front of their line of sight can be captured. By analyzing this video, when a prompt is needed, such as when there is a potential hazard ahead, the descriptive information generated from the video can be analyzed to provide the user with a corresponding prompt, or the descriptive information can be played back to the user.

[0440] As another example, Figure 31 The diagram illustrates a flowchart of a method for generating video description information according to this application. The 3D visual feature encoder (i.e., spatiotemporal feature extraction network), region encoder, and semantic encoder (i.e., semantic prediction network) shown in the diagram are encoders used to extract local visual features (local features shown in the diagram), spatiotemporal visual features, and semantic features of each target region in each frame of the video, respectively. Based on the local visual features, a spatiotemporal scene map of each frame can be constructed. Then, graph convolutional features (updated local features shown in the diagram) can be obtained through a graph convolutional network. In this example, a 3D visual feature encoder can be used to extract spatiotemporal visual features from the video, and a semantic encoder can be used to extract semantic features. For the obtained spatiotemporal visual features, semantic features, and graph convolutional features, a feature selection network can be used to select features, i.e., determine the weights of each feature. Based on these weights, the features can be weighted and fused to obtain the fused features. A decoder group (the decoder shown in the figure, consisting of several decoders) decodes these features according to the fused features and the desired length of the descriptive information. Finally, the final descriptive information (the output descriptive information shown in the figure) can be obtained based on the results of each decoder. For example, the decoding results of each decoder can be averaged to obtain the final descriptive information. Optionally, the decoder group can include a self-attention-based intra-frame decoder, into which the desired length of the descriptive information can be input to control the length of the final generated descriptive information.

[0441] Similarly, to obtain a well-trained video description model, before using the model with the codec structure shown in this example for video description, it can be trained using semi-supervised learning, i.e., semi-supervised training. Adversarial training can be used in this case. For details on the training process, please refer to the previous sections on model training and the aforementioned adversarial training methods. Figure 30 The descriptions of the relevant parts are not repeated here.

[0442] Furthermore, as described above, in practical applications, when using... Figure 30 and Figure 31 After obtaining the feature information of the video in the encoding part shown, before decoding by the decoder, the extracted feature information can be encoded by an intra-frame or inter-frame encoder based on self-attention, and the encoded features are then input into the decoder.

[0443] Based on the same principle as the method for generating description information of multimedia data provided in the embodiments of this application, this application also provides an apparatus for generating description information of multimedia data, such as... Figure 32 As shown, the descriptive information generation device 100 may include a feature information extraction module 110 and a descriptive information generation module 120. Wherein:

[0444] The feature information extraction module 110 is used to extract feature information of the multimedia data to be processed, wherein the multimedia data includes video or images;

[0445] The description information generation module 120 is used to generate text descriptions of multimedia data based on the extracted feature information.

[0446] Optionally, the description information generation module 120 is specifically used to perform at least one of the following:

[0447] Extract local visual features of the target contained in each target region of each image in the multimedia data;

[0448] Extracting semantic features from multimedia data;

[0449] If the multimedia data is video, extract the spatiotemporal visual features of the multimedia data;

[0450] Extracting global visual features from multimedia data;

[0451] Extract the attribute features of the target contained in each target region of each image in the multimedia data;

[0452] Extract global attribute features for each image in the multimedia data.

[0453] Optionally, the aforementioned feature information includes the local visual features of the target contained in each target region of each image of the multimedia data, and the description information generation module 120 is specifically used for:

[0454] For each image, the relationship features between the targets are obtained based on the local visual features of each target in the image, and a scene graph of the image is constructed based on the local visual features and the relationship features between the targets. For each image, the graph convolution features of the image are obtained based on the scene graph of the image. Based on the graph convolution features of each image of the multimedia data, the text description of the multimedia data is obtained.

[0455] Optionally, the scene graph includes multiple nodes and multiple edges, where each node represents a local visual feature, and each edge includes features representing the relationship between two connected nodes.

[0456] Optionally, the aforementioned feature information includes the attribute features of the targets contained in each target region of each image in the multimedia data; when constructing the scene graph of the image, the description information generation module 120 is specifically used to: construct the scene graph of the image based on the local visual features of each target, the relationship features between each target, and the attribute features of each target, wherein each node in the scene graph represents the local visual features or attribute features of the target corresponding to the target region.

[0457] Optionally, if the multimedia data is video, and each image of the multimedia data is a number of frames selected from the video, if the target regions in two adjacent frames contain the same target, then the nodes corresponding to the target regions containing the same target in the scene graphs of the two adjacent frames have a temporal connection.

[0458] Optionally, when the description information generation module 120 obtains the graph convolutional features of the image based on the scene graph of the image, it is used to: encode the nodes and edges in the scene graph to obtain feature vectors with the same dimension as the target, and use a graph convolutional network to obtain graph convolutional features based on the obtained feature vectors.

[0459] Optionally, if the feature information of the multimedia data includes at least two of the following: local visual features, semantic features, spatiotemporal visual features, and global features, the description information generation module 120 can be used to:

[0460] Determine the weight of each feature; perform weighted processing on each feature based on its weight; and generate a text description of the multimedia data based on the weighted feature information.

[0461] Optionally, the description information generation module 120 can be used for:

[0462] The obtained feature information is encoded using a self-attention-based encoder; the encoded feature information is then input into a decoder to generate a text description of the multimedia data; wherein, if the multimedia data is an image, the self-attention-based encoder is a self-attention-based intra-frame encoder; if the multimedia data is video, the self-attention-based encoder includes a self-attention-based intra-frame encoder and / or a self-attention-based inter-frame encoder.

[0463] Optionally, the description information generation module 120 can be used to: input the extracted feature information into multiple decoders respectively, and obtain a text description of the multimedia data based on the decoding results of each decoder.

[0464] Optionally, the description information generation module 120 can be used to: obtain the length information of the text description to be generated, and generate the text description of the video based on the length information and the extracted feature information.

[0465] Optionally, the description information generation module 120 can specifically use a multimedia data description model to obtain a textual description of the multimedia data. The multimedia data description model is trained by a model training device, which may include:

[0466] The sample acquisition module is used to acquire training samples, wherein the training samples contain first sample multimedia data with descriptive annotations;

[0467] The model training module is used to train the initial description model based on the first sample multimedia data until the model loss function converges, and the trained description model is used as the multimedia data description model.

[0468] Optionally, the training samples also include second-sample multimedia data without descriptive annotations, and the model loss function includes a first loss function and a second loss function; the model training module can be used for:

[0469] The first loss function is obtained by training a pre-defined description model based on the first sample multimedia data, and the second loss function is obtained by training the description model based on the second sample multimedia data.

[0470] Based on the values ​​of the first and second loss functions, the value of the final loss function is obtained.

[0471] The model is trained based on the value of the final loss function until the final loss function converges.

[0472] Optionally, when the model training module trains the description model based on the second sample multimedia data to obtain the value of the second loss function, it can be specifically used for:

[0473] Perform at least one data augmentation on the second sample multimedia data to obtain the third sample multimedia data; input the second sample multimedia data into the description model to obtain at least one multimedia description; determine the score of each multimedia description based on the second sample multimedia data and the third sample multimedia data; obtain the value of the second loss function based on the score of each multimedia description.

[0474] Optionally, the descriptive annotations of the first sample multimedia data include at least one original descriptive annotation of the first sample multimedia data, and enhanced descriptive annotations corresponding to each original descriptive annotation; wherein, the enhanced descriptive annotations are obtained in the following way:

[0475] Based on the original descriptive annotations of the first sample multimedia data, enhanced image descriptive annotations are generated for each original descriptive annotation.

[0476] Based on the same principles as the methods and apparatus provided in the embodiments of this application, the embodiments of this application also provide an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the methods shown in any optional embodiments of this application when running the computer program.

[0477] This application also provides a computer-readable storage medium storing a computer program that, when run by a processor, can execute the methods shown in any optional embodiment of this application.

[0478] As an example, Figure 33 The diagram shows a structural schematic of an electronic device to which an embodiment of this application applies, such as... Figure 33 As shown, Figure 33 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of this electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0479] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0480] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 33 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0481] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0482] The memory 4003 stores application code that executes the scheme of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in any of the foregoing method embodiments.

[0483] It is understood that the methods and models (such as video description models, image description models, etc.) provided in the various optional embodiments of this application can run in any terminal (which can be a user terminal or a server, etc.) that needs to generate video description information or image description information. Optionally, the terminal may have the following characteristics:

[0484] (1) In terms of hardware, the device has a central processing unit, memory, input components, and output components. In other words, the device is often a microcomputer device with communication capabilities. In addition, it can have multiple input methods, such as keyboard, mouse, touch screen, microphone, and camera, and the input can be adjusted as needed. At the same time, the device often has multiple output methods, such as receiver, display screen, etc., which can also be adjusted as needed;

[0485] (2) In terms of software, the device must have an operating system, such as Windows Mobile, Symbian, Palm, Android, iOS, etc. At the same time, these operating systems are becoming increasingly open, and personalized applications developed based on these open operating system platforms are emerging in an endless stream, such as address books, calendars, notepads, calculators, and various games, which greatly meet the needs of personalized users;

[0486] (3) In terms of communication capabilities, the device has flexible access methods and high bandwidth communication performance, and can automatically adjust the selected communication method according to the selected service and the environment, thus facilitating user use. The device can support GSM (Global System for Mobile Communication), WCDMA (Wideband Code Division Multiple Access), CDMA2000 (Code Division Multiple Access), TDSCDMA (Time Division-Synchronous Code Division Multiple Access), Wi-Fi (Wireless-Fidelity), and WiMAX (Worldwide Interoperability for Microwave Access), thus adapting to multiple network standards and supporting not only voice services but also various wireless data services;

[0487] (4) In terms of functionality, the equipment places greater emphasis on humanization, personalization, and multi-functionality. With the development of computer technology, the equipment has shifted from a "device-centric" model to a "human-centric" model, integrating embedded computing, control technology, artificial intelligence technology, and biometric authentication technology, fully embodying the principle of human-centeredness. Due to the development of software technology, the equipment can be customized according to individual needs, becoming more personalized. At the same time, the equipment itself integrates numerous software and hardware components, making its functions increasingly powerful.

[0488] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0489] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of generating description information of multimedia data, characterized by, The method comprises: extracting feature information of multimedia data to be processed, wherein the multimedia data comprises video or image; generating a text description of the multimedia data based on the extracted feature information; wherein generating the text description of the multimedia data based on the extracted feature information comprises: determining a first application scenario of the multimedia data by analyzing the multimedia data; obtaining length information of the text description generated for the first application scenario, wherein the length information indicates at least one length of a plurality of lengths, and the plurality of lengths correspond to a plurality of application scenarios, and the plurality of application scenarios comprises the first application scenario; generating the text description based on the length information and the extracted feature information.

2. The method of claim 1, wherein, The extraction of the feature information of the multimedia data to be processed comprises at least one of the following: extracting local visual features of targets contained in each target region of each image in the multimedia data; extracting semantic features of the multimedia data; if the multimedia data is video, extracting spatio-temporal visual features of the multimedia data; extracting global visual features of the multimedia data; extracting attribute features of targets contained in each target region of each image in the multimedia data; extracting global attribute features of each image in the multimedia data.

3. The method of claim 2, wherein, The feature information comprises local visual features of targets contained in each target region of each image in the multimedia data, and the generation of the text description of the multimedia data based on the extracted feature information comprises: for each image, obtaining relationship features between targets according to the local visual features of the targets in the image, and constructing a scene graph of the image based on the local visual features and the relationship features between the targets; for each image, obtaining graph convolution features of the image according to the scene graph of the image; obtaining the text description of the multimedia data based on the graph convolution features of each image of the multimedia data.

4. The method of claim 3, wherein, The scene graph comprises a plurality of nodes and a plurality of edges, wherein one node represents the local visual features of one target, and each edge represents the relationship features between the connected two nodes.

5. The method of claim 3, wherein, The feature information comprises attribute features of targets contained in each target region of each image in the multimedia data. The construction of the scene graph of the image based on the local visual features of the targets and the relationship features between the targets comprises: constructing the scene graph of the image based on the local visual features of the targets, the relationship features between the targets, and the attribute features of the targets, wherein one node in the scene graph represents the local visual features or the attribute features of one target.

6. The method of claim 3, wherein, If the multimedia data is video, each image of the multimedia data is a frame selected from the video, and if the target regions of adjacent two frames contain the same target, the nodes corresponding to the same target in the scene graphs of the adjacent two frames have a time edge.

7. The method of claim 3, wherein, The obtaining of the graph convolution features of the image according to the scene graph of the image comprises: encoding the nodes and edges in the scene graph to obtain feature vectors with the same dimension as the targets; The graph convolution features are obtained using a graph convolution network based on the obtained feature vectors.

8. The method of claim 2, wherein, If the feature information of the multimedia data includes at least two of local visual features, semantic features, spatio-temporal visual features, and global features, the generating of the text description of the multimedia data based on the extracted feature information includes: determining the weight of each feature information; performing weighted processing on each feature information based on the weight of each feature information; generating the text description of the multimedia data based on the weighted processed feature information.

9. The method of claim 2, wherein, The generating of the text description of the multimedia data based on the extracted feature information includes: performing encoding processing on the obtained feature information using a self-attention-based encoder; inputting the encoded feature information into a decoder to generate the text description of the multimedia data; wherein, if the multimedia data is an image, the self-attention-based encoder is a self-attention-based intra-frame encoder; if the multimedia data is a video, the self-attention-based encoder includes a self-attention-based intra-frame encoder and / or a self-attention-based inter-frame encoder.

10. The method of claim 1, wherein, The generating of the text description of the multimedia data based on the extracted feature information includes: inputting the extracted feature information into multiple decoders respectively; obtaining the text description of the multimedia data based on the decoding results of the decoders.

11. The method of claim 10, wherein, When training the decoder, a start symbol BOS is replaced by length information to control the length of the text description to be generated.

12. The method of claim 1, wherein, The text description of the multimedia data is obtained by a multimedia data description model, wherein the multimedia data description model is trained in the following manner: obtaining training samples, wherein the training samples include first sample multimedia data with description annotations; training an initial description model based on the first sample multimedia data until a model loss function converges, and taking the trained description model as the multimedia data description model.

13. The method of claim 12, wherein, The training samples also include second sample multimedia data without description annotations, and the model loss function includes a first loss function and a second loss function. The training of the initial description model based on the first sample multimedia data until the model loss function converges includes: training a preset description model based on the first sample multimedia data to obtain the value of the first loss function, and training the description model based on the second sample multimedia data to obtain the value of the second loss function; obtaining the value of the final loss function based on the value of the first loss function and the value of the second loss function, training the description model based on the value of the final loss function until the final loss function converges.

14. The method of claim 13, wherein, The training of the description model based on the second sample multimedia data to obtain the value of the second loss function includes: performing at least one data augmentation on the second sample multimedia data to obtain third sample multimedia data; inputting the second sample multimedia data into the description model to obtain at least one multimedia description; determining the score of each multimedia description based on the second sample multimedia data and the third sample multimedia data; obtaining the value of the second loss function based on the scores of the multimedia descriptions.

15. The method according to any one of claims 12 to 14, characterized in that, The description labels of the first sample multimedia data include at least one original description label of the first sample multimedia data, and an enhanced description label corresponding to each original description label; The enhanced description label is obtained by the following way: According to each original description label of the first sample multimedia data, an enhanced image description label corresponding to each original description label is generated.

16. An electronic device, comprising: Comprise: Memory; And Processor; The memory has a computer program stored therein; The processor is configured to execute the method of any one of claims 1 to 15 when the computer program is run.

17. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored therein, and the computer program is configured to execute the method of any one of claims 1 to 15 when the processor is run.

18. A computer program product, characterised in that, Comprise non-volatile computer storage medium, the storage medium has a computer program stored therein, and the computer program is configured to execute the method of any one of claims 1 to 15 when the processor is run.

Images