Multimedia classification method, device, equipment and storage medium

By cross-fusion processing of multimodal information in multimedia files, the problem of inability to align features in the deep semantic stage is solved, and the multimodal information fusion performance and modal feature extraction performance are improved.

CN114385839BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210035591.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-12
Publication Date
2024-11-26
Estimated Expiration
2042-01-12

AI Technical Summary

Technical Problem

The existing multimodal information fusion method is based on the late fusion strategy of deep learning, which leads to the problem of insufficient coordination or suppression of information, resulting in poor fusion effect and inability to align features in the deep semantic stage.

Method used

By extracting at least two modal information of the multimedia file, cross-fusion processing is performed using isomorphic models, modal features are extracted layer by layer, and fused to obtain semantic features to avoid the inability to align features in the deep semantic stage. question.

Benefits of technology

The maximum alignment of modal information is achieved, the multimodal information fusion performance is improved, and the performance and characterization ability of single modal feature extraction are promoted.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114385839B_ABST
    Figure CN114385839B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimedia classification method, device, equipment and storage medium, the method comprising processing a multimedia file to obtain at least two modal information of the multimedia file; extracting features of the at least two modal information of the multimedia file respectively to obtain at least two modal features, wherein, in the i-th stage of feature extraction of any modal information in the at least two modal information, the modal features of the i-th stage of the any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information; fusing the at least two modal features to obtain the semantic features of the multimedia file; and classifying the multimedia file according to the semantic features of the multimedia file. By adopting this method, the problem of feature alignment failure at the deep semantic stage is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a multimedia classification method, device, equipment and storage medium. Background Art

[0002] Multimedia files, such as videos, are a medium for transmitting multiple types of information. For example, a video may contain text, visual, and auditory information at the same time. Therefore, multimodal learning has gradually developed into the main means of analyzing and understanding video content.

[0003] Multimodal information fusion is a very important part of multimodal technology. Multimodal information fusion is how to fuse multimodal information, such as text, sound and image information, together to better represent the semantic information of multimedia files such as videos.

[0004] The current mainstream modal information fusion methods are mainly based on deep learning methods, which first extract the semantic information of each modality and then fuse it. However, these multimodal information fusions are all based on late fusion strategies, and the information fusion is too delayed, resulting in insufficient coordination or suppression of information, so the fusion effect is very poor. Summary of the invention

[0005] The embodiments of the present invention provide a multimedia classification method, apparatus, device and storage medium, which can avoid the problem of feature alignment failure in the deep semantic stage.

[0006] In a first aspect, an embodiment of the present invention provides a multimedia classification method, comprising:

[0007] Processing the multimedia file to obtain at least two modal information of the multimedia file;

[0008] Performing feature extraction on at least two modal information of the multimedia file respectively to obtain at least two modal features, wherein, at the i-th stage of feature extraction of any modal information among the at least two modal information, the modal features of the i-th stage of the any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information among the at least two modal information except the any modal information, the i-th stage is any stage of feature extraction of the any modal information, and i is an integer not less than 2;

[0009] Fusing the at least two modal features to obtain semantic features of the multimedia file;

[0010] The multimedia files are classified according to the semantic features of the multimedia files.

[0011] As an implementation method, the modal features of the i-1th stage of any modal information are fused with the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information, so as to obtain the first modal features of the i-th stage of any modal information;

[0012] The modal features of the i-th stage of any modal information are obtained according to the first modal features of the i-th stage of any modal information.

[0013] Specifically, the extracting features of the at least two modal information of the multimedia file respectively to obtain at least two modal features includes:

[0014] The at least two modal information of the multimedia file are respectively input into at least two models for processing to obtain at least two modal features, wherein, for any one of the at least two models, the input of the i-th layer of the model is obtained by fusing the output result of the i-1th layer of the model with the output result of the i-1th layer of other models among the at least two models except the model, and the at least two modal information correspond one-to-one to the at least two models.

[0015] As a specific implementation method, the at least two modal information include first modal information and second modal information, and the input of the i-th layer of the model corresponding to the first modal information is obtained by taking the i-1th layer output result of the model as a query, and processing the i-1th layer output result of the model corresponding to the second modal information as a key and value.

[0016] Wherein, the at least two models are isomorphic models.

[0017] In a second aspect, an embodiment of the present invention provides a multimedia classification device, including:

[0018] A processing module, used for processing the multimedia file to obtain at least two modal information of the multimedia file;

[0019] An extraction module, configured to perform feature extraction on at least two modal information of the multimedia file respectively to obtain at least two modal features, wherein, at the i-th stage of feature extraction of any modal information among the at least two modal information, the modal features of the i-th stage of the any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information among the at least two modal information except the any modal information, the i-th stage is any stage of feature extraction of the any modal information, and i is an integer not less than 2;

[0020] A fusion module, used for fusing the at least two modal features to obtain the semantic features of the multimedia file;

[0021] The classification module is used to classify the multimedia files according to the semantic features of the multimedia files.

[0022] Optionally, the extraction module is used to fuse the modal features of the i-1th stage of any modal information with the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information, so as to obtain the first modal features of the i-th stage of any modal information;

[0023] The modal features of the i-th stage of any modal information are obtained according to the first modal features of the i-th stage of any modal information.

[0024] Among them, the extraction module is also used to input at least two modal information of the multimedia file into at least two models for processing respectively to obtain at least two modal features, wherein, for any model of the at least two models, the input of the i-th layer of the model is obtained by fusing the output result of the i-1th layer of the model with the output result of the i-1th layer of other models among the at least two models except the model, and the at least two modal information corresponds one-to-one to the at least two models.

[0025] In a third aspect, an embodiment of the present invention provides a multimedia classification device, including: a processor and a memory;

[0026] The processor is connected to a memory, wherein the memory is used to store program code, and the processor is used to call the program code to execute the multimedia classification method as described in any one of the first aspects.

[0027] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method for multimedia classification as described in any one of the first aspects is executed.

[0028] In an embodiment of the present invention, at least two modal features are obtained based on feature extraction of at least two modal information of a multimedia file, wherein in the feature extraction stage, the modal features of the i-th stage of any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information; then, the semantic features of the multimedia file are obtained by fusing the at least two modal features to perform subsequent task processing. With this method, each stage of the single modal feature extraction is extracted based on each modal information of the multimedia file, and the alignment of the modal information can be performed to the greatest extent, avoiding the problem of feature alignment failure in the deep semantic stage; and based on the fusion of multimodal information, the performance of the single modal feature extraction can be simultaneously promoted, and the representation capability is stronger. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0030] Figure 1a is a scene schematic diagram of a multimedia classification method provided by an embodiment of the present invention;

[0031] Figure 1b is a schematic diagram of a multimedia file to be classified provided by an embodiment of the present invention;

[0032] Figure 1c is a schematic diagram of a multimedia classification method provided by an embodiment of the present invention;

[0033] Figure 2 is a flowchart of a multimedia classification method provided by an embodiment of the present invention;

[0034] Figure 3 is a flow chart of another multimedia classification method provided by an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of model processing provided by an embodiment of the present invention;

[0036] Figure 5 is a structural schematic diagram of a multimedia classification device provided by an embodiment of the present invention;

[0037] Figure 6 It is a structural schematic diagram of another multimedia classification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0040] Reference to "embodiments" in the present invention means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present invention may be combined with other embodiments.

[0041] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0042] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0043] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0044] See also Figure 1a , Figure 1a FIG. 1 is a schematic diagram of a scenario application of a multimedia classification method provided by an embodiment of the present invention. Figure 1a As shown, the multimedia classification server 10d receives multimedia files provided by electronic devices, and then implements multimedia classification. The electronic devices may include: computers 10a, laptop computers 10b, ..., mobile phones 10c, televisions, etc. Of course, the multimedia classification server 10d can also obtain the multimedia files to be classified through other means. For example, the multimedia classification server 10d receives a multimedia file sent by a laptop computer 10b, and the multimedia file is as follows Figure 1b The video of the ball game shown includes badminton game footage, expert commentary footage, and basketball game footage.

[0045] like Figure 1c As shown, it is a schematic diagram of the multimedia classification method provided by an embodiment of the present application. Among them, the video shown in Figure 1b is processed to obtain pure image information, pure voice information and pure text information. Then, the obtained pure image information, pure voice information and pure text information are respectively subjected to feature extraction, wherein in the feature extraction stage, the extraction of image features is not only related to image information, but also to other voice information and text information; correspondingly, in the feature extraction stage of voice information, the extraction of voice features is not only related to voice information, but also to other image information and text information; and so on, each modal feature can be obtained. Then it is fused and then video classification is performed. By adopting this method, each stage of single modal feature extraction is extracted based on each modal information of the video, which can align the modal information to the greatest extent, avoiding the problem of feature alignment in the deep semantic stage; and based on multimodal information fusion, the performance of single modal feature extraction can be promoted at the same time, and the representation ability is stronger.

[0046] This solution cross-integrates different modal information, so that multi-modal information is easier to align and coordinate, improving the fusion performance of multi-modal information. This method can be applied to various applications of multimedia understanding such as video, such as video classification, video generation, video summary, video summarization, image classification, image generation, etc., and has a wide range of application value.

[0047] See also Figure 2 , Figure 2 It is a flowchart of a multimedia classification method provided by an embodiment of the present invention; Figure 2 In the method, the method includes steps 201-204, which are as follows:

[0048] 201. Process a multimedia file to obtain at least two modal information of the multimedia file;

[0049] Multimodal information in multimedia files may include text, voice, image information, etc.

[0050] For example, by converting the format of a multimedia file, voice information in the multimedia file can be extracted; by inputting the multimedia file into a preset software, text information, image information, etc. in the multimedia file can be obtained.

[0051] That is to say, by processing the multimedia file, pure text information, pure voice information, pure image information, etc. in the multimedia file can be obtained.

[0052] 202. Perform feature extraction on at least two modal information of the multimedia file respectively to obtain at least two modal features, wherein, at the i-th stage of feature extraction of any modal information among the at least two modal information, the modal features of the i-th stage of the any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information among the at least two modal information except the any modal information, the i-th stage is any stage of feature extraction of the any modal information, and i is an integer not less than 2;

[0053] That is to say, multiple extraction stages are included when extracting features for each modal information.

[0054] The modal features of the i-th stage of any modal information are obtained based on the modal features of the i-1-th stage of the any modal information and the modal features of the i-1-th stage of other modal information in the at least two modal information except the any modal information. That is, in the stage of feature extraction of each modal information, feature extraction is performed through the multimodal information.

[0055] As an implementation method, the modal features of the i-1th stage of any modal information are fused with the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information, so as to obtain the first modal features of the i-th stage of any modal information;

[0056] The modal features of the i-th stage of any modal information are obtained according to the first modal features of the i-th stage of any modal information.

[0057] For example, in the fusion process, if it is at the stage of extracting text information features, the proportion of text information features in the fusion is larger, and the proportion of other modal features is lower than that of text information features. Correspondingly, if it is at the stage of extracting voice information features, the proportion of voice information features in the fusion is larger, and the proportion of other modal features is lower than that of voice information features, etc.

[0058] The above is only an example, and there may be other ways, which are not specifically limited in this solution.

[0059] 203. Fusing the at least two modal features to obtain semantic features of the multimedia file;

[0060] Based on each modality feature obtained above, the semantic features of the multimedia file are obtained by final fusion processing, wherein the fusion processing may be a simple vector superposition, etc., which is not specifically limited in this solution.

[0061] 204. Classify the multimedia files according to the semantic features of the multimedia files.

[0062] Based on the semantic features of the multimedia file, a label category is provided for the multimedia file so as to automatically archive and classify the multimedia file.

[0063] The embodiments of the present application are only explained by taking multimedia file classification as an example. It should be noted that the embodiments of the present application can also perform different processing based on different tasks. For example, if the current task is to determine a summary for the multimedia file, then based on the semantic features of the multimedia file, the multimedia file is summarized, and a title can be automatically given to the multimedia file to facilitate retrieval and recommendation, etc.; this solution can also be applied to multimedia file splitting, providing multimedia file fragments of independent subject units, and providing materials for multimedia file production and multimedia file cataloging. For example, if an interview video of someone is needed, the corresponding scene fragments can be obtained through the scene segmentation capability. This solution does not make specific limitations on this.

[0064] In the embodiment of the present application, at least two modal features are obtained based on feature extraction of at least two modal information of a multimedia file, wherein in the feature extraction stage, the modal features of the i-th stage of any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information; then, the semantic features of the multimedia file are obtained by fusing the at least two modal features for subsequent task processing. With this method, each stage of the extraction of a single modal feature is extracted based on each modal information of the multimedia file, and the alignment of the modal information can be performed to the greatest extent, avoiding the problem of feature alignment failure in the deep semantic stage; and based on the fusion of multimodal information, the performance of the extraction of a single modal feature can be simultaneously promoted, with a stronger representation capability.

[0065] This application embodiment is described by taking the multimedia file as a video as an example. Figure 3 , Figure 3 It is a flowchart of a multimedia classification method provided by an embodiment of the present invention; Figure 3 Taking video classification as an example, the method includes steps 301-304, which are as follows:

[0066] 301. Process a video to obtain at least two modal information of the video;

[0067] Multimodal information in a video may include text, voice, image information, etc.

[0068] For example, by converting the video format, the voice information in the video can be extracted; by inputting the video into the preset software, the text information, image information, etc. in the video can be obtained.

[0069] In other words, by processing the video, pure text information, pure voice information, pure image information, etc. in the video can be obtained.

[0070] 302. Input at least two modal information of the video into at least two models for processing respectively to obtain at least two modal features, wherein, for any model of the at least two models, the input of the i-th layer of the model is obtained by fusing the output result of the i-1th layer of the model with the output result of the i-1th layer of other models among the at least two models except the model, the at least two modal information corresponds to the at least two models one-to-one, and i is an integer not less than 2;

[0071] That is to say, each modal information is input into the corresponding model respectively. For any model, the input of the first layer of the model is any modal information obtained in step 301. The input of the second layer of the model is a vector obtained by fusing the output result of the first layer of the model with the output result of the first layer of other models. Correspondingly, the input of the third layer of the model is a vector obtained by fusing the output result of the second layer of the model with the output result of the second layer of other models. By analogy, the input of each layer of the model can be obtained, and then the output result of each layer of the model can be obtained. By adopting this means, the output result of each model, that is, the above-mentioned at least two modal features, can be obtained.

[0072] Specifically, the frameworks of the at least two models are isomorphic. The model can be a Transformer structure. The Transformer model adopts an encoder-decoder architecture. For the encoder, it can include two layers, a self-attention layer and a feedforward neural network. Self-attention can help the current node not only focus on the current word, so as to obtain the semantics of the context. The decoder also includes the two-layer network mentioned by the encoder, but there is an attention layer between the two layers to help the current node obtain the key content that needs to be paid attention to.

[0073] First, the model needs to perform an embedding operation on the input data. After the embedding is completed, it is input to the encoder layer. After the self-attention processes the data, it sends the data to the feedforward neural network. The calculation of the feedforward neural network can be parallelized, and the output result will be input to the next encoder. Self-attention calculates three new vectors, which are called Query, Key, and Value. These three vectors are the result of multiplying the embedding vector with a matrix. This matrix is ​​randomly initialized and its value will be updated during the BP process.

[0074] By calculating the score of self-attention and processing the result based on softmax, the result obtained is the relevance of each word to the word at the current position.

[0075] The decoder part is similar to the encoder part and will not be described in detail here.

[0076] for Figure 4The input of the i+1th layer in the neural network shown is obtained by fusing the output of the ith layer of the neural network with the output of the ith layer of other neural networks. For example, if the text information shown in the video is "I am a student", then position 1 in the figure represents "I", position 2 represents "I am", position 3 represents "a", and position 4 represents "student". H represents the feature vector corresponding to each position. For example, H V Represents the visual feature vector, H A Characterizes the sound feature vector, etc. For the network currently extracting visual features, the input of the corresponding i+1 layer is the output Q of the i-th layer of the network V and the output K of the i-th layer of the network that extracts sound features A 、V A Similarly, for the network currently extracting sound features, the input of the corresponding layer is the output Q of the previous layer on the network. A And the output K of the previous layer of the network that extracts visual features V 、V V This embodiment only takes information of two modes as an example for explanation. When information of multiple modes is to be fused, the output K and V of the upper layer of other networks are also included.

[0077] That is to say, the output result of the feature extraction corresponding to the modal information is used as the Query, and the output results of the feature extraction corresponding to other modal information are used as the Key and Value, which are processed as the input of the feature extraction corresponding to the modal information, and then repeated until the last layer of the feature extraction corresponding to the modal information is executed.

[0078] By adopting this method, cross-fusion is performed at each stage of single modal feature extraction, which can maximally align the modal information and avoid the problem of feature alignment failure at the deep semantic stage. Moreover, based on multimodal information fusion, the performance of single modal feature extraction can be promoted at the same time, and the representation ability is stronger.

[0079] 303. Fusing the at least two modal features to obtain semantic features of the video;

[0080] Based on each modality feature obtained above, the semantic features of the video are obtained by final fusion processing, wherein the fusion processing may be a simple vector superposition, etc., which is not specifically limited in this solution.

[0081] For example, a late fusion strategy is used to fuse multimodal information. Late fusion refers to fusion on the prediction score. Specifically, by training multiple models, each model will have a prediction score, and the final prediction result is obtained by fusing the results of all models. Common late fusion methods include taking the average, maximum, and weighted average of the scores. In addition, there is also the LogisticsRegression method for late fusion. Of course, other means can also be used, and this solution does not make specific restrictions on this.

[0082] 304. Classify the video according to the semantic features of the video.

[0083] Based on the semantic features of the video, a label category is provided for the video so as to automatically archive and classify the video.

[0084] The embodiments of the present application are only explained by taking video classification as an example. It should be noted that the embodiments of the present application can also perform different processing based on different tasks. For example, if the current task is to determine a summary for the video, then based on the semantic features of the video, the video is summarized, and a title can be automatically given to the video to facilitate retrieval and recommendation, etc.; this solution can also be applied to video splitting, providing video clips of independent subject units, and providing materials for video production and video cataloging. For example, if an interview video of someone is needed, the corresponding scene clips can be obtained through the scene segmentation capability. This solution does not make specific limitations on this.

[0085] Specifically, if the current task is to classify the video, the fused video semantic vector is input into the fully connected layer for video classification and label classification, and the video can be automatically archived. If the current task is to determine a summary for the video, the fused video semantic vector is input into the decoder module to summarize the video, and the video can be automatically titled for easy retrieval and recommendation. Of course, it can also be applied to video splitting, providing video clips of independent subject units, and providing materials for video production and video cataloging. For example, if a video of an interview with someone is needed, the corresponding scene clips can be obtained through the scene segmentation capability.

[0086] In the embodiment of the present application, at least two modal features are obtained based on feature extraction of at least two modal information of the video, wherein in the feature extraction stage, the modal features of the i-th stage of any modal information are obtained by fusing the modal features of the i-1th stage of any modal information and the modal features of the i-1th stage of other modal information except the any modal information in the at least two modal information; then, the semantic features of the video are obtained by fusing the at least two modal features to perform subsequent task processing. By adopting this method, by cross-fusion at each stage of single modal feature extraction, the alignment of modal information can be performed to the greatest extent, avoiding the problem of feature alignment failure at the deep semantic stage; and based on multimodal information fusion, the performance of single modal feature extraction can be simultaneously promoted, and the representation ability is stronger.

[0087] Based on the description of the above multimedia classification method embodiment, the embodiment of the present invention also discloses a multimedia classification device, referring to Figure 5 , Figure 5 5 is a schematic diagram of the structure of a multimedia classification device provided by an embodiment of the present invention, wherein the multimedia classification device includes a processing module 501, an extraction module 502, a fusion module 503 and a classification module 504, wherein:

[0088] The processing module 501 is used to process the multimedia file to obtain at least two modal information of the multimedia file;

[0089] An extraction module 502 is used to perform feature extraction on at least two modal information of the multimedia file respectively to obtain at least two modal features, wherein, at the i-th stage of feature extraction of any modal information among the at least two modal information, the modal features of the i-th stage of the any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information among the at least two modal information except the any modal information, the i-th stage is any stage of feature extraction of the any modal information, and i is an integer not less than 2;

[0090] A fusion module 503, configured to fuse the at least two modal features to obtain a semantic feature of the multimedia file;

[0091] The classification module 504 is used to classify the multimedia files according to the semantic features of the multimedia files.

[0092] Optionally, the extraction module 502 is used to fuse the modal features of the i-1th stage of any modal information with the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information, so as to obtain the first modal features of the i-th stage of any modal information;

[0093] The modal features of the i-th stage of any modal information are obtained according to the first modal features of the i-th stage of any modal information.

[0094] Among them, the extraction module 502 is also used to: input the at least two modal information of the multimedia file into at least two models for processing respectively to obtain at least two modal features, wherein, for any model of the at least two models, the input of the i-th layer of the model is obtained by fusing the output result of the i-1th layer of the model with the output result of the i-1th layer of other models in the at least two models except the model, and the at least two modal information corresponds one-to-one to the at least two models.

[0095] As a specific implementation method, the at least two modal information include first modal information and second modal information, and the input of the i-th layer of the model corresponding to the first modal information is obtained by taking the i-1th layer output result of the model as a query, and processing the i-1th layer output result of the model corresponding to the second modal information as a key and value.

[0096] Wherein, the at least two models are isomorphic models.

[0097] In the embodiment of the present application, at least two modal features are obtained based on feature extraction of at least two modal information of a multimedia file, wherein in the feature extraction stage, the modal features of the i-th stage of any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information; then, the semantic features of the multimedia file are obtained by fusing the at least two modal features for subsequent task processing. With this method, each stage of the extraction of a single modal feature is extracted based on each modal information of the multimedia file, and the alignment of the modal information can be performed to the greatest extent, avoiding the problem of feature alignment failure in the deep semantic stage; and based on the fusion of multimodal information, the performance of the extraction of a single modal feature can be simultaneously promoted, with a stronger representation capability.

[0098] It is worth pointing out that, among them, the specific functional implementation method of the multimedia classification device can refer to the description of the multimedia classification method mentioned above, which will not be repeated here. The various units or modules in the multimedia classification device can be respectively or completely merged into one or several other units or modules to constitute, or one (some) of the units or modules can also be divided into multiple smaller units or modules to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided based on logical functions. In practical applications, the function of a unit (or module) can also be implemented by multiple units (or modules), or the functions of multiple units (or modules) are implemented by one unit (or module).

[0099] Based on the description of the above method embodiment and device embodiment, an embodiment of the present invention further provides a device for multimedia classification.

[0100] See also Figure 6 , is a schematic diagram of the structure of a multimedia classification device provided by an embodiment of the present invention. Figure 6 The device 600 shown (the device 600 may be a computer device) includes a memory 601, a processor 602, a communication interface 603 and a bus 604. The memory 601, the processor 602 and the communication interface 603 are connected to each other through the bus 604.

[0101] The memory 601 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0102] The memory 601 can store programs. When the program stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 603 are used to perform each step of the multimedia classification method of the embodiment of the present application.

[0103] The processor 602 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the multimedia classification device of the embodiment of the present application, or to execute the multimedia classification method of the method embodiment of the present application.

[0104] The processor 602 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the multimedia classification method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 602. The above-mentioned processor 602 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601, and the processor 602 reads the information in the memory 601, and combines its hardware to complete the functions required to be performed by the units included in the multimedia classification device of the embodiment of the present application, or executes the multimedia classification method of the method embodiment of the present application.

[0105] The communication interface 603 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 600 and other devices or a communication network. For example, data can be acquired through the communication interface 603 .

[0106] The bus 604 may include a path for transmitting information between various components of the device 600 (eg, the memory 601 , the processor 602 , and the communication interface 603 ).

[0107] It should be noted that although Figure 6 The device 600 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 600 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 600 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 600 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 6 All devices shown in .

[0108] An embodiment of the present application also provides a chip system, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the multimedia classification method.

[0109] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0110] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the specific descriptions of the corresponding steps in the aforementioned method embodiments, and will not be repeated here.

[0112] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0113] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0116] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A multimedia classification method, characterized in that: include: Processing the multimedia file to obtain at least two modal information of the multimedia file; Performing feature extraction on at least two modal information of the multimedia file respectively to obtain at least two modal features, wherein, at the i-th stage of feature extraction of any modal information among the at least two modal information, the modal features of the i-th stage of the any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information among the at least two modal information except the any modal information, the i-th stage is any stage of feature extraction of the any modal information, and i is an integer not less than 2; Fusing the at least two modal features to obtain semantic features of the multimedia file; Classifying the multimedia files according to the semantic features of the multimedia files; The extracting features of the at least two modal information of the multimedia file respectively to obtain at least two modal features includes: The at least two modal information of the multimedia file are respectively input into at least two models of a corresponding number for processing to obtain at least two modal features, wherein, for any one of the at least two models, the input of the i-th layer of the model is obtained by fusing the output result of the i-1th layer of the model with the output result of the i-1th layer of other models among the at least two models except the model, and the at least two modal information correspond one-to-one to the at least two models; the at least two modal information include first modal information and second modal information, the input of the i-th layer of the model corresponding to the first modal information is the output result of the i-1th layer of the model and the output result of the i-1th layer of the model corresponding to the second modal information, and the corresponding number of at least two models are isomorphic models.

2. The method according to claim 1, characterized in that: include: fusing the modal features of the i-1th stage of any modal information with the modal features of the i-1th stage of other modal information among the at least two modal information except the any modal information, so as to obtain the first modal features of the i-th stage of any modal information; The modal features of the i-th stage of any modal information are obtained according to the first modal features of the i-th stage of any modal information.

3. A multimedia classification device, characterized in that: include: A processing module, used for processing the multimedia file to obtain at least two modal information of the multimedia file; An extraction module, configured to perform feature extraction on at least two modal information of the multimedia file respectively to obtain at least two modal features, wherein, at the i-th stage of feature extraction of any modal information among the at least two modal information, the modal features of the i-th stage of the any modal information are obtained based on the modal features of the i-1th stage of the any modal information and the modal features of the i-1th stage of other modal information among the at least two modal information except the any modal information, the i-th stage is any stage of feature extraction of the any modal information, and i is an integer not less than 2; The extraction module, when performing feature extraction on the at least two modal information of the multimedia file to obtain at least two modal features, is specifically used to input the at least two modal information of the multimedia file into at least two models of a corresponding number for processing to obtain at least two modal features, wherein, for any one of the at least two models, the input of the i-th layer of the model is obtained by fusing the output result of the i-1th layer of the model with the output result of the i-1th layer of other models in the at least two models except the model, and the at least two modal information correspond to the at least two models one-to-one; the at least two modal information include first modal information and second modal information, the input of the i-th layer of the model corresponding to the first modal information is the output result of the i-1th layer of the model and the output result of the i-1th layer of the model corresponding to the second modal information, and the at least two models of a corresponding number are isomorphic models; A fusion module, used for fusing the at least two modal features to obtain the semantic features of the multimedia file; The classification module is used to classify the multimedia files according to the semantic features of the multimedia files.

4. The device according to claim 3, characterized in that The extraction module is used to fuse the modal features of the i-1th stage of any modal information with the modal features of the i-1th stage of other modal information in the at least two modal information except the any modal information, so as to obtain the first modal features of the i-th stage of any modal information; The modal features of the i-th stage of any modal information are obtained according to the first modal features of the i-th stage of any modal information.

5. A multimedia classification device, characterized in that: include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the multimedia classification method as claimed in claim 1 or 2.

6. A computer storage medium, characterized in that: The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the method for multimedia classification according to claim 1 or 2 is executed.

Citation Information

Patent Citations

  • Multimedia content description generation method and device, training method and device, equipment and medium

    CN110234018A

  • System and method for searching similarity of multimedia files

    KR1020180099126A

  • Recognizing salient video events through learning-based multimodal analysis of visual features and audio-based analytics

    US20160004911A1

  • Deep hierarchical fusion for machine intelligence applications

    US20200335092A1