Multimedia information processing method, device, equipment and storage medium
By performing intermodal feature fusion in the feature extraction network, the problem that traditional algorithms cannot identify risks in multimedia information is solved, and the complete extraction and accurate identification of information are achieved.
Patent Information
- Application Number
- CN202210220964.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-08
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-08
AI Technical Summary
Traditional video recognition algorithms cannot accurately identify risks in multimedia information, and some information is lost prematurely during feature extraction.
By inputting multiple modal information from multimedia information into corresponding feature extraction networks, and using preset computational layers to fuse feature information, feature fusion between modalities is achieved until the last computational layer of the feature extraction network corresponding to each modal information outputs feature information.
The feature extraction process avoids premature information loss, fully utilizes the correlation between multiple modal information, achieves more effective information fusion, and accurately identifies risks in multimedia information.
Smart Images

Figure CN114821385B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of information technology, and in particular to a multimedia information processing method, device, equipment and storage medium. Background Art
[0002] With the continuous development of technology, users can install various applications (APP) on their terminals to meet their different needs. For example, users can upload their own multimedia information, such as text, pictures, voice, video, etc., to a third-party platform through a social APP, or the third-party platform can create or pre-store multimedia information so that other users can browse and watch it through the third-party platform.
[0003] In order to ensure that the network transmission of multimedia information is safe, it is usually necessary to identify whether there are risks in the multimedia information through algorithms or manual review, so as to shield the risky multimedia information.
[0004] However, the inventors of the present application have discovered that conventional video recognition algorithms are unable to accurately identify risks present in multimedia information. Summary of the invention
[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a multimedia information processing method, device, equipment and storage medium, so as to avoid the problem that each modal information causes partial information to be lost prematurely during the feature extraction process. Therefore, in the feature extraction process, since the correlation between multiple modal information is fully utilized, more effective information fusion is achieved, so that the risks existing in the multimedia information can be accurately identified.
[0006] In a first aspect, an embodiment of the present disclosure provides a multimedia information processing method, including:
[0007] Acquire multiple modal information in the multimedia information, wherein the multiple modal information corresponds one-to-one to multiple feature extraction networks;
[0008] Inputting each modal information of the plurality of modal information into a feature extraction network corresponding to the modal information;
[0009] For at least one target modal information among the multiple modal information, feature information of the target modal information and feature information of at least one other modal information are fused through a preset computing layer in a feature extraction network corresponding to the target modal information;
[0010] The risk assessment of the multimedia information is performed according to the feature information output by the last computing layer in each feature extraction network.
[0011] In a second aspect, an embodiment of the present disclosure provides an information fusion method, the method comprising:
[0012] Acquire real world information in the surrounding environment of the virtual reality device and virtual world information presented by a display component of the virtual reality device;
[0013] Inputting the real world information into a first feature extraction network, and inputting the virtual world information into a second feature extraction network;
[0014] The feature information of the real world information and the feature information of the virtual world information are fused through a preset computing layer in the first feature extraction network, and / or the feature information of the virtual world information and the feature information of the real world information are fused through a preset computing layer in the second feature extraction network.
[0015] In a third aspect, an embodiment of the present disclosure provides an object recognition method, the method comprising:
[0016] Acquire a first image, where the first image is obtained of the target object under a first imaging condition;
[0017] adjusting the first imaging condition to obtain a second imaging condition;
[0018] Acquire a second image, where the second image is obtained from the target object under the second imaging condition;
[0019] It is determined whether the target object is a physical object according to the first image, the second image, the first imaging condition, and the second imaging condition.
[0020] In a fourth aspect, an embodiment of the present disclosure provides a multimedia information processing device, including:
[0021] An acquisition module, used to acquire multiple modal information in the multimedia information, wherein the multiple modal information corresponds one-to-one to multiple feature extraction networks;
[0022] A first input module, used for inputting each modal information of the plurality of modal information into a feature extraction network corresponding to the modal information;
[0023] A fusion module, for fusing feature information of at least one target modal information among the multiple modal information with feature information of at least one other modal information through a preset computing layer in a feature extraction network corresponding to the target modal information;
[0024] The risk assessment module is used to perform risk assessment on the multimedia information according to the feature information output by the last computing layer in each feature extraction network.
[0025] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:
[0026] Memory;
[0027] Processor; and
[0028] Computer programs;
[0029] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.
[0030] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.
[0031] The multimedia information processing method, device, equipment and storage medium provided by the embodiments of the present disclosure, by inputting each modal information in the multimedia information into the feature extraction network corresponding to the modal information, there is at least one target modal information in the multiple modal information, and for each target modal information, the feature information of the target modal information and the feature information of at least one other modal information can be fused through the preset calculation layer in the feature extraction network corresponding to the target modal information. Since the output of the previous calculation layer in the feature extraction network can be the input of the next calculation layer, the fusion result obtained after the fusion process or part of the information in the fusion result can be input to the next calculation layer of the preset calculation layer, until the last calculation layer of the feature extraction network corresponding to each modal information outputs feature information. Finally, the multimedia information is risk assessed according to the feature information output by the last calculation layer in each feature extraction network. Since the embodiment of the present disclosure can fuse the feature information of different modal information, that is, feature fusion between modalities, during the process of extracting the feature information of each modal information, it is ensured that the feature fusion between modalities can be performed before part of the information in each modal information is lost. This avoids the problem of premature loss of part of the information caused by each modal information during the feature extraction process. Therefore, in the feature extraction process, the correlation between multiple modal information is fully utilized to achieve more effective information fusion, so that the risks in multimedia information can be accurately identified. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0034] Figure 1 A flowchart of a multimedia information processing method provided by an embodiment of the present disclosure;
[0035] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0036] Figure 3 A schematic diagram of feature fusion provided in an embodiment of the present disclosure;
[0037] Figure 4 A schematic diagram of fusion computing provided by an embodiment of the present disclosure;
[0038] Figure 5 A schematic diagram of feature fusion provided by another embodiment of the present disclosure;
[0039] Figure 6 A schematic diagram of feature fusion provided by another embodiment of the present disclosure;
[0040] Figure 7 A schematic diagram of the structure of a multimedia information processing device provided by an embodiment of the present disclosure;
[0041] Figure 8 A schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. DETAILED DESCRIPTION
[0042] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0044] Generally, in order to ensure the safety of multimedia information transmission on the Internet, it is usually necessary to identify whether there are risks in the multimedia information through algorithms or manual review, so as to block the multimedia information with risks. However, traditional video recognition algorithms cannot accurately identify the risks in multimedia information.
[0045] For example, multimedia information includes multiple modal information such as vision, voice, and text, but traditional video recognition algorithms find it difficult to fully utilize multiple modal information. For example, traditional video recognition algorithms process multiple modal information in parallel and independently, and extract feature information of each modal information such as vision, voice, and text. Further, the feature information of each modal information is fused, so as to obtain the recognition result of the multimedia information based on the fused feature information, and the recognition result includes the probability that the multimedia information is risky and the probability that it is risk-free. Alternatively, the recognition result includes the probability of what type of risk the risk of the multimedia information is, etc. Among them, after the feature information of each modal information is extracted, the feature information of each modal information only retains part of the information in the modal information, and the remaining part of the information has been lost in the process of feature extraction. When fusion is performed on the basis of such incomplete feature information, it will be impossible to fully utilize each modal information.
[0046] To address this problem, an embodiment of the present disclosure provides a multimedia information processing method, which is described below in conjunction with a specific embodiment.
[0047] Figure 1 This is a flowchart of a multimedia information processing method provided by an embodiment of the present disclosure. The method can be executed by a multimedia information processing device, which can be implemented in software and / or hardware, and can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc. Figure 2 Taking the application scenario shown as an example, the terminal 21 can send multimedia information to the server 22. The multimedia information can include at least two modal information of multiple modal information such as vision, voice, and text, wherein vision can be a continuous sequence of pictures obtained after the video is framed. Vision, voice, and text are different modal information. The server 22 can process the multimedia information to identify whether there are risks or types of risks in the multimedia information. Alternatively, the server 22 can receive multimedia information from other servers and process the multimedia information. Alternatively, the multimedia information can also be processed by the terminal 21. The following takes the processing of multimedia information by the server 22 as an example to introduce the multimedia information processing method described in the embodiment of the present disclosure. Figure 1 As shown, the specific steps of this method are as follows:
[0048] S101. Acquire multiple modal information in multimedia information, where the multiple modal information corresponds one-to-one to multiple feature extraction networks.
[0049] For example, when receiving multimedia information, the server 22 can obtain multiple modal information in the multimedia information. For example, if the multimedia information includes vision and voice, two modal information can be obtained. If the multimedia information includes vision, voice, and text, three modal information can be obtained. Among them, voice can also be recorded as audio. It can be understood that the multimedia information can also include other modal information in addition to vision, voice, and text. In addition, in this embodiment, each modal information can be feature extracted by a feature extraction network. If two modal information is obtained from the multimedia information, for example, vision and voice, then the feature information of vision can be extracted by a feature extraction network A, and the feature information of voice can be extracted by another feature extraction network B. In other words, multiple modal information corresponds one-to-one to multiple feature extraction networks.
[0050] S102: Input each modal information among the multiple modal information into a feature extraction network corresponding to the modal information.
[0051] For example, after obtaining multiple modal information from the multimedia information, each modal information can be input into a feature extraction network corresponding to the modal information. For example, vision is input into feature extraction network A, and speech is input into feature extraction network B.
[0052] Optionally, the structures of the multiple feature extraction networks are the same, and each feature extraction network includes multiple computing layers. For example, the structures of feature extraction network A and feature extraction network B are the same, and feature extraction network A and feature extraction network B include 12 computing layers respectively. However, the parameters of feature extraction network A and feature extraction network B are different. For example, feature extraction network A and feature extraction network B can be transformer networks respectively. In addition, feature extraction network A and feature extraction network B can also be other networks such as residual networks (Residual Network, ResNet). Taking the Transformer network as an example, the Transformer network includes multiple computing layers. For example, the Transformer network is composed of L cascaded computing layers. In this embodiment, the specific value of L is not limited, for example, L=12. Specifically, in the L computing layers, each computing layer takes the output features of the previous layer as input and outputs features to the next layer. Taking the feature extraction network A corresponding to vision as an example, each computing layer in the feature extraction network A can calculate a feature information of the vision, and the feature information output by the previous computing layer can be used as the input of the next computing layer, and the feature information output by different computing layers can be different. Among them, Transformer is a multi-layer neural network structure used to extract features of input information such as pictures and texts.
[0053] S103. For at least one target modal information among the multiple modal information, feature information of the target modal information and feature information of at least one other modal information are fused through a preset computing layer in a feature extraction network corresponding to the target modal information.
[0054] For example, the multimedia information includes three modal information of vision, speech, and text, and the text can be input into the feature extraction network C. One or more target modal information can be determined from the three modal information, so that the preset calculation layer in the feature extraction network corresponding to the target modal information performs fusion processing on the feature information of the target modal information and the feature information of at least one other modal information.
[0055] For example, when speech is used as the target modal information, the feature information of the speech and the feature information of at least one other modal information can be fused in a preset computing layer in the feature extraction network B. The at least one other modal information can be at least one of vision and text. The preset computing layer in the feature extraction network B can be one or more selected computing layers. For example, the 8th computing layer of the feature extraction network B can be selected for fusion processing. Or the 8th computing layer of the feature extraction network B can be selected for fusion processing once, and the 9th computing layer of the feature extraction network B can be selected for fusion processing again. It can be understood that when multiple computing layers are selected in the feature extraction network B for fusion processing, the multiple computing layers for fusion processing can be adjacent or non-adjacent.
[0056] It can be understood that if speech is taken as the target modal information, and vision and text are not the target modal information, then the preset computing layer in the feature extraction network B can be fused, and the computing layers in the feature extraction network A and the feature extraction network C are not fused. For example, each computing layer in the feature extraction network A is used to calculate the feature information of vision, and each computing layer in the feature extraction network C is used to calculate the feature information of text.
[0057] If speech and vision are used as target modal information, and text is not the target modal information, then the preset computing layer in feature extraction network B can be fused, and the preset computing layer in feature extraction network A can also be fused. Specifically, the preset computing layer in feature extraction network A can fuse the feature information of vision and the feature information of at least one of speech and text. The computing layer in feature extraction network C does not perform fusion processing.
[0058] Similarly, if the three modal information of vision, speech, and text are all used as target modal information, the preset calculation layers in feature extraction network A, feature extraction network B, and feature extraction network C can be fused separately. Among them, the preset calculation layer in feature extraction network C can fuse the feature information of text and the feature information of at least one of speech and vision. At this time, the levels of the preset calculation layers in feature extraction network A, feature extraction network B, and feature extraction network C can be the same or different.
[0059] S104: Perform risk assessment on the multimedia information according to the feature information output by the last computing layer in each feature extraction network.
[0060] It can be understood that, since the output of the previous calculation layer in the feature extraction network can be used as the input of the next calculation layer, after the preset calculation layer in the feature extraction network performs fusion processing, the preset calculation layer can input the fusion result or part of the information in the fusion result into the next calculation layer until the last calculation layer of the feature extraction network outputs feature information. In addition, the feature information output by the last calculation layer of feature extraction network A, feature extraction network B, and feature extraction network C may be different. For example, when speech is used as the target modal information and vision and text are not the target modal information, since the preset calculation layer of feature extraction network B can perform fusion processing on the feature information of speech and the feature information of at least one of vision and text, the fusion result or part of the information in the fusion result will include the feature information of speech and the feature information of at least one of vision and text. Therefore, the feature information output by the last calculation layer of feature extraction network B will also include the feature information of speech and the feature information of at least one of vision and text. The feature information output by the last calculation layer of feature extraction network A will be the final feature information of vision. The feature information output by the last calculation layer of feature extraction network C will be the final feature information of text.
[0061] Furthermore, the feature information outputted by the last calculation layer of feature extraction network A, feature extraction network B, and feature extraction network C is subjected to simple merging calculation, such as addition, cascading, maximum pooling, mean pooling, etc. Then, the result of the merging calculation is inputted into the classifier, and the classifier outputs the recognition result of the multimedia information. Specifically, the classifier may be a plurality of sigmoid binary classifiers or a softmax multi-classifier.
[0062] In the embodiment of the present disclosure, each modal information in the multimedia information is input into the feature extraction network corresponding to the modal information. There is at least one target modal information in the multiple modal information. For each target modal information, the feature information of the target modal information and the feature information of at least one other modal information can be fused through the preset computing layer in the feature extraction network corresponding to the target modal information. Since the output of the previous computing layer in the feature extraction network can be the input of the next computing layer, the fusion result obtained after the fusion processing or part of the information in the fusion result can be input to the next computing layer of the preset computing layer until the last computing layer of the feature extraction network corresponding to each modal information outputs feature information. Finally, the multimedia information is risk assessed according to the feature information output by the last computing layer in each feature extraction network. Since the embodiment of the present disclosure can fuse the feature information of different modal information, that is, feature fusion between modalities, during the process of extracting the feature information of each modal information, it is ensured that the feature fusion between modalities can be performed before part of the information in each modal information is lost. This avoids the problem of premature loss of part of the information caused by each modal information during the feature extraction process. Therefore, in the feature extraction process, the correlation between multiple modal information is fully utilized to achieve more effective information fusion, so that the risks in multimedia information can be accurately identified.
[0063] Optionally, feature information of the target modal information and feature information of at least one other modal information are fused through a preset computing layer in the feature extraction network corresponding to the target modal information, including: fusing implicit features and feature information output by a previous computing layer of the preset computing layer through a preset computing layer in the feature extraction network corresponding to the target modal information, the implicit features including feature information of at least one other modal information, and the feature information output by the previous computing layer including at least feature information of the target modal information.
[0064] The following uses visual and textual information as examples to introduce the process of fusing the feature information of these two modal information.
[0065] like Figure 3 As shown, the visual input V represents the visual modality information as described above. For example, the multimedia information includes a large number of continuous image sequences, which can be divided into several groups here, each group can include multiple frames (for example, 50 frames) of continuous image sequences, and each group is recorded as a visual input V. The multiple groups are sequentially input into the feature extraction network A, and the feature extraction network A can process the multiple groups in sequence. However, for the same group, the feature extraction network A can process 50 frames of continuous image sequences in the group at the same time. Figure 3 As shown, the feature extraction network A includes 12 computing layers, which are numbered L0, ..., L11. Each of the 12 computing layers takes the visual features output by the previous computing layer as input, and each computing layer inputs the visual features output by itself to the next computing layer. Among them, the visual feature refers to the feature information of the visual modal information. In addition, the feature information of the speech modal information can be recorded as speech features, and the feature information of the text modal information can be recorded as text features. In some embodiments, the feature extraction network A can also be called a visual network, which can use an open source video sliding window pre-order codec predictor (video swin transformer) to extract three-dimensional visual features.
[0066] like Figure 3 As shown, the visual input V can be used as the first modal information, and the text input T can be used as the second modal information. The text input represents the text information contained in the multimedia information, and the text information can be the title and description of the multimedia information, as well as the text information recognized by optical character recognition (OCR) and automatic speech recognition (ASR). Among them, OCR is an algorithm for extracting text from images, and ASR is an algorithm for converting speech recognition into text. For example, the text in the image contained in the multimedia information is extracted by OCR technology to obtain text information. Alternatively, the voice or audio contained in the multimedia information is converted into text information by ASR technology. As shown in FIG. Figure 3 As shown, the text input T is input into the feature extraction network C, which can also be called the text network. Similarly, the feature extraction network C also includes 12 computing layers, which are numbered L0, ..., L11.
[0067] For example, the text input T is used as the target modal information, and the L9 layer in the feature extraction network C is a preset calculation layer. The L9 layer of the feature extraction network C performs a fusion process on the text features and the visual features. The fusion process includes the following steps: Figure 3 Before introducing the four steps, let’s first introduce how Figure 3 The process of determining the implicit feature latent9 is shown.
[0068] For example, Figure 3The feature extraction network A, feature extraction network C, and classifier shown in the figure constitute the overall machine learning model. The process of risk identification of multimedia information described in this embodiment can be the application stage or the reasoning stage of the machine learning model. Before the application stage or the reasoning stage, the machine learning model needs to be trained. The purpose of the training is to determine the following: Figure 3 The implicit feature latent9 shown in FIG. 1 can be used as a parameter of the machine learning model. Before training, the implicit feature latent9 can be initialized. Specifically, the implicit feature latent9 can include N implicit feature vectors, such as Figure 3 As shown, N=4. During the initialization process, the value in each implicit feature vector in the implicit feature latent9 can be controlled to obey a preset distribution, and the mean of the preset distribution is 0. After the implicit feature latent9 is initialized, the machine learning model is trained. During the training process, the implicit feature latent9 is continuously updated. When the training is completed, the implicit feature latent9 is determined for use in the application phase or the inference phase of the machine learning model.
[0069] Specifically, in the application phase or inference phase of the machine learning model, such as Figure 3 Step ① shown in the figure represents cascading and merging the visual features output by the L8 layer in the visual network and the implicit features latent9 determined in the training phase. The cascade merging is, for example, concatenation. The implicit feature latent9 includes 4 implicit feature vectors. Assume that the visual features include 50 vectors. After cascading and merging, a matrix containing 54 vectors is obtained, for example Figure 4 41 shown. Further, the matrix 41 containing 54 vectors obtained after cascading step ① is input into the L9 layer in the visual network. The L9 layer in the visual network can perform Transformer calculation on the visual features and implicit features latent9 in the matrix 41. For example, the L9 layer in the visual network can fuse and exchange information between the visual features and implicit features latent9 in the matrix 41, thereby outputting an updated matrix 42, which includes the updated visual features and the updated implicit features. The updated visual features contain all or part of the information in the implicit feature latent9. Contains all or part of the information in the visual features output by the visual network L8 layer.
[0070] In step ②, the updated implicit features It is extracted from the matrix 42, leaving the updated visual features.
[0071] In step ③, the updated implicit features The text features output by the L8 layer in the text network are cascaded and merged, and the cascaded and merged results are input to the L9 layer in the text network. Similarly, the L9 layer in the text network performs cascade merge on the text features and implicit features. Perform information fusion and exchange to output updated text features and updated implicit features Similarly, the updated text features contain implicit features The updated text features contain all or part of the information in the visual features output by the visual network L8 layer, thereby realizing the fusion of text features and visual features in the L9 layer of the text network.
[0072] In addition, the updated implicit features is Based on the update, and the updated implicit features Contains all or part of the information in the text features output by the text network L8 layer. Therefore, the updated implicit features It contains not only all or part of the visual features output by the visual network L8 layer, but also all or part of the text features output by the text network L8 layer. In other words, implicit features Contains such as Figure 3 Characteristic information of each of the two modal information shown.
[0073] like Figure 3 As shown in the figure, if the text input T is used as the target modal information and the visual input V is not the target modal information, the text features and the visual features can be fused at the L9 layer of the feature extraction network C, that is, the L10 layer of the feature extraction network A does not perform multi-modal feature fusion. If the text input T and the visual input V are used as the target modal information respectively, the text features and the visual features can be fused not only at the L9 layer of the feature extraction network C, but also at the preset calculation layer of the feature extraction network A. For example, the preset calculation layer of the feature extraction network A is the L10 layer.
[0074] like Figure 3 As shown, in step ④, the updated visual features and updated implicit features output by the visual network L9 layer are Perform cascade merging and input the cascade merging result to the L10 layer in the visual network, so that the L10 layer in the visual network can update the updated visual features and updated implicit features output by the L9 layer of the visual network. Perform information fusion. Since the updated implicit features It not only contains all or part of the information in the visual features output by the visual network L8 layer, but also contains all or part of the information in the text features output by the text network L8 layer. Therefore, the L10 layer in the visual network can fuse the visual features and the text features, so that the feature information output by the L10 layer in the visual network includes not only the visual features, but also all or part of the information in the text features.
[0075] like Figure 3 As shown, when the text input T and the visual input V are respectively used as the target modal information, there is a preset computing layer in the visual network and the text network respectively. In some other embodiments, the same network may also include multiple preset computing layers, and the multiple preset computing layers in the same network may be adjacent or non-adjacent. For example Figure 5 As shown in Figure 1, there is one preset computing layer in the visual network, such as L10. There are two preset computing layers in the text network, such as L9 and L10. Specifically, implicit features Can be used as the initialization implicit feature latent corresponding to the L10 layer of the visual network 10 , thus the implicit feature The text features and visual features contained in the L9 layer of the visual network are passed to the L10 layer of the visual network. The output of the L10 layer of the visual network includes the updated visual features and The calculation process is similar to The calculation process of , further, in the L10 layer of the text network The updated text features calculated by the L9 layer of the text network are fused, and the output of the L10 layer of the text network includes the updated text features calculated by the L10 layer and The calculation process is similar to The calculation process of is not described here. Figure 5 Steps ① to ④ shown in the figure can be the first round of complete fusion process. Steps ④ to ⑦ are the second round of complete fusion process. Can be used as the initial implicit feature latent required for the second round of fusion 10 In addition, if Figure 5As shown, the first round of fusion process is implemented in the L9 layer of the visual network and the text network, and the second round of fusion process is implemented in the L10 layer of the visual network and the text network. That is to say, the first round of fusion process and the second round of fusion process are implemented on adjacent computing layers. In some other embodiments, the first round of fusion process and the second round of fusion process can also be implemented on non-adjacent computing layers. For example, the computing layers included in the visual network and the text network are not limited to 12 layers, for example, more than 12 layers. The first round of fusion process is implemented in the L9 layer, and the second round of fusion process is implemented in the L11 layer. Then the first round of fusion results in Can be used as the initial implicit feature latent required for the second round of fusion 11 In addition, similar to Figure 5 The situation shown can be recorded as multi-level multi-modal fusion, where multi-level refers to a multi-round fusion process, and multi-modal fusion means that feature information of at least two modal information can be fused in each round of fusion.
[0076] also, Figure 3 and Figure 5 The example shown is a case of two modal information. In other embodiments, the multimedia information is not limited to including two modal information, and may include three modal information, for example. Figure 6 The text input T and speech input Y shown in the figure are target modal information. The text features and visual features are fused in the L9 layer of the text network, and the speech features, text features and visual features are fused in the L9 layer of the speech network, i.e., the feature extraction network B. These two fusion processes involve the following: Figure 6 Steps ① to ⑦ are shown. Figure 6 As shown, the multimedia information includes three modal information: visual input V, text input T, and voice input Y. The voice input Y outputs the feature extraction network B. The structure of the feature extraction network B is the same as that of other feature extraction networks. The feature extraction network B can also be called a voice network. Among them, steps ① to ③ are as described above and will not be repeated here. Figure 6 In step ④ shown in the figure, the updated implicit features are extracted from the output of the text network L9 layer Similar to the above Contains all or part of the information in the visual features and all or part of the information in the text features. Figure 6 In step ⑤ shown in the figure, the speech features output by the speech network L8 layer are compared with the updated implicit features Perform cascade merging and input the cascade merging results to the L9 layer of the speech network. The L9 layer of the speech network can output updated speech features and updated implicit features. Thus, the updated speech features can include all or part of the information in the visual features and all or part of the information in the text features, and the updated implicit features It may include feature information of each of the three modal information: visual input V, text input T, and voice input Y. Figure 6 In step ⑥ shown in the figure, the updated implicit features are extracted from the output of the L9 layer of the speech network If the visual input V is not the target modality information, the L10 layer of the visual network does not perform multi-modal feature fusion. If the visual input V is also the target modality information, then Figure 6 In step ⑦ shown in the figure, the updated visual features and updated implicit features output by the visual network L9 layer are The cascaded merge is performed, and the result of the cascaded merge is input into the L10 layer in the visual network, so that the L10 layer in the visual network can fuse the feature information of each of the three modal information of visual input V, text input T, and voice input Y.
[0077] In addition, in the scenario of three modal information, the complete fusion process between the feature information of these three modal information is not limited to Figure 6 The round from step ① to step ⑦ shown in the figure can be repeated multiple times. The multiple rounds of fusion process can be referred to Figure 5 The process shown will not be repeated here. It can be understood that in the second round and the fusion process after the second round, the preset computing layer in each network can fuse the feature information of the three modal information. Thereby, the feature information output by the last computing layer of each network in the visual network, the text network, and the voice network can simultaneously include the feature information of the three modal information. However, the visual features account for the largest proportion of the feature information output by the last computing layer of the visual network, and the text features account for the largest proportion of the feature information output by the last computing layer of the text network. Similarly, the voice features account for the largest proportion of the feature information output by the last computing layer of the voice network.
[0078] It is understandable that not every feature information of each modality information needs to be involved in the fusion. For example, when the visual input V and the text input T are respectively used as the target modality information, the visual features and the text features are involved in the fusion, but the speech features are not involved in the fusion. At this time, the feature information output by the last computing layer of each network in the visual network and the text network can include both visual features and text features. However, the feature information output by the last computing layer of the speech network only includes speech features.
[0079] In summary, the various implicit features in the embodiments of the present disclosure can serve as a medium for information exchange between modalities.
[0080] Optionally, before performing risk assessment on the multimedia information based on the feature information output by the last computing layer in each feature extraction network, the method further includes: inputting part of the information in the fusion result obtained by the fusion processing into the next computing layer of the preset computing layer.
[0081] like Figure 3 As shown, after the L9 layer in the text network fuses the text features and the visual features, a fusion result is obtained, which includes the updated text features calculated by the L9 layer in the text network and the implicit features. Furthermore, the updated text features are input into the L0 layer of the text network so that the subsequent computing layers can continue to perform calculations. Similarly, if the L10 layer of the visual network does not perform fusion processing, the L9 layer of the visual network inputs the updated visual features calculated by it into the L10 layer of the visual network so that the subsequent computing layers can continue to perform calculations.
[0082] Optionally, the fusion result includes feature information after the modal information is updated; and part of the information in the fusion result obtained by the fusion processing is input into the next calculation layer of the preset calculation layer, including: inputting the feature information after the modal information is updated into the next calculation layer of the preset calculation layer, and the feature information after the modal information is updated includes the implicit feature.
[0083] like Figure 3 As shown, if the L10 layer corresponding to the visual input V has implicit features After being fused with the visual features output by the visual network L9 layer, the fusion result includes the updated visual features. Further, the updated visual features output by the L10 layer can be input into the next calculation layer L11 layer. For example, the updated visual features output by the L10 layer can include implicit features
[0084] Optionally, the fusion result also includes updated implicit features, and the method also includes: inputting the updated implicit features into a preset computing layer in a feature extraction network corresponding to the next target modal information, and the updated implicit features include feature information output by the previous computing layer.
[0085] like Figure 5 As shown, if the L10 layer corresponding to the visual input V has implicit features After being fused with the visual features output by the visual network L9 layer, the fusion result includes the updated visual features and the updated implicit features Furthermore, the updated visual features output by the L10 layer can be input into the next computational layer, namely the L11 layer of the visual network. Input the next target modality information, such as the preset calculation layer corresponding to the text input T, for example, the L10 layer in the text network. Among them, the updated implicit feature Includes the visual features output by the L9 layer of the vision network.
[0086] For example Figure 6 As shown, the L9 layer in the text network can be used for implicit features It is fused with the text features output by the L8 layer in the text network. Includes feature information of the visual input V before the text input T.
[0087] In addition, if Figure 6 As shown, the L9 layer in the speech network can be used for implicit features The implicit features are fused with the speech features output by layer L8 in the speech network. Includes feature information of visual input V before voice input Y and feature information of text input T.
[0088] If the target modality information is the last target modality information among the at least one target modality information, then the next target modality information is the first target modality information among the at least one target modality information, and the level of the preset calculation layer in the feature extraction network corresponding to the next target modality information is higher than the level of the preset calculation layer in the feature extraction network corresponding to the target modality information.
[0089] like Figure 6 As shown in Figure 1, it is assumed that the visual input V, text input T and speech input Y are the target modality information respectively, and the speech input Y is the last target modality information. The L9 layer in the speech network is used for implicit features. After being fused with the speech features output by the L8 layer in the speech network, the fusion result includes the updated speech features and the updated implicit features. The updated speech features are input into the L10 layer in the speech network, and the updated implicit features The next target modality information is input into the preset calculation layer of the feature extraction network corresponding to the next target modality information. At this time, the next target modality information is the first target modality information, that is, the visual input V. In addition, the L9 layer in the speech network updates the implicit feature The input is into the L10 layer in the visual network, which is one level higher than the L9 layer.
[0090] If the target modal information is not the last target modal information among the at least one target modal information, the level of the preset computing layer in the feature extraction network corresponding to the next target modal information is the same as the level of the preset computing layer in the feature extraction network corresponding to the target modal information.
[0091] like Figure 6 As shown in the figure, assuming that the visual input V, text input T and voice input Y are the target modality information respectively, and the text input T is not the last target modality information, the L9 layer in the text network is implicit in the feature After being fused with the text features output by the L8 layer in the text network, the fusion result includes the updated text features and the updated implicit features The updated text features are input into the L10 layer in the text network, and the updated implicit features It is input to the preset calculation layer corresponding to the next target modality information, for example, the L9 layer corresponding to the voice input Y, that is, the L9 layer in the text network will update the implicit feature It is input to the L9 layer at the same level in the voice network.
[0092] The disclosed embodiment includes a plurality of preset computing layers through a feature extraction network corresponding to target modal information, thereby being able to perform multiple rounds of fusion of feature information of multiple modal information. Before part of the information in each modal information is lost, sufficient fusion and exchange between the multi-modal information is achieved, thereby further improving the accuracy of identifying potential risks in multimedia information.
[0093] In addition, the present disclosure also provides an information fusion method, which includes the following steps:
[0094] S11. Acquire real world information in the environment surrounding the virtual reality device and virtual world information presented by a display component of the virtual reality device.
[0095] For example, the virtual reality device may be an augmented reality (AR) device. The AR device may obtain real world information in its surrounding environment and virtual world information presented by a display component of the AR device.
[0096] S12: Input the real world information into a first feature extraction network, and input the virtual world information into a second feature extraction network.
[0097] For example, the AR device may input the real world information into a first feature extraction network and input the virtual world information into a second feature extraction network, wherein the first feature extraction network is used to extract features from the real world information and the second feature extraction network is used to extract features from the virtual world information.
[0098] S13. Fusing the feature information of the real world information and the feature information of the virtual world information through a preset computing layer in the first feature extraction network, and / or fusing the feature information of the virtual world information and the feature information of the real world information through a preset computing layer in the second feature extraction network.
[0099] Specifically, the first feature extraction network and the second feature extraction network may be similar to the feature extraction network described above, that is, include multiple computing layers. Among them, the feature information between multiple modalities may be fused in the preset computing layer. For example, the real world information may be used as one modal information, and the virtual world information may be used as another modal information. The preset computing layer in the first feature extraction network may fuse the feature information of the real world information and the feature information of the virtual world information, and / or, the preset computing layer in the second feature extraction network may fuse the feature information of the virtual world information and the feature information of the real world information.
[0100] In addition, the present disclosure also provides an object recognition method, which includes the following steps:
[0101] S21. Acquire a first image, where the first image is obtained of the target object under a first imaging condition.
[0102] S22: Adjust the first imaging condition to obtain a second imaging condition.
[0103] S23. Acquire a second image, where the second image is obtained of the target object under the second imaging condition.
[0104] S24: Determine whether the target object is a physical object according to the first image, the second image, the first imaging condition, and the second imaging condition.
[0105] Optionally, determining whether the target object is a physical object according to the first image, the second image, the first imaging condition, and the second imaging condition comprises the following steps:
[0106] S31. Input the first image into a first feature extraction network, and input the second image into a second feature extraction network.
[0107] S32. The image features of the first image and the image features of the second image are fused through the first feature extraction network to obtain a first group of image features, and / or the image features of the second image and the image features of the first image are fused through the second feature extraction network to obtain a second group of image features.
[0108] S33. Determine one or more target attributes of the target object according to the difference between the first group of image features and the second group of image features, and the difference between the first imaging condition and the second imaging condition, wherein the target attributes are independent of the imaging conditions.
[0109] S34. Determine whether the target object is a physical object according to the one or more target attributes.
[0110] Figure 7 The multimedia information processing device provided by the embodiment of the present disclosure can execute the processing flow provided by the embodiment of the multimedia information processing method, such as Figure 7 As shown, the multimedia information processing device 70 includes:
[0111] An acquisition module 71, used to acquire multiple modal information in the multimedia information, wherein the multiple modal information corresponds one-to-one to multiple feature extraction networks;
[0112] A first input module 72, configured to input each modal information of the plurality of modal information into a feature extraction network corresponding to the modal information;
[0113] A fusion module 73 is used to fuse feature information of at least one target modal information among the multiple modal information with feature information of at least one other modal information through a preset computing layer in a feature extraction network corresponding to the target modal information;
[0114] The risk assessment module 74 is used to perform risk assessment on the multimedia information according to the feature information output by the last computing layer in each feature extraction network.
[0115] Optionally, the fusion module 73 is specifically used for:
[0116] The implicit features and the feature information output by the previous computing layer of the preset computing layer are fused through the preset computing layer in the feature extraction network corresponding to the target modal information, wherein the implicit features include feature information of at least one other modal information, and the feature information output by the previous computing layer includes at least feature information of the target modal information.
[0117] Optionally, the structures of the multiple feature extraction networks are the same, and each feature extraction network includes multiple computing layers; the multimedia information processing device 70 also includes a second input module 75, which is used to input part of the information in the fusion result obtained by the fusion processing into the next computing layer of the preset computing layer before the risk assessment module 74 performs risk assessment on the multimedia information based on the feature information output by the last computing layer in each feature extraction network.
[0118] Optionally, the fusion result includes feature information after the modal information is updated; the second input module 75 is specifically used to: input the feature information after the modal information is updated to the next calculation layer of the preset calculation layer, and the feature information after the modal information is updated includes the implicit feature.
[0119] Optionally, the fusion result also includes updated implicit features, and the second input module 75 is also used to: input the updated implicit features into a preset computing layer in a feature extraction network corresponding to the next target modal information, and the updated implicit features include feature information output by the previous computing layer.
[0120] Optionally, if the target modal information is the last target modal information among the at least one target modal information, the next target modal information is the first target modal information among the at least one target modal information, and the level of the preset computing layer in the feature extraction network corresponding to the next target modal information is higher than the level of the preset computing layer in the feature extraction network corresponding to the target modal information.
[0121] Optionally, if the target modal information is not the last target modal information among the at least one target modal information, the level of the preset computing layer in the feature extraction network corresponding to the next target modal information is the same as the level of the preset computing layer in the feature extraction network corresponding to the target modal information.
[0122] Figure 7 The multimedia information processing device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.
[0123] The internal functions and structure of the multimedia information processing device are described above. The device can be implemented as an electronic device. Figure 8 This is a schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. Figure 8 As shown, the electronic device includes a memory 81 and a processor 82 .
[0124] The memory 81 is used to store programs. In addition to the above-mentioned programs, the memory 81 can also be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.
[0125] The memory 81 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0126] The processor 82 is coupled to the memory 81 and executes the program stored in the memory 81 to:
[0127] Acquire multiple modal information in the multimedia information, wherein the multiple modal information corresponds one-to-one to multiple feature extraction networks;
[0128] Inputting each modal information of the plurality of modal information into a feature extraction network corresponding to the modal information;
[0129] For at least one target modal information among the multiple modal information, feature information of the target modal information and feature information of at least one other modal information are fused through a preset computing layer in a feature extraction network corresponding to the target modal information;
[0130] The risk assessment of the multimedia information is performed according to the feature information output by the last computing layer in each feature extraction network.
[0131] Further, if Figure 8 As shown, the electronic device may also include: a communication component 83, a power component 84, an audio component 85, a display 86 and other components. Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 Components shown.
[0132] The communication component 83 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 83 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 83 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0133] The power supply component 84 provides power to various components of the electronic device. The power supply component 84 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.
[0134] The audio component 85 is configured to output and / or input audio signals. For example, the audio component 85 includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 81 or sent via the communication component 83. In some embodiments, the audio component 85 also includes a speaker for outputting audio signals.
[0135] The display 86 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0136] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the multimedia information processing method described in the above embodiment.
[0137] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0138] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multimedia information processing method, wherein: The method comprises: Acquire multiple modal information in the multimedia information, wherein the multiple modal information corresponds one-to-one to multiple feature extraction networks; Inputting each modal information of the plurality of modal information into a feature extraction network corresponding to the modal information; For at least one target modal information among the multiple modal information, feature information of the target modal information and feature information of at least one other modal information are fused through a preset computing layer in a feature extraction network corresponding to the target modal information to obtain a fusion result, wherein the fusion result includes updated feature information of the target modal information and updated implicit features, wherein the updated feature information of the target modal information includes implicit features before being updated, and the updated implicit features include feature information of at least one other modal information; The risk assessment of the multimedia information is performed according to the feature information output by the last computing layer in each feature extraction network.
2. The method according to claim 1, wherein: The method further comprises: fusing feature information of the target modal information and feature information of at least one other modal information through a preset computing layer in a feature extraction network corresponding to the target modal information, including: The implicit features and the feature information output by the previous computing layer of the preset computing layer are fused through the preset computing layer in the feature extraction network corresponding to the target modal information, and the feature information output by the previous computing layer at least includes the feature information of the target modal information.
3. The method according to claim 2, wherein: The structures of the multiple feature extraction networks are the same, and each feature extraction network includes multiple computing layers; Before performing risk assessment on the multimedia information based on the feature information output by the last computing layer in each feature extraction network, the method further includes: Part of the information in the fusion result obtained by the fusion process is input into the next calculation layer of the preset calculation layer.
4. The method according to claim 3, wherein: Inputting part of the information in the fusion result obtained by the fusion processing into a calculation layer next to the preset calculation layer, comprising: The updated feature information of the target modal information is input into the next calculation layer of the preset calculation layer.
5. The method according to claim 4, wherein: The method further comprises: The updated implicit features are input into a preset computing layer in a feature extraction network corresponding to the next target modal information, and the updated implicit features include feature information output by the previous computing layer.
6. The method according to claim 5, wherein: If the target modality information is the last target modality information among the at least one target modality information, then the next target modality information is the first target modality information among the at least one target modality information, and the level of the preset calculation layer in the feature extraction network corresponding to the next target modality information is higher than the level of the preset calculation layer in the feature extraction network corresponding to the target modality information.
7. The method according to claim 5, wherein: If the target modal information is not the last target modal information among the at least one target modal information, the level of the preset computing layer in the feature extraction network corresponding to the next target modal information is the same as the level of the preset computing layer in the feature extraction network corresponding to the target modal information.
8. A multimedia information processing device, wherein: include: An acquisition module, used to acquire multiple modal information in the multimedia information, wherein the multiple modal information corresponds one-to-one to multiple feature extraction networks; A first input module, used for inputting each modal information of the plurality of modal information into a feature extraction network corresponding to the modal information; A fusion module, for fusing feature information of at least one target modal information among the multiple modal information with feature information of at least one other modal information through a preset computing layer in a feature extraction network corresponding to the target modal information to obtain a fusion result, wherein the fusion result includes the updated feature information of the target modal information and the updated implicit features, wherein the updated feature information of the target modal information includes the implicit features before the update, and the updated implicit features include the feature information of at least one other modal information; The risk assessment module is used to perform risk assessment on the multimedia information according to the feature information output by the last computing layer in each feature extraction network.
9. An electronic device, wherein: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
