Multimodal information extraction method, device, computer equipment and storage medium

Abstract

The present invention discloses a multimodal information extraction method, device, computer equipment and storage medium, comprising: performing type matching on multimodal information to obtain at least two target types; for each target type, performing feature extraction using a feature extraction method corresponding to the target type to obtain a target feature corresponding to the target type; performing feature fusion on all target features to obtain a fusion feature, thereby achieving comprehensive extraction and fusion of data features in multimodal information and improving the accuracy of multimodal information recognition.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multimodal information extraction method, device, computer equipment and medium. Background Art

[0002] With the advent of the artificial intelligence era, smart products have become an important channel for consumers to obtain product information. In daily use scenarios, the interaction between humans and smart assistants generates information in various modalities, including text, voice, image, etc. Therefore, feature extraction of different modal information becomes crucial.

[0003] The development of multimodal feature extraction technology is in a stage of continuous integration of new technologies and continuous evolution to meet the needs of multimodal information feature extraction in various fields and tasks. In the existing multimodal feature extraction methods, only specific types of data are often extracted, which leads to inaccurate information recognition. In addition, the quality and dimensionality imbalance of the data lead to modal noise and modal missing problems, which limits the ability to obtain and analyze comprehensive information.

[0004] Therefore, there is an urgent need for a multimodal information extraction method that is adapted to the input requirements of realistic fusion models in order to improve the recognition accuracy of multimodal information, thereby achieving high efficiency of fusion and enhancing the diversity of fusion information. Summary of the invention

[0005] Embodiments of the present invention provide a multimodal information extraction method, apparatus, computer equipment and storage medium to improve the accuracy of multimodal information recognition.

[0006] In order to solve the above technical problems, the present application provides a multimodal information extraction method, including:

[0007] Perform type matching on multimodal information to obtain at least two target types;

[0008] For each of the target types, feature extraction is performed using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type;

[0009] All target features are fused to obtain fused features.

[0010] Optionally, the target type is at least two of an image type, a text type, a knowledge type, and an audio type.

[0011] Optionally, the target type is an image type, and the feature extraction is performed using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type, including:

[0012] Using a Faster R-CNN object detection model to process an initial image in the multimodal information, converting the initial image into an object sequence;

[0013] Generate a bounding box for each object in the object sequence, wherein the bounding box is used to represent the position information of the object;

[0014] In the object sequence, randomly selecting a preset proportion of objects for masking;

[0015] The masked object sequence is embedded, and each object is embedded as an object by extracting the feature vector of the object;

[0016] Adding position embedding for the position and dimension to obtain a position embedding vector, which is used to encode the position information of the object in the image as the first position information;

[0017] The object embedding and the first position information are added to obtain a comprehensive representation of each object as a target feature corresponding to the image type.

[0018] Optionally, the target type is a text type, and the feature extraction is performed using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type, including:

[0019] Marking the text in the multimodal information as a mark sequence, and randomly masking a preset proportion of the marks;

[0020] Perform embedding operation on the tag sequence. For each tag, the corresponding word vector representation is obtained by searching the pre-trained word vector model as the tag embedding.

[0021] Add position embedding to encode the position information of the token in the sentence as the second position information;

[0022] The tag embedding and the second position information are combined to obtain a comprehensive representation of each tag, and the sum of the tag embedding and the position embedding is input into a Transformer-based text encoder. The text encoder processes the input tag sequence and outputs an initial feature representation of the text as a target feature corresponding to the text type.

[0023] Optionally, the knowledge type is a text type, and the feature extraction is performed using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type, including:

[0024] Segmenting the Chinese text in the multimodal information to obtain a word sequence;

[0025] The word-inverse text frequency index is used to evaluate the importance of each word in the document and obtain the word score;

[0026] Sort the words according to the word scores, and select the word with the highest score as the core keyword to construct a knowledge graph;

[0027] All relations and tail entities of triples in the knowledge graph are concatenated together to form a long knowledge text, and the tag sequence of the knowledge text is encoded using the same text encoder as that used to extract the initial features of the text to obtain the target encoding.

[0028] Optionally, the performing feature fusion on all target features to obtain fused features includes:

[0029] For each of the target features, a self-attention mechanism is used to dynamically assign weights to obtain the weights corresponding to the target features, and the target features are multiplied by the weights corresponding to the target features to obtain weighted features;

[0030] The weighted features are calculated, connected and fused to obtain the fused features.

[0031] Optionally, a self-attention mechanism is used to dynamically allocate weights, and the weights corresponding to the target features include:

[0032] Calculate the attention score of each target feature to obtain an attention score matrix;

[0033] The attention score matrix is ​​normalized to obtain an attention weight matrix.

[0034] In order to solve the above technical problems, the embodiment of the present application further provides a multimodal information extraction device, including:

[0035] A type matching module, used to perform type matching on multimodal information to obtain at least two target types;

[0036] A feature extraction module is used to extract features for each target type using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type;

[0037] The feature fusion module is used to fuse all target features to obtain fused features.

[0038] Optionally, the feature extraction module includes:

[0039] An image conversion unit, configured to process an initial image in the multimodal information using a Faster R-CNN object detection model, and convert the initial image into an object sequence;

[0040] A boundary generating unit, used to generate a boundary box for each object in the object sequence, wherein the boundary box is used to represent the position information of the object;

[0041] A first masking unit, configured to randomly select a preset proportion of objects from the object sequence for masking;

[0042] A first embedding unit is used to perform an embedding operation on the masked object sequence, and each object is embedded as an object by extracting a feature vector of the object;

[0043] A first adding unit, configured to add a position embedding according to a position and a dimension to obtain a position embedding vector, which is used to encode position information of an object in an image as first position information;

[0044] The first superposition unit is used to add the object embedding and the first position information to obtain a comprehensive representation of each object as a target feature corresponding to the image type.

[0045] Optionally, the feature extraction module further includes:

[0046] A second masking unit, configured to mark the text in the multimodal information as a tag sequence, and randomly mask a preset proportion of the tags;

[0047] The second embedding unit is used to embed the tag sequence. Each tag is obtained by searching the pre-trained word vector model to obtain the corresponding word vector representation as the tag embedding;

[0048] A second adding unit, used for adding a position embedding, used for encoding position information of the tag in the sentence as second position information;

[0049] The second superposition unit is used to embed the tag and the second position information to obtain a comprehensive representation of each tag, and input the sum of the tag embedding and the position embedding into a Transformer-based text encoder. The text encoder processes the input tag sequence and outputs an initial feature representation of the text as a target feature corresponding to the text type.

[0050] Optionally, the feature extraction module further includes:

[0051] A word segmentation unit, used to segment the Chinese text in the multimodal information to obtain a word sequence;

[0052] A scoring unit, used to evaluate the importance of each word in the document using a word-inverse text frequency index to obtain a word score;

[0053] A knowledge graph construction unit, used to sort the words according to the word scores, and select the word with the highest score as the core keyword to construct the knowledge graph;

[0054] The text encoding unit is used to concatenate all relations and tail entities of triples in the knowledge graph to form a long knowledge text, and encode the tag sequence of the knowledge text using the same text encoder as that used to extract the initial features of the text to obtain the target encoding.

[0055] Optionally, the feature fusion module includes:

[0056] A weighting unit, for dynamically allocating weights for each of the target features using a self-attention mechanism to obtain a weight corresponding to the target feature, and multiplying the target feature by the weight corresponding to the target feature to obtain a weighted feature;

[0057] The fusion unit is used to perform calculation connection fusion on the weighted features to obtain the fused features.

[0058] Optionally, the weighting unit includes:

[0059] A score calculation subunit, used to calculate the attention score of each target feature to obtain an attention score matrix;

[0060] The normalization subunit is used to normalize the attention score matrix to obtain an attention weight matrix.

[0061] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above multimodal information extraction method when executing the computer program.

[0062] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above multimodal information extraction method are implemented.

[0063] The multimodal information extraction method, apparatus, computer equipment and storage medium provided in the embodiments of the present invention obtain at least two target types by performing type matching on multimodal information; for each target type, feature extraction is performed using a feature extraction method corresponding to the target type to obtain a target feature corresponding to the target type; feature fusion is performed on all target features to obtain a fused feature, thereby achieving comprehensive extraction and fusion of data features in multimodal information and improving the accuracy of multimodal information recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0065] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0066] Figure 2 is a flow chart of an embodiment of the multimodal information extraction method of the present application;

[0067] Figure 3 is a structural schematic diagram of an embodiment of a multimodal information extraction device according to the present application;

[0068] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0070] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] See also Figure 1 ,like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0073] Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0074] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic image experts compression standard audio layer), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image experts compression standard audio layer) players, laptop computers and desktop computers, etc.

[0075] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0076] It should be noted that the multimodal information extraction method provided in the embodiment of the present application is executed by a server, and accordingly, the multimodal information extraction device is arranged in the server.

[0077] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0078] See also Figure 2 , Figure 2 A multimodal information extraction method provided by an embodiment of the present invention is shown. The method is applied in Figure 1 The server in is used as an example to illustrate the details as follows:

[0079] S201: Perform type matching on multimodal information to obtain at least two target types.

[0080] Furthermore, the target type is at least two of an image type, a text type, a knowledge type, and an audio type.

[0081] S202: For each target type, feature extraction is performed using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type.

[0082] In a specific optional implementation, in step S202, the target type is an image type, and feature extraction is performed using a feature extraction method corresponding to the target type, and the target features corresponding to the target type are obtained, including:

[0083] Use the Faster R-CNN object detection model to process the initial image in the multimodal information and convert the initial image into an object sequence;

[0084] Generate a bounding box for each object in the object sequence, and the bounding box is used to represent the location information of the object;

[0085] In the object sequence, a preset proportion of objects are randomly selected for masking;

[0086] The masked object sequence is embedded, and each object is embedded as an object by extracting the feature vector of the object;

[0087] Adding position embedding for the position and dimension to obtain a position embedding vector for encoding the position information of the object in the image as the first position information;

[0088] The object embedding and the first position information are added to obtain a comprehensive representation of each object as the target feature corresponding to the image type.

[0089] Specifically, first, the Faster R-CNN object detection model is used to process the given image and convert the given image into an object sequence. The object detection model detects a series of objects in the given image and generates a bounding box for each object to represent the location information of the object. In the object sequence, 15% of the objects are randomly selected for masking. This masking operation is to introduce a masking training task when training the model so that the model can learn to deal with the situation where some object information is missing. Then, the masked object sequence is embedded, and each object is extracted by extracting the feature vector of the object. Embedded as an object, For the i The feature vector corresponding to the object. At the same time, for the position and dimensions , get the position embedding vector , used to encode the location information of the object in the image, where the location and dimensions Respectively i The position and dimension of each object. Then, embed the object and position embedding Add them together to get a comprehensive representation of each object. Finally, these comprehensive representations are input into the image encoder Vision Transformer, which outputs the initial features of the image. ,in is the maximum object sequence length.

[0090] In a specific optional implementation, in step S202, the target type is a text type, and feature extraction is performed using a feature extraction method corresponding to the target type, and the target features corresponding to the target type are obtained, including:

[0091] Mark the text in the multimodal information into a tag sequence, and randomly mask a preset proportion of tags;

[0092] Perform embedding operation on the tag sequence. For each tag, the corresponding word vector representation is obtained by searching the pre-trained word vector model as the tag embedding.

[0093] Add position embedding to encode the position information of the token in the sentence as the second position information;

[0094] The token embedding and the second position information are combined to obtain a comprehensive representation of each token. The sum of the token embedding and the position embedding is input into the Transformer-based text encoder. The text encoder processes the input token sequence and outputs the initial feature representation of the text as the target feature corresponding to the text type.

[0095] Specifically, first, the text is marked as a sequence of tokens by WordPieces, and 15% of the tokens are randomly masked. Then the token sequence is embedded, and each token is represented by the corresponding word vector by searching the pre-trained Word2Vec model. As a marker embed, Represents the word vector corresponding to the tag. At the same time, position embedding is added to encode the position information of the tag in the sentence Then, we add the token embedding and the position embedding to get a comprehensive representation of each token. Input to the Transformer-based text encoder, which processes the input token sequence and outputs the initial feature representation of the text .in, is the serial number of the sequence composed of tags, is the maximum length of a token sequence. The maximum length of a token sequence refers to the maximum length of a token sequence after word segmentation and tagging. If the sequence exceeds the maximum length, truncation or padding can be performed.

[0096] In a specific optional implementation, in step S202, the knowledge type is a text type, and feature extraction is performed using a feature extraction method corresponding to the target type, and the target features corresponding to the target type are obtained, including:

[0097] Segment the Chinese text in the multimodal information to obtain word sequences;

[0098] The word-inverse text frequency index is used to evaluate the importance of each word in the document and obtain the word score;

[0099] Sort the words according to their scores, and select the words with the highest scores as core keywords to construct the knowledge graph;

[0100] All relations and tail entities of triples in the knowledge graph are concatenated together to form a long knowledge text, and the tag sequence of the knowledge text is encoded using the same text encoder as that used to extract the initial features of the text to obtain the target encoding.

[0101] Specifically, the knowledge graph can collect and integrate a large amount of related information, which can make the header entity information more comprehensive and accurate, and provide users with more detailed product understanding and selection. Therefore, the core keywords are extracted based on the input text information and the product knowledge graph is constructed to obtain the triples in the knowledge graph.

[0102] First, use the word segmentation tool jieba to segment the given Chinese text, use the default precise mode words = jieba.lcut(sentence) to convert the text into a word sequence, and use Baidu's stop word list to filter stop words. Then, use TF-IDF to evaluate the importance of a word in a document. TF (term frequency) refers to the frequency of a word appearing in a document. The higher the TF, the more times the word appears in the document and the more important it is. Among them, , Represents words in text The number of occurrences, Represents text The number of all words in the document; IDF (Inverse Document Frequency) refers to the inverse of the frequency of a word appearing in all documents. If a word appears frequently in all documents, its IDF will be very low, indicating that this word is not very useful in distinguishing documents. , Indicates the total number of texts, Indicates that it contains words Finally, according to the comprehensive score Sort the words and select the words with the highest scores as core keywords to construct the knowledge graph.

[0103] Because the triple The surface form of the head entity overlaps with the text content and has no semantics, so all the relations of the triples and the tail entity are spliced ​​together to form a long knowledge text. , and the same text encoder used to extract the initial features of the text is used to encode the token sequence of the knowledge text. The text encoder embeds Output ,in, is the serial number of each knowledge text in the tag sequence of the knowledge text, is the maximum token sequence length of the knowledge text.

[0104] Optionally, when the target type is audio type, Mel-Frequency Cepstral Coefficients (MFCC) are used to extract speech features from the speech in the multimodal information. First, the audio signal is converted into a digital signal, and each frame of the speech signal is Perform N-point FFT transformation to obtain spectrum information ,in, .

[0105] Then, by Find the power spectrum , using the Mel frequency filter bank Find the logarithmic energy spectrum signal ,in represents a filter bank and M represents a frequency grouping.

[0106] Finally, the energy spectrum signal is transformed by using the discrete cosine transform (DCT) method. Perform cepstrum processing and find the MFCC cepstrum coefficients of order i , , the value of i is usually in the range of 12 to 16. In order to better describe the correlation between the signals of each frame, based on the cepstrum coefficient Calculate the cepstral difference coefficients , get the MFCC first-order difference coefficient, k is the difference parameter, with a value of 2. Then the first-order difference coefficient is substituted into the formula and calculated again to obtain the second-order difference coefficient. Finally, the 24-dimensional MFCC feature parameter is obtained. , thus completing the feature extraction, where, is the label number of the sequence composed of feature parameters, is the maximum token sequence length.

[0107] S203: Perform feature fusion on all target features to obtain fused features.

[0108] In a specific optional implementation, in step S203, all target features are fused to obtain fused features including:

[0109] For each target feature, the self-attention mechanism is used to dynamically assign weights to obtain the weight corresponding to the target feature, and the target feature is multiplied by the weight corresponding to the target feature to obtain the weighted feature;

[0110] The weighted features are calculated, connected and fused to obtain fused features.

[0111] In a specific optional implementation, a self-attention mechanism is used to dynamically allocate weights, and the weights corresponding to the target features include:

[0112] Calculate the attention score of each target feature and obtain the attention score matrix;

[0113] The attention score matrix is ​​normalized to obtain the attention weight matrix.

[0114] In this embodiment, type matching is performed on multimodal information to obtain at least two target types; for each target type, feature extraction is performed using a feature extraction method corresponding to the target type to obtain a target feature corresponding to the target type; feature fusion is performed on all target features to obtain a fused feature, thereby achieving comprehensive extraction and fusion of data features in multimodal information and improving the accuracy of multimodal information recognition.

[0115] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0116] Figure 3 FIG. 2 shows a principle block diagram of a multimodal information extraction device corresponding to the multimodal information extraction method of the above embodiment. Figure 3 As shown, the multimodal information extraction device includes a type matching module 31, a feature extraction module 32 and a feature fusion module 33. The detailed description of each functional module is as follows:

[0117] A type matching module 31 is used to perform type matching on the multimodal information to obtain at least two target types;

[0118] A feature extraction module 32 is used to extract features for each target type using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type;

[0119] The feature fusion module 33 is used to fuse all target features to obtain fused features.

[0120] Optionally, the feature extraction module 32 includes:

[0121] An image conversion unit, used to process the initial image in the multimodal information using the Faster R-CNN object detection model, and convert the initial image into an object sequence;

[0122] A boundary generation unit, used for generating a bounding box for each object in the object sequence, where the bounding box is used for representing the position information of the object;

[0123] A first masking unit, used for randomly selecting a preset proportion of objects in the object sequence for masking;

[0124] A first embedding unit is used to perform an embedding operation on the masked object sequence, and each object is embedded as an object by extracting a feature vector of the object;

[0125] A first adding unit, used for adding position embedding according to the position and dimension to obtain a position embedding vector, which is used for encoding position information of the object in the image as the first position information;

[0126] The first superposition unit is used to add the object embedding and the first position information to obtain a comprehensive representation of each object as a target feature corresponding to the image type.

[0127] Optionally, the feature extraction module 32 further includes:

[0128] A second masking unit is used to mark the text in the multimodal information as a tag sequence and randomly mask a preset proportion of the tags;

[0129] The second embedding unit is used to embed the tag sequence. Each tag is obtained by searching the pre-trained word vector model to obtain the corresponding word vector representation as the tag embedding;

[0130] A second adding unit, used for adding a position embedding, used for encoding position information of the tag in the sentence as second position information;

[0131] The second superposition unit is used to embed the tag and the second position information to obtain a comprehensive representation of each tag, and input the sum of the tag embedding and the position embedding into the Transformer-based text encoder. The text encoder processes the input tag sequence and outputs the initial feature representation of the text as the target feature corresponding to the text type.

[0132] Optionally, the feature extraction module 32 further includes:

[0133] A word segmentation unit is used to segment Chinese text in multimodal information to obtain a word sequence;

[0134] A scoring unit, used to evaluate the importance of each word in the document using a word-inverse text frequency index to obtain a word score;

[0135] A sorting unit is used to sort the words according to their scores and select the words with the highest scores as core keywords to construct a knowledge graph;

[0136] The text encoding unit is used to concatenate all relations and tail entities of triples in the knowledge graph to form a long knowledge text, and encode the tag sequence of the knowledge text using the same text encoder as that used to extract the initial features of the text to obtain the target encoding.

[0137] Optionally, the feature fusion module 33 includes:

[0138] The weighting unit is used to dynamically assign weights to each target feature using the self-attention mechanism to obtain the weight corresponding to the target feature, and multiply the target feature by the weight corresponding to the target feature to obtain the weighted feature;

[0139] The fusion unit is used to calculate the connection and fusion of weighted features to obtain fused features.

[0140] Optionally, the weighting unit includes:

[0141] The score calculation subunit is used to calculate the attention score of each target feature and obtain the attention score matrix;

[0142] The normalization subunit is used to normalize the attention score matrix to obtain the attention weight matrix.

[0143] For the specific definition of the multimodal information extraction device, please refer to the definition of the multimodal information extraction method above, which will not be repeated here. Each module in the above-mentioned multimodal information extraction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0144] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0145] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0146] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0147] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the multimodal information extraction method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0148] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the multimodal information extraction method.

[0149] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0150] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the multimodal information extraction method as described above.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0152] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multimodal information extraction method, characterized in that: include: Performing type matching on the multimodal information to obtain at least two target types, wherein one target type is a text type, and the other target type is at least one of an image type, a knowledge type, and an audio type; For each of the target types, feature extraction is performed using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type, wherein when the target type is a text type, a knowledge graph is constructed based on the text content, and knowledge features are extracted from the knowledge graph; Performing feature fusion on each of the target features and the knowledge features to obtain fused features; Wherein, when another of the target types is an image type, the feature extraction method corresponding to the target type is used to extract the target features to obtain the target features corresponding to the target type, including: Using a Faster R-CNN object detection model to process an initial image in the multimodal information, converting the initial image into an object sequence; Generate a bounding box for each object in the object sequence, wherein the bounding box is used to represent the position information of the object; In the object sequence, randomly selecting a preset proportion of objects for masking; The masked object sequence is embedded, and each object is embedded as an object by extracting the feature vector of the object; Adding position embedding for the position and dimension to obtain a position embedding vector for encoding the position information of the object in the image as the first position information; The object embedding and the first position information are added to obtain a comprehensive representation of each object as a target feature corresponding to the image type.

2. The multimodal information extraction method according to claim 1, characterized in that: The feature fusion of each of the target features and the knowledge features to obtain the fused features includes: For each of the target features, a self-attention mechanism is used to dynamically assign weights to obtain the weights corresponding to the target features, and the target features are multiplied by the weights corresponding to the target features to obtain weighted features; The weighted features are calculated, connected and fused to obtain the fused features.

3. The multimodal information extraction method according to claim 2, characterized in that: The self-attention mechanism is used to dynamically allocate weights, and the weights corresponding to the target features include: Calculate the attention score of each target feature to obtain an attention score matrix; The attention score matrix is ​​normalized to obtain an attention weight matrix.

4. A multimodal information extraction device, characterized in that: The multimodal information extraction method according to any one of claims 1 to 3 comprises: A type matching module, used for performing type matching on multimodal information to obtain at least two target types, one of which is a text type; A feature extraction module is used to extract features for each target type using a feature extraction method corresponding to the target type to obtain target features corresponding to the target type, wherein when the target type is a text type, a knowledge graph is constructed based on the text content, and knowledge features are extracted from the knowledge graph; The feature fusion module is used to fuse each of the target features and the knowledge features to obtain a fused feature.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multimodal information extraction method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal information extraction method according to any one of claims 1 to 3 is implemented.

Images