Entity attribute relation identification method, device and equipment for multi-modal data
Through the fusion of multimodal data and fine-tuning of LoRA strategy, the complexity and inefficiency of multi-language multimodal data are solved, and the recognition accuracy of entity attribute relationships is improved.
Patent Information
- Application Number
- CN202510351195.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-24
- Publication Date
- 2025-08-15
AI Technical Summary
The existing entity attribute relationship recognition technology is difficult to effectively process multilingual and multimodal data, resulting in limited recognition accuracy.
By obtaining multimodal data (text, images, video), using a pre-constructed entity attribute relationship recognition model for identification, and fine-tuning the initial language model using LoRA strategy to generate a unified multimodal representation.
It significantly improves the recognition accuracy of multilingual multimodal data, improves the training efficiency of the model and the accuracy and reliability of the recognition results.
Smart Images
Figure CN120492995A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, and device for identifying entity attribute relationships in multimodal data. Background Art
[0002] With the increasing diversification and complexity of scientific and technological intelligence, traditional single-modal data processing methods are no longer able to meet the processing needs of multilingual and multimodal data. Existing entity attribute relationship recognition technologies mainly focus on processing data in a single language or a single modality, and are unable to cope with the complexity and diversity of multilingual and multimodal data. In the field of scientific and technological intelligence, in particular, data sources are extensive and diverse, covering multiple modalities such as text, images, audio, and video. These different modal data often complement each other, and single-modal processing methods are prone to losing key information, resulting in limited recognition accuracy.
[0003] Therefore, how to effectively integrate multimodal information becomes a key challenge to improve the accuracy of entity attribute relationship recognition. Summary of the Invention
[0004] The present application provides a method, apparatus, and device for identifying entity attribute relationships in multimodal data to solve the problems of complexity and inefficiency in identifying entity attribute relationships in multimodal data.
[0005] The first aspect of the present application provides a method for identifying entity attribute relationships in multimodal data, comprising the following steps: obtaining current multimodal data, wherein the current multimodal data is multimodal data including text, images, and videos; inputting the current multimodal data into a pre-constructed entity attribute relationship recognition model to identify entities, attributes, and relationships, thereby obtaining entity attribute relationships of the current multimodal data, wherein the pre-constructed entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data.
[0006] Optionally, before inputting the current multimodal data into a pre-built entity attribute relationship recognition model for identifying entities, attributes and relationships, it includes: obtaining initial multimodal data, and pre-processing the initial multimodal data to obtain pre-processed multimodal data; fusing the pre-processed multimodal data to obtain multimodal data to be trained; using the multimodal data to be trained to train the initial language model, and using a preset LoRA strategy to fine-tune the trained initial language model to obtain the pre-built entity attribute relationship recognition model.
[0007] Optionally, when the initial language model is trained using the multimodal data to be trained, and the trained initial language model is fine-tuned using the preset LoRA strategy to obtain the pre-constructed entity attribute relationship recognition model, it also includes: dividing the multimodal data to be trained into a training set, a test set and a validation set according to a preset ratio; using the training set to train the initial language model, and using the preset LoRA strategy to fine-tune the trained initial language model, using the validation set to verify the accuracy of entity extraction, attribute extraction and relationship extraction of the fine-tuned initial language model, and after the accuracy is greater than the preset value, using the test set to test whether the fine-tuned initial language model meets the preset entity attribute relationship consistency condition; if the fine-tuned initial language model meets the preset entity attribute relationship consistency condition, the pre-constructed entity attribute relationship recognition model is obtained; otherwise, the pre-constructed entity attribute relationship recognition model is obtained after deleting the entities, attributes or relationships that do not meet the preset entity attribute relationship consistency condition.
[0008] Optionally, the fusion of preprocessed multimodal data to obtain multimodal data to be trained includes: using a pretrained language model to encode text data in the preprocessed multimodal data to generate a text embedding representation; using a pretrained image encoder to extract image features in the preprocessed multimodal data to generate a video representation; using an audio encoder to extract audio features in the preprocessed multimodal data to generate an audio representation; and splicing the text embedding representation, the video representation and the audio representation to obtain the multimodal data to be trained.
[0009] A second aspect of the present application provides an entity attribute relationship identification device for multimodal data, including: an acquisition module for acquiring current multimodal data, wherein the current multimodal data is multimodal data including text, images, and videos; an identification module for inputting the current multimodal data into a pre-constructed entity attribute relationship identification model to identify entities, attributes, and relationships, and obtain the entity attribute relationship of the current multimodal data, wherein the pre-constructed entity attribute relationship identification model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data.
[0010] Optionally, before inputting the current multimodal data into a pre-built entity attribute relationship recognition model for entity, attribute and relationship recognition, the recognition module is also used to: obtain initial multimodal data, and pre-process the initial multimodal data to obtain pre-processed multimodal data; fuse the pre-processed multimodal data to obtain multimodal data to be trained; use the multimodal data to be trained to train the initial language model, and use a preset LoRA strategy to fine-tune the trained initial language model to obtain the pre-built entity attribute relationship recognition model.
[0011] Optionally, when the initial language model is trained using the multimodal data to be trained, and the trained initial language model is fine-tuned using a preset LoRA strategy to obtain the pre-constructed entity attribute relationship recognition model, the recognition module is also used to: divide the multimodal data to be trained into a training set, a test set and a validation set according to a preset ratio; train the initial language model using the training set, fine-tune the trained initial language model using a preset LoRA strategy, and verify the accuracy of entity extraction, attribute extraction and relationship extraction of the fine-tuned initial language model using the validation set, and after the accuracy is greater than a preset value, use the test set to test whether the fine-tuned initial language model meets the preset entity attribute relationship consistency condition; if the fine-tuned initial language model meets the preset entity attribute relationship consistency condition, the pre-constructed entity attribute relationship recognition model is obtained; otherwise, the pre-constructed entity attribute relationship recognition model is obtained after deleting the entities, attributes or relationships that do not meet the preset entity attribute relationship consistency condition.
[0012] Optionally, the recognition module is further used to: encode the text data in the preprocessed multimodal data using a pretrained language model to generate a text embedding representation; extract image features in the preprocessed multimodal data using a pretrained image encoder to generate a video representation; extract audio features in the preprocessed multimodal data using an audio encoder to generate an audio representation; and splice the text embedding representation, the video representation and the audio representation to obtain the multimodal data to be trained.
[0013] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the entity attribute relationship identification method for multimodal data as described in the above embodiment.
[0014] The fourth aspect of the present application provides a computer program product having a computer program stored thereon, which is executed by a processor to implement the entity attribute relationship identification method of multimodal data as described in the above embodiment.
[0015] In the above embodiment, the current multimodal data is obtained, wherein the current multimodal data is multimodal data including text, images, and videos, and the current multimodal data is input into a pre-built entity attribute relationship recognition model to identify entities, attributes, and relationships, thereby obtaining the entity attribute relationship of the current multimodal data. The pre-built entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data. Thus, the complexity and inefficiency of multimodal data in entity attribute relationship recognition are solved, and the recognition accuracy of entity attribute relationships in multilingual multimodal scientific and technological intelligence can be effectively improved, which has broad application prospects.
[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0018] Figure 1 A flowchart of a method for identifying entity attribute relationships in multimodal data according to an embodiment of the present application;
[0019] Figure 2 Flowchart of a method for identifying entity attribute relationships in multimodal data according to one embodiment of the present application;
[0020] Figure 3 1 is an example diagram of a device for identifying entity attribute relationships of multimodal data according to an embodiment of the present application;
[0021] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0022] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0023] The following describes the entity attribute relationship recognition method, device and equipment of the multimodal data of the embodiment of the present application with reference to the accompanying drawings. In response to the complexity and inefficiency of the multimodal data in the entity attribute relationship recognition mentioned in the above background technology, the present application provides a method for entity attribute relationship recognition of multimodal data, in which the current multimodal data is obtained, wherein the current multimodal data is multimodal data including text, image and video, and the current multimodal data is input into a pre-built entity attribute relationship recognition model to recognize entities, attributes and relationships, so as to obtain the entity attribute relationship of the current multimodal data, wherein the pre-built entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data. Thus, the complexity and inefficiency of the multimodal data in the entity attribute relationship recognition is solved, and the recognition accuracy of the entity attribute relationship in the multilingual multimodal scientific and technological intelligence can be effectively improved, which has a broad application prospect.
[0024] Specifically, Figure 1 A flowchart of a method for identifying entity attribute relationships in multimodal data provided in an embodiment of the present application.
[0025] like Figure 1 As shown, the entity attribute relationship recognition method of multimodal data includes the following steps:
[0026] In step S101 , current multimodal data is acquired, wherein the current multimodal data is multimodal data including text, image, and video.
[0027] Specifically, the current multimodal data of the target web page, such as text, image, video, etc., is obtained through a crawler program. The current multimodal data includes multimodal data such as title, content, picture, video, etc.
[0028] In step S102, the current multimodal data is input into a pre-built entity attribute relationship recognition model to identify entities, attributes and relationships, and obtain the entity attribute relationship of the current multimodal data, wherein the pre-built entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data.
[0029] Specifically, the pre-built entity attribute relationship recognition model is used to identify entities, attributes and relationships in the current multimodal data, and the recognition results of entities, attributes and relationships are output.
[0030] Optionally, in some embodiments, before inputting the current multimodal data into a pre-built entity attribute relationship recognition model for identifying entities, attributes and relationships, it includes: obtaining initial multimodal data, and pre-processing the initial multimodal data to obtain pre-processed multimodal data; fusing the pre-processed multimodal data to obtain multimodal data to be trained; using the multimodal data to be trained to train an initial language model, and using a preset LoRA strategy to fine-tune the trained initial language model to obtain a pre-built entity attribute relationship recognition model.
[0031] Specifically, if Figure 2 As shown in the figure, a crawler program is used to obtain initial multimodal data such as text, images, and videos from the target webpage. For this initial multimodal data, noise is removed, the data format is standardized, and text processing techniques are used to further clean the data. The cleaned data is then imported into an annotation platform for cross-annotation and undergoes multiple rounds of review and quality checks.
[0032] According to different task types, corresponding task templates are formulated. The multimodal data format is id and text. The text contains the input text, task prompt, and corresponding entities or relationships, thereby obtaining preprocessed multimodal data.
[0033] Optionally, in some embodiments, the preprocessed multimodal data is fused to obtain the multimodal data to be trained, including: using a pretrained language model to encode the text data in the preprocessed multimodal data to generate a text embedding representation; using a pretrained image encoder to extract image features in the preprocessed multimodal data to generate a video representation; using an audio encoder to extract audio features in the preprocessed multimodal data to generate an audio representation; and splicing the text embedding representation, video representation and audio representation to obtain the multimodal data to be trained.
[0034] In the text data processing stage, the text data is encoded through the pre-trained language model to generate text embedding.
[0035] During the image processing stage, the pre-trained image encoder from BLIP-2 is used to extract independent features from each frame. BLIP-2 is an advanced vision-language pre-trained model that can extract rich visual information from images. First, frame extraction is performed on the video to obtain image data for each frame. Next, the image encoder from BLIP-2 is used to encode each frame, generating a high-dimensional visual feature vector. To preserve the temporal information between frames, the visual features of each frame are combined with positional embedding to generate a video representation with temporal information. This video representation not only contains the visual information of each frame but also captures the temporal relationship between frames, providing high-quality visual input for subsequent multimodal fusion.
[0036] In the audio processing stage, ImageBind is used as the audio encoder to extract the features of the audio clips. ImageBind is a multimodal pre-trained model that can process data in multiple modalities, including audio, images, text, etc. First, multiple two-second audio clips are evenly sampled from the original video to ensure coverage of the entire audio content. Then, each audio clip is encoded using ImageBind's audio encoder to generate an audio feature vector. In order to enhance the expressiveness of audio features, learnable positional embedding is introduced and combined with audio features. Subsequently, the audio features are aggregated through Audio Q-Former to generate a fixed-length audio representation. Audio Q-Former is a Transformer-based module that can effectively capture key information in audio clips and generate audio representations with global context information.
[0037] Further Figure 2 As shown in the figure, the text embedding representation, image representation and audio representation are concatenated to generate multimodal data to be trained.
[0038] The text data is then encoded using a pretrained language model to generate text embeddings. Image and audio data are then processed using BLIP-2 and ImageBind to generate visual and audio embeddings, respectively. These three modal embeddings are then concatenated to form a unified multimodal representation. This multimodal representation not only incorporates the semantic information of the text but also incorporates contextual information from the visual and audio data, more comprehensively reflecting the multimodal nature of the input data. Multimodal fusion fully leverages the complementarity between different modalities and improves the accuracy of entity attribute relationship recognition.
[0039] Optionally, in some embodiments, when the initial language model is trained using the multimodal data to be trained, and the trained initial language model is fine-tuned using a preset LoRA strategy to obtain a pre-constructed entity attribute relationship recognition model, it also includes: dividing the multimodal data to be trained into a training set, a test set, and a validation set according to a preset ratio; using the training set to train the initial language model, and using the preset LoRA strategy to fine-tune the trained initial language model, using the validation set to verify the accuracy of entity extraction, attribute extraction, and relationship extraction of the fine-tuned initial language model, and after the accuracy is greater than the preset value, using the test set to test whether the fine-tuned initial language model meets the preset entity attribute relationship consistency condition; if the fine-tuned initial language model meets the preset entity attribute relationship consistency condition, the pre-constructed entity attribute relationship recognition model is obtained; otherwise, the pre-constructed entity attribute relationship recognition model is obtained after deleting the entities, attributes, or relationships that do not meet the preset entity attribute relationship consistency condition.
[0040] The preset value may be a threshold value pre-set by the user, a threshold value obtained through a limited number of experiments, or a threshold value obtained through a limited number of computer simulations. No specific limitation is made here. The initial language model is a pre-trained Llama model.
[0041] During the fine-tuning phase of the large model, the pre-trained Llama model is fine-tuned using the preset LoRA (Low-Rank Adaptation) strategy. Specifically, LoRA introduces low-rank decomposition into the model's weight matrix, enabling efficient adjustment of model parameters without significantly increasing computing resources. During the fine-tuning process, the dataset is divided into training, validation, and test sets according to a preset ratio, which in this embodiment is 8:1:1.
[0042] Specifically, the Llama model is trained using the training set, and the preset LoRA strategy is used to fine-tune the trained Llama model. During training, the core parameters of LoRA are set to --lora_r=8 and --lora_alpha=32. The training is carried out for 8 cycles, and the intermediate results are recorded every 100 steps. The verification set is used to verify the accuracy of entity extraction, attribute extraction and relationship extraction of the fine-tuned Llama model. After the accuracy is greater than the preset value, the test set is used to test whether the fine-tuned Llama model meets the preset entity attribute relationship consistency condition, that is, to check whether the entity recognized by the fine-tuned Llama model exists in the input data. If not, the corresponding entity or attribute or relationship is deleted to obtain the pre-built entity attribute relationship recognition model, thereby improving the accuracy and reliability of the recognition results.
[0043] If the fine-tuned initial language model meets the preset entity attribute relationship consistency conditions, a pre-constructed entity attribute relationship recognition model is obtained. After experimental evaluation of the pre-constructed entity attribute relationship recognition model in the embodiment of the present application, the entity extraction accuracy is 88.38%, the attribute extraction accuracy is 97.75%, and the relationship extraction accuracy is 94.60%.
[0044] Compared with the prior art, the present invention has the following advantages:
[0045] (1) Multimodal information fusion: By converting multiple modal information such as text, images, and audio into a unified embedded representation, the processing efficiency and recognition accuracy of multimodal data are significantly improved. Traditional single-modal processing methods often fail to fully utilize the complementarity between different modalities, resulting in information loss or poor recognition effects. This application uses multimodal fusion technology to splice the embedded representations of text, images, and audio to generate a comprehensive multimodal representation that can more comprehensively reflect the multimodal characteristics of the input data, thereby improving the accuracy and robustness of entity attribute relationship recognition.
[0046] (2) Multi-language support: The system supports the processing of multi-language data and can handle scientific and technological intelligence data in different languages. Existing entity attribute relationship recognition technologies are mostly limited to single language processing and are difficult to cope with the complexity and diversity of multi-language data in the context of globalization. This application uses multi-language and multi-modal instruction data and pre-trained multi-language models for processing, which can effectively identify and extract entities, attributes and relationships in multiple languages, significantly improving the applicability and scalability of the system in a multi-language environment.
[0047] (3) Efficient fine-tuning: Using LoRA (Low-Rank Adaptation) technology to fine-tune the pre-trained large model can significantly reduce training time and computing resource consumption while ensuring model performance. Traditional model fine-tuning methods usually require adjusting the parameters of the entire model, resulting in large computing resource consumption and long training time. The LoRA strategy of the embodiment of the present application only adjusts parameters in the low-rank subspace of the model, which not only retains the powerful capabilities of the pre-trained model, but also significantly reduces the computational overhead during the fine-tuning process, thereby improving the training efficiency and deployment flexibility of the model.
[0048] (4) By ensuring the consistency of the identified entities, attributes, and relationships, the accuracy and reliability of the recognition results are improved. During the entity attribute relationship recognition process, the model may generate some entities, attributes, and relationships that do not conform to the logical or contextual relationship. The embodiment of the present application verifies and corrects the model to ensure that the identified entities must exist in the input data and deletes attributes or relationships that do not meet the conditions. This mechanism effectively reduces the occurrence of misidentification and improves the accuracy and reliability of the final results.
[0049] According to the entity attribute relationship identification method for multimodal data proposed in the embodiment of the present application, the current multimodal data is obtained, wherein the current multimodal data is multimodal data including text, images, and videos, and the current multimodal data is input into a pre-built entity attribute relationship identification model to identify entities, attributes, and relationships, thereby obtaining the entity attribute relationship of the current multimodal data. The pre-built entity attribute relationship identification model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data. Thus, the complexity and inefficiency of multimodal data in entity attribute relationship identification are solved, and the recognition accuracy of entity attribute relationships in multilingual multimodal scientific and technological intelligence can be effectively improved, which has broad application prospects.
[0050] Next, a device for identifying entity attribute relationships of multimodal data proposed in an embodiment of the present application will be described with reference to the accompanying drawings.
[0051] Figure 3 It is a block diagram of an entity attribute relationship identification device for multimodal data according to an embodiment of the present application.
[0052] like Figure 3 As shown, the entity attribute relationship identification device 10 for multimodal data includes: an acquisition module 100 and an identification module 200.
[0053] Among them, the acquisition module 100 is used to obtain the current multimodal data, wherein the current multimodal data is multimodal data including text, image, and video; the recognition module 200 is used to input the current multimodal data into a pre-built entity attribute relationship recognition model to identify entities, attributes, and relationships, and obtain the entity attribute relationship of the current multimodal data, wherein the pre-built entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data.
[0054] Optionally, in some embodiments, before inputting the current multimodal data into a pre-built entity attribute relationship recognition model for entity, attribute and relationship recognition, the recognition module 200 is further used to: obtain initial multimodal data, and pre-process the initial multimodal data to obtain pre-processed multimodal data; fuse the pre-processed multimodal data to obtain multimodal data to be trained; use the multimodal data to be trained to train an initial language model, and use a preset LoRA strategy to fine-tune the trained initial language model to obtain a pre-built entity attribute relationship recognition model.
[0055] Optionally, in some embodiments, when the initial language model is trained using the multimodal data to be trained, and the trained initial language model is fine-tuned using a preset LoRA strategy to obtain a pre-constructed entity attribute relationship recognition model, the recognition module 200 is also used to: divide the multimodal data to be trained into a training set, a test set, and a validation set according to a preset ratio; use the training set to train the initial language model, fine-tune the trained initial language model using a preset LoRA strategy, and use the validation set to verify the accuracy of entity extraction, attribute extraction, and relationship extraction of the fine-tuned initial language model, and after the accuracy is greater than the preset value, use the test set to test whether the fine-tuned initial language model meets the preset entity attribute relationship consistency condition; if the fine-tuned initial language model meets the preset entity attribute relationship consistency condition, the pre-constructed entity attribute relationship recognition model is obtained; otherwise, the pre-constructed entity attribute relationship recognition model is obtained after deleting the entities, attributes, or relationships that do not meet the preset entity attribute relationship consistency condition.
[0056] Optionally, in some embodiments, the recognition module 200 is further used to: encode the text data in the preprocessed multimodal data using a pretrained language model to generate a text embedding representation; extract image features in the preprocessed multimodal data using a pretrained image encoder to generate a video representation; extract audio features in the preprocessed multimodal data using an audio encoder to generate an audio representation; and concatenate the text embedding representation, video representation, and audio representation to obtain the multimodal data to be trained.
[0057] It should be noted that the aforementioned explanation of the embodiment of the method for identifying entity attribute relationships of multimodal data is also applicable to the apparatus for identifying entity attribute relationships of multimodal data of this embodiment, and will not be repeated here.
[0058] According to the entity attribute relationship recognition device for multimodal data proposed in the embodiment of the present application, current multimodal data is obtained, wherein the current multimodal data is multimodal data including text, images, and videos, and the current multimodal data is input into a pre-built entity attribute relationship recognition model to recognize entities, attributes, and relationships, thereby obtaining the entity attribute relationship of the current multimodal data. The pre-built entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data. Thus, the complexity and inefficiency of multimodal data in entity attribute relationship recognition are solved, and the recognition accuracy of entity attribute relationships in multilingual multimodal scientific and technological intelligence can be effectively improved, which has broad application prospects.
[0059] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0060] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .
[0061] When the processor 402 executes the program, the method for identifying entity attribute relationships of multimodal data provided in the above embodiment is implemented.
[0062] Furthermore, the electronic device further includes:
[0063] The communication interface 403 is used for communication between the memory 401 and the processor 402 .
[0064] The memory 401 is used to store computer programs that can be run on the processor 402 .
[0065] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0066] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.
[0067] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.
[0068] The processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0069] An embodiment of the present application also provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for identifying entity attribute relationships of multimodal data.
[0070] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.
[0071] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.
[0072] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.
[0073] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer program product for use with, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer program product" can be any device that can contain, store, communicate, propagate, or transmit a program for use with, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer program products include the following: an electrical connection having one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer program product may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or, if necessary, processing it in another suitable manner, and then storing it in a computer memory.
[0074] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0075] Those skilled in the art will understand that all or part of the steps in the method for implementing the above-mentioned embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer program product, which, when executed, includes one or a combination of the steps of the method embodiment.
[0076] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer program product.
[0077] The computer program product mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.
Claims
1. A method for identifying entity attribute relationships in multimodal data, characterized in that: The following steps are involved: Acquiring current multimodal data, wherein the current multimodal data is multimodal data including text, image, and video; The current multimodal data is input into a pre-built entity attribute relationship recognition model to identify entities, attributes and relationships, so as to obtain the entity attribute relationship of the current multimodal data, wherein the pre-built entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained, and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data.
2. The method according to claim 1, characterized in that Before inputting the current multimodal data into a pre-built entity attribute relationship recognition model for entity, attribute and relationship recognition, the method includes: Acquiring initial multimodal data, and preprocessing the initial multimodal data to obtain preprocessed multimodal data; Fusing the pre-processed multimodal data to obtain the multimodal data to be trained; The initial language model is trained using the multimodal data to be trained, and the trained initial language model is fine-tuned using a preset LoRA strategy to obtain the pre-built entity attribute relationship recognition model.
3. The method according to claim 2, characterized in that When the initial language model is trained using the multimodal data to be trained, and the trained initial language model is fine-tuned using the preset LoRA strategy to obtain the pre-built entity attribute relationship recognition model, the method further includes: Dividing the multimodal data to be trained into a training set, a test set, and a validation set according to a preset ratio; The initial language model is trained using the training set, and the preset LoRA strategy is used to fine-tune the trained initial language model. The accuracy of entity extraction, attribute extraction, and relationship extraction of the fine-tuned initial language model is verified using the validation set. After the accuracy is greater than a preset value, the test set is used to test whether the fine-tuned initial language model meets the preset entity attribute relationship consistency condition; If the fine-tuned initial language model meets the preset entity attribute relationship consistency condition, the pre-constructed entity attribute relationship recognition model is obtained; otherwise, the pre-constructed entity attribute relationship recognition model is obtained after deleting the entities, attributes or relationships that do not meet the preset entity attribute relationship consistency condition.
4. The method according to claim 2, characterized in that The method of fusing the pre-processed multimodal data to obtain the multimodal data to be trained includes: Encoding the text data in the preprocessed multimodal data using a pretrained language model to generate a text embedding representation; extracting image features from the preprocessed multimodal data using a pretrained image encoder to generate a video representation; extracting audio features from the preprocessed multimodal data using an audio encoder to generate an audio representation; The text embedding representation, the video representation, and the audio representation are concatenated to obtain the multimodal data to be trained.
5. A device for identifying entity attribute relationships of multimodal data, characterized in that: include: An acquisition module, configured to acquire current multimodal data, wherein the current multimodal data is multimodal data including text, image, and video; The recognition module is used to input the current multimodal data into a pre-built entity attribute relationship recognition model to identify entities, attributes and relationships, and obtain the entity attribute relationship of the current multimodal data, wherein the pre-built entity attribute relationship recognition model is obtained by training an initial language model using the multimodal data to be trained and fine-tuning the trained initial language model using a preset LoRA strategy, and the multimodal data to be trained is obtained by fusing the initial multimodal data.
6. The device according to claim 5, characterized in that Before inputting the current multimodal data into the pre-built entity attribute relationship recognition model for entity, attribute and relationship recognition, the recognition module is further configured to: Acquiring initial multimodal data, and preprocessing the initial multimodal data to obtain preprocessed multimodal data; Fusing the pre-processed multimodal data to obtain the multimodal data to be trained; The initial language model is trained using the multimodal data to be trained, and the trained initial language model is fine-tuned using a preset LoRA strategy to obtain the pre-built entity attribute relationship recognition model.
7. The device according to claim 6, characterized in that When the initial language model is trained using the multimodal data to be trained and the trained initial language model is fine-tuned using a preset LoRA strategy to obtain the pre-built entity attribute relationship recognition model, the recognition module is further used to: Dividing the multimodal data to be trained into a training set, a test set, and a validation set according to a preset ratio; The initial language model is trained using the training set, and the trained initial language model is fine-tuned using a preset LoRA strategy. The accuracy of entity extraction, attribute extraction, and relationship extraction of the fine-tuned initial language model is verified using the validation set. After the accuracy is greater than a preset value, the fine-tuned initial language model is tested using the test set to determine whether it meets a preset entity attribute relationship consistency condition. If the fine-tuned initial language model meets the preset entity attribute relationship consistency condition, the pre-constructed entity attribute relationship recognition model is obtained; otherwise, the pre-constructed entity attribute relationship recognition model is obtained after deleting the entities, attributes or relationships that do not meet the preset entity attribute relationship consistency condition.
8. The device according to claim 6, characterized in that The identification module is further used to: Encoding the text data in the preprocessed multimodal data using a pretrained language model to generate a text embedding representation; extracting image features from the preprocessed multimodal data using a pretrained image encoder to generate a video representation; extracting audio features from the preprocessed multimodal data using an audio encoder to generate an audio representation; The text embedding representation, the video representation, and the audio representation are concatenated to obtain the multimodal data to be trained.
9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the entity attribute relationship recognition method for multimodal data as described in any one of claims 1 to 4.
10. A computer program product having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the entity attribute relationship identification method of multimodal data as described in any one of claims 1 to 4.