Training method, device, electronic device and storage medium for multimodal coding model

Abstract

The present application provides a training method, device, electronic device and storage medium for a multimodal coding model; the method includes: calling the multimodal coding model to be trained, encoding each sample entity in the first and second sample knowledge graphs, and obtaining modal features of at least two modalities of each sample entity; performing the following processing for each sample entity: based on each modal feature of the sample entity, determining the inter-modal sub-contrast loss corresponding to the sample entity; performing the following processing for each aligned entity pair under each modality: based on the two modal features of the aligned entity pair, determining the intra-modal sub-contrast loss of the aligned entity pair; determining the inter-modal total contrast loss and the intra-modal total contrast loss of the multimodal coding model; based on the inter-modal total contrast loss and the intra-modal total contrast loss, updating the parameters of the multimodal coding model. Through the present application, the accuracy of the multimodal coding model for knowledge graph alignment processing can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a training method, device, electronic device and storage medium for a multimodal coding model. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0003] Multi-Modal Knowledge Graph contains data of multiple modalities. Entity alignment is to determine whether entities in different multi-modal knowledge graphs point to the same object. It is difficult to determine whether entities are aligned based on data of the same modality alone. Therefore, data of different modalities need to be encoded. There are differences between the features of different modalities, which in turn affects the accuracy of the aligned knowledge graph. The relevant technology currently has no encoding scheme with good alignment effect. Summary of the invention

[0004] The embodiments of the present application provide a training method, device, electronic device and storage medium, and computer program product for a multimodal coding model, which can improve the encoding accuracy of the multimodal coding model and improve the accuracy of the alignment results of the knowledge graph alignment processing.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for training a multimodal coding model, including:

[0007] Calling the multimodal encoding model to be trained, encoding each sample entity in the first sample knowledge graph and the second sample knowledge graph, and obtaining modal features of at least two modalities of each sample entity, wherein the first sample knowledge graph and the second sample knowledge graph are aligned with each other;

[0008] The following processing is performed for each of the sample entities: based on each of the modal features of the sample entity, determining the inter-modal sub-contrast loss corresponding to the sample entity;

[0009] Each of the sample entities in the first sample knowledge graph and the sample entities aligned with each sample entity in the second sample knowledge graph form an aligned entity pair;

[0010] The following processing is performed for each aligned entity pair in each modality: based on the two modal features of the aligned entity pair, the intra-modal sub-contrast loss of the aligned entity pair is determined;

[0011] Determining an inter-modality total contrastive loss of a multimodal coding model based on each inter-modality sub-contrastive loss, and determining an intra-modality total contrastive loss of the multimodal coding model based on each intra-modality sub-contrastive loss;

[0012] Based on the inter-modal total contrast loss and the intra-modal total contrast loss, the parameters of the multimodal encoding model are updated.

[0013] An embodiment of the present application provides a training device for a multimodal coding model, including:

[0014] an encoding module configured to call the multimodal encoding model to be trained, perform encoding processing on each sample entity in the first sample knowledge graph and the second sample knowledge graph, and obtain modal features of at least two modalities of each sample entity, wherein the first sample knowledge graph and the second sample knowledge graph are aligned with each other;

[0015] The loss acquisition module is configured to perform the following processing for each sample entity: based on each modal feature of the sample entity, determine the inter-modal sub-contrast loss corresponding to the sample entity;

[0016] The loss acquisition module is further configured to form an aligned entity pair with each sample entity in the first sample knowledge graph and the sample entity aligned with each sample entity in the second sample knowledge graph;

[0017] The loss acquisition module is further configured to perform the following processing for each aligned entity pair in each modality: determining the intra-modal sub-contrast loss of the aligned entity pair based on two modal features of the aligned entity pair;

[0018] A loss acquisition module configured to determine an inter-modality total contrast loss of the multimodal coding model based on each inter-modality sub-contrast loss, and to determine an intra-modality total contrast loss of the multimodal coding model based on each intra-modality sub-contrast loss;

[0019] A training module is configured to update parameters of a multimodal encoding model based on an inter-modality total contrast loss and an intra-modality total contrast loss.

[0020] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0021] A memory for storing executable instructions;

[0022] The processor is used to implement the training method of the multimodal coding model of the embodiment of the present application when executing the executable instructions stored in the memory.

[0023] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implements the training method of the multimodal coding model of the embodiment of the present application.

[0024] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the training method of the multimodal coding model of the embodiment of the present application.

[0025] The embodiments of the present application have the following beneficial effects:

[0026] By obtaining the intra-modal contrast loss and the inter-modal contrast loss, the multimodal knowledge graph encoding model can reduce the encoding loss from both the inter-modal and intra-modal aspects, making the encoding results more accurate and improving the accuracy of the multimodal knowledge graph encoding. The multimodal encoding model can then perform more accurate alignment processing on the knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of an application scenario of the training method of the multimodal coding model provided in an embodiment of the present application;

[0028] Figure 2 It is a structural schematic diagram of an electronic device for training a multimodal coding model provided in an embodiment of the present application;

[0029] Figure 3A It is a flowchart of a training method for a multimodal coding model provided in an embodiment of the present application;

[0030] Figure 3B It is a flowchart of a training method for a multimodal coding model provided in an embodiment of the present application;

[0031] Figure 3C It is a flowchart of a training method for a multimodal coding model provided in an embodiment of the present application;

[0032] Figure 4 It is a flowchart of a training method for a multimodal coding model provided in an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of the knowledge graph provided in the embodiment of the present application;

[0034] Figure 6 is a schematic diagram of the encoding process of the multimodal encoding model provided in an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of the structure of a multimodal coding model provided in an embodiment of the present application;

[0036] Figure 8It is an optional flowchart of the training method of the multimodal coding model provided in the embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] It should be pointed out that in the embodiments of the present application, when user information, user feedback data and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] 1) Modality: Each source or form of information can be called a modality. For example, the media of information include voice, image, text, etc.; the sources of information include various sensors, such as radar, infrared, accelerometer, etc. Each of the above can be called a modality.

[0044] 2) Knowledge Graph (KG). Knowledge Graph is the most effective way to represent relationships. It is a relational network that connects all different types of information. Knowledge Graph is a semantic network that reveals the relationships between entities. It can formally describe things in the real world and their relationships. "Entity" is used to represent nodes in the graph, and "Relation" is used to represent "edges" in the graph. In addition to text data and structured data, Multi-Modal Knowledge Graph also contains visual data, such as assigning one or more images to each entity to provide sufficient visual information for the entities in the knowledge graph.

[0045] 3) Alignment refers to determining whether at least two entities point to the same object. For example, if the entity "Liu" in Knowledge Graph 1 and the entity "Lao Liu" in Knowledge Graph 2 point to the same person in the real world, the two entities are aligned. Multimodal Knowledge Graph Alignment refers to determining whether entities in two or more different knowledge graphs point to the same object in the real world. The characteristic of multimodal entity alignment is that in the process of alignment, knowledge of different modalities can be fully integrated for alignment, including text, image and other information.

[0046] 4) Contrastive Learning: Contrastive learning is a task method in machine learning to describe similar and different objects. It learns the feature representation of samples by comparing data with positive and negative samples in feature space. Through contrastive learning, machine learning models can be trained to distinguish between similar but different objects.

[0047] 5) Convolutional Neural Networks (CNN): It is a type of feed forward neural network (FNN) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input images according to their hierarchical structure.

[0048] 6) Graph Neural Network (GNN): Traditional neural networks are more suitable for Euclidean space data. Graph Neural Network is a network model that uses neural networks in graph structures. There are many types of graph neural networks, including graph convolutional networks (GCN), graph attention networks (GAT), and graph auto-encoders (GAE).

[0049] The embodiments of the present application provide a training method for a multimodal coding model, a training device for a multimodal coding model, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the encoding accuracy of the multimodal coding model and improve the accuracy of the alignment results of the knowledge graph alignment processing.

[0050] The following describes an exemplary application of an electronic device provided by an embodiment of the present application. The electronic device provided by an embodiment of the present application can be implemented as a laptop, a tablet computer, a desktop computer, a set-top box, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), a vehicle-mounted terminal, and other types of user terminals, and can also be implemented as a server. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. The embodiments of the present application can be implemented by a server, or by a terminal device and a server in collaboration. Below, an exemplary application when the electronic device is implemented as a server will be described.

[0051] refer to Figure 1 , Figure 1 It is a schematic diagram of the application mode of the training method of the multimodal coding model provided in the embodiment of the present application; for example, the servers involved include: a training server 201, an alignment server 202 and a recommendation server 203 (belonging to a recommendation system, such as a medical system), a network 300 and a first terminal device 401. The training server 201, the alignment server 202 and the recommendation server 203 communicate through the network 300, or communicate through other means, and the first terminal device 401 is connected to the recommendation server 202 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0052] The knowledge graph to be aligned can be a medical-related knowledge graph, which includes data on image modalities (for example, drug display images, packaging display images, etc.), entity attribute modalities (attribute parameters of entities, such as the ingredients of drugs, the dosage each time, etc.), and entity structure modalities (relationships between entities, such as the relationship between drugs and other drugs).

[0053] The training server 201 obtains the aligned sample knowledge graph from the network 300, trains the multimodal encoding model based on the sample knowledge graph, and synchronizes the trained multimodal encoding model to the alignment server 202. The alignment server 202 obtains the knowledge graph to be aligned from the network 300 or the database, calls the multimodal encoding model to align the knowledge graph, and sends the alignment result to the recommendation server 203. The recommendation server 203 sends the recommendation result to the user's terminal device 401 based on the alignment result of the knowledge graph.

[0054] The embodiment of the present application can be applied in the medical field to align knowledge graphs of different granularities. For example, the knowledge graphs to be aligned are the general knowledge graph in the medical field and the knowledge graph in the vertical field of medication. The alignment server 202 calls the trained multimodal encoding model to align the two, obtains the alignment result, and sends the alignment result to the recommendation server 203. The user is a doctor. When the doctor prescribes a medical treatment order for the patient, the doctor inputs the symptom information of the patient through the human-computer interaction interface of the terminal device 401. The recommendation server 203 can find the corresponding disease in the general knowledge graph in the medical field based on the symptom information, obtain the recommended medication for the disease in the knowledge graph in the vertical field of medication based on the alignment result and the disease, and send the recommended medication and the recommended dosage of the medication to the terminal device 401, so that the doctor can prescribe a standardized medical treatment order for the patient according to the recommended medication and dosage, thereby improving the processing efficiency of the automated medical process.

[0055] For example, continue to refer to Figure 1 , Assume that: the third knowledge graph is a general knowledge graph in the medical field, the fourth knowledge graph is a knowledge graph in the vertical field of medication, and the recommendation server 203 is a server of the medical consultation platform. Ordinary users who lack medical knowledge input symptom information through the human-computer interaction interface of the terminal device 401, and the terminal device 401 sends a query request to the recommendation server 203. The recommendation server 203 queries the target disease in the general knowledge graph in the medical field based on the symptom information, queries the entity (symptomatic drug) corresponding to the disease in the knowledge graph in the vertical field of medication based on the target disease and the alignment result, and recommends the target disease, symptomatic drug and the information associated with the symptomatic drug in the knowledge graph in the vertical field of medication (for example: attributes, dosage, etc.; structure, relationship with other drugs; image, drug packaging picture, drug physical picture) to the user's terminal device 401. Users can purchase drugs based on the recommended information or determine to see a doctor based on the recommended information, which improves the efficiency of users in obtaining medical consultation information.

[0056] In some embodiments, it is assumed that: the recommendation server 203 is a server of the teaching platform, the user is a student, the third knowledge graph is a general knowledge graph in the field of physics, the fourth knowledge graph is a knowledge graph in the field of mathematics, the trained multimodal coding model aligns the entities in the third knowledge graph and the fourth knowledge graph, the user inputs the physics problem to be solved through the human-computer interaction interface of the terminal device 401, the terminal device 401 sends the problem to the recommendation server 203, the recommendation server obtains the corresponding problem-solving method in the general knowledge graph in the field of physics and the corresponding problem-solving formula in the schematic diagram of physical principles and the general knowledge graph in the field of mathematics based on the physics problem, the alignment result and the attribute domain knowledge graph, and sends the problem-solving method, the problem-solving formula and the physics schematic diagram to the user's terminal device 401, so that the user obtains the answer to the physics problem. In the embodiment of the present application, the coding accuracy of the multimodal coding model is improved, and the knowledge graphs in different fields are aligned using the multimodal coding model, and the knowledge data in different fields are integrated, so as to facilitate the recommendation of knowledge in different fields to the user, and improve the accuracy of the recommendation.

[0057] In some embodiments, the trained multimodal encoding model provided by the present application can also be used to merge redundant data. For example, the third knowledge graph and the fourth knowledge graph are feature information knowledge graphs with different information sources in the database (for example, feature information obtained by users actively filling in information, or feature information obtained by users selecting preset options). The two knowledge graphs can be aligned through the multimodal encoding model to obtain an alignment result. Based on the alignment result, the information pointing to the same entity (such as users or organizations) in the two knowledge graphs is reintegrated into the same new knowledge graph to eliminate redundant data, thereby saving storage space in the database and saving computing resources consumed when retrieving data based on the database.

[0058] In some embodiments, some or all of the training server 201 , the alignment server 202 , and the recommendation server 203 may also be implemented as a unified server.

[0059] The embodiment of the present application can be implemented through blockchain technology, and the knowledge graph processed by the multimodal coding model alignment of the embodiment of the present application can be stored in the blockchain to enhance the reliability of the alignment results between the knowledge graphs. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain can include the blockchain underlying platform, the platform product service layer, and the application service layer.

[0060] The embodiments of the present application can be implemented through database technology. In short, a database can be regarded as an electronic file cabinet where electronic files are stored. Users can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.

[0061] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0062] In some embodiments, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0063] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. With the high development and application of the Internet industry, as well as the promotion of search services, social networks, mobile commerce and open collaboration, each item may have its own hash code identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0064] Medical cloud refers to the use of cloud computing to create a cloud platform for medical and health services based on new technologies such as cloud computing, mobile technology, multimedia, 4G communications, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the application of cloud computing technology, medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. For example, hospital appointment registration, medical insurance, etc. are all products of the combination of cloud computing and the medical field. Medical cloud also has the advantages of data security, information sharing, dynamic expansion, and overall layout.

[0065] See Figure 2 , Figure 2 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, including: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 440 .

[0066] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0068] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0071] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0072] In some embodiments, the training device for the multimodal coding model provided in the embodiments of the present application can be implemented in software. Figure 2 A training device 455 of a multimodal encoding model stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: an encoding module 4551, a loss acquisition model 4552, and a training module 4553. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0073] The training method of the multimodal coding model provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0074] See also Figure 3A , Figure 3A is a flowchart of a training method for a multimodal coding model provided in an embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0075] In step 301, the multimodal encoding model to be trained is called to encode each sample entity in the first sample knowledge graph and the second sample knowledge graph to obtain modal features of at least two modalities for each sample entity.

[0076] Here, the first sample knowledge graph and the second sample knowledge graph are aligned with each other.

[0077] For example, the types of modalities in the embodiments of the present application include: structural modality, attribute modality, and image modality. In some embodiments, the types of modalities may also include: voice modality, electronic signal modality, etc. Figure 3B , Figure 3B It is a flow chart of the training method of the multimodal coding model provided in an embodiment of the present application; step 301 can be implemented by calling the multimodal coding model and performing steps 3011 to 3014 for each sample entity in the first sample knowledge graph and the second sample knowledge graph, as described in detail below.

[0078] In step 3011, based on the sample knowledge graph corresponding to the sample entity, the structural modal data, attribute modal data and image modal data corresponding to the sample entity are determined.

[0079] For example, the structural modal data of an entity is the relationship data between an entity and other entities, and the knowledge graph is the most effective way to represent relationships. The knowledge graph is a relationship network obtained by connecting different types of information together. The structural modal data of an entity is the relationship data between entities in a knowledge graph, and the structural modal data of an entity can represent the entity structure information in a knowledge graph. The entities in an aligned knowledge graph are aligned with each other, and thus, the aligned knowledge graph has aligned entity structure information. Attribute modal data is the attribute parameters corresponding to an entity (for example: if an entity is a certain type of food, then the attribute modal data of the entity is: the shelf life of the food, the ingredients of the food, etc.), and image modal data is an image related to an entity (for example: if an entity is a home appliance, the image modal data of the entity is the product display picture of the entity).

[0080] In an embodiment of the present application, by improving the encoding accuracy of different types of modal data of the multimodal encoding model, the alignment effect on the knowledge graph can be improved, making the alignment result more accurate.

[0081] Example, reference Figure 7 , Figure 77 is a schematic diagram of the structure of the multimodal coding model provided in the embodiment of the present application, and the multimodal coding model 700 includes a structure encoder 601, an attribute encoder 602, and an image encoder 603. Each encoder is independent of each other, and steps 3012 to 3014 can be performed concurrently to improve the coding efficiency, thereby improving the training speed of the multimodal coding model.

[0082] In step 3012, mapping processing is performed on the structural modal data, and convolution processing is performed on the entity relationship graph obtained based on the mapping processing to obtain the structural modal features of the sample entity.

[0083] Here, the structural modal data characterizes the relationship between the sample entity and other sample entities.

[0084] For example, step 3012 can be implemented in the following way: mapping the structural modal data to obtain an entity relationship diagram of the entity, wherein the entity relationship diagram includes: an entity node and multiple neighbor nodes corresponding to the entity node; performing multiple feature aggregation processes based on the entity node and each neighbor node in the entity relationship diagram to obtain the structural modal features of the sample entity.

[0085] For example, each node in the entity relationship diagram corresponds to an entity, and the lines between entities represent the association between the entities. For example, the entity is stationery A. In the entity relationship diagram, stationery A has an association relationship with stationery B, stationery C, and stationery D respectively, and the nodes corresponding to stationery B, stationery C, and stationery D are neighbor nodes of the node of stationery A.

[0086] Continue to refer Figure 7 , the structural encoder 601 can be a multi-level graph convolutional neural network model, including multiple graph convolutional layers 701A. The graph convolutional neural network model maps the structural modal data in each knowledge graph to the same feature space, converts it into an entity relationship graph corresponding to each entity, and encodes the entity relationship graph of each entity. Except for the first level of graph convolutional layer 701A, each level of graph convolutional layer 701A takes the output result of the previous level as input, and each level of graph convolutional layer 701A performs weighted calculation processing on the features of each neighbor node of the entity node. The output of the graph convolutional layer 701A of the last level is used as the structural modal feature of the entity.

[0087] In step 3013, the attribute modal data are combined and processed, and a dimensionality reduction process is performed based on the combined feature vector obtained by the combined processing to obtain the attribute modal features of the sample entity.

[0088] Here, the dimension of the combined feature vector is higher than that of the attribute modality features.

[0089] In some embodiments, attribute modal data refers to various parameters of an entity. For example, if the entity is a user, the attributes corresponding to the user include age, gender, job, place of residence, place of birth, etc. Figure 5 , Figure 5 It is a knowledge graph for an entity provided by the embodiment of the present application. The attributes of the user "Lao Liu" include: job (singer), birthplace (such and such city), gender (male). Image 501 is a photo of "Lao Liu", which is the image modality data of the entity.

[0090] In some embodiments, step 3013 can be implemented by: combining each attribute corresponding to the attribute modal data to obtain a combined feature vector of the entity; performing dimensionality reduction mapping on the combined feature vector to obtain the attribute modal characteristics of the sample entity.

[0091] For example, continue to refer to Figure 7 , the attribute encoder 602 includes a data conversion unit 701B and a feedforward neural network 702B. The data conversion unit 701B constructs a one-hot vector (i.e., a combined feature vector) according to the attribute data corresponding to each entity in the knowledge graph. Each dimension of the one-hot vector corresponds to an attribute, and the component value corresponding to the attribute can be represented by 0 or 1. 0 indicates that the entity contains this attribute, and 1 indicates that the entity does not have this attribute. Due to the complexity of the types of entities, the types of attributes corresponding to different entities are not the same. For example, if the entity is a user, the attribute modal data of the user includes: gender, age, etc.; if the entity is a drug, the attribute modal data of the drug includes: dosage, effect, etc. The following example illustrates the combined feature vector. For example, the attributes corresponding to each dimension of the combined feature vector are: age, height, gender, person name, drug name, drug dosage, drug shelf life, food name, food shelf life, etc. For user A, the combined feature vector can be [1.1.1.1.0.0.0.0.0], for drug B, the combined feature vector can be [0.0.0.0.1.1.1.0.0], and for food C, the combined feature vector can be [0.0.0.0.0.0.0.1.1].

[0092] In some embodiments, the component value of each dimension in the combined feature vector may also be represented by a numerical value from 0 to 1. When the entity does not contain the attribute corresponding to the dimension, the component value is 0, and when the entity contains the attribute corresponding to the dimension, the component value is determined according to the parameter value corresponding to the attribute.

[0093] For example, to facilitate statistics, only attribute types with higher occurrence frequencies may be considered, and the one-hot vectors may be subjected to dimensionality reduction mapping processing through the feedforward neural network 702B to obtain attribute modal features of the sample entity.

[0094] In some embodiments, in order to improve the computational efficiency, the unique hot vector corresponding to each entity can be composed of the unique hot vector feature matrix of the entire knowledge graph. Then, the attribute modal features of each entity in the knowledge graph are composed of the attribute feature matrix F A , which can be expressed as the following formula (2):

[0095] F A =W A ·A+b A (2)

[0096] Among them, A represents the feature matrix composed of the attribute unique hot vectors of each entity in the knowledge graph; W A , b A Represents trainable parameters.

[0097] In some embodiments, the types of attribute modal data include: relationship attribute data, parameter attribute data; step 3013 can be implemented in the following ways: combine each attribute corresponding to the parameter attribute data to obtain the parameter combination feature vector of the entity; determine the relationship feature vector of the entity based on the relationship attribute data, wherein the component value corresponding to each dimension in the relationship feature vector represents: the relationship between the sample entity and the entity corresponding to each dimension; perform dimensionality reduction mapping processing on the parameter high-dimensional feature vector and the relationship feature vector respectively, and fuse the mapped parameter low-dimensional feature vector and the relationship low-dimensional feature vector to obtain the attribute modal characteristics of the sample entity.

[0098] For example, the relationship between entities can also be used as an attribute of the entity. The relationship corresponding to the entity can be used as an attribute to construct an attribute unique hot vector (relation feature vector) representing the relationship between entities. Each dimension of the unique hot vector corresponds to a relationship. The parameter corresponding to the relationship can be represented by 0 or 1. 0 indicates that the entity contains this relationship, and 1 indicates that the entity does not have this relationship. Call the feedforward neural network 702B to perform dimensionality reduction processing on the relationship feature vector to obtain a relationship low-dimensional feature vector. The relationship low-dimensional feature vector can also be composed into a feature matrix. The attribute feature matrix F composed of the structural attribute modal features of each entity in the knowledge graph R , which can be expressed as the following formula (3):

[0099] F R =W R ·R+b R (3)

[0100] Where R represents the feature matrix composed of the low-dimensional feature vectors of the relationship between each entity in the knowledge graph; W R , b R Represents trainable parameters.

[0101] For example, the attribute modal features of the sample entity can be obtained by obtaining the average value of the parameter low-dimensional feature vector and the relationship low-dimensional feature vector. Or, by using formula (4), F A and F R The feature matrix F of the attribute modality of each entity in the knowledge graph is obtained by fusion T , formula (4) is as follows:

[0102] F T =σ(W T ·[F A ; F R ]+b T ) (4)

[0103] Where [;] represents the concatenation operation between matrices; W T , b T represents a trainable parameter; σ represents the ReLU activation function. T Represents the attribute features corresponding to each entity in the attribute mode.

[0104] In an embodiment of the present application, by treating the relationship between entities as an attribute of the entity, the entity relationship attribute is merged with the attributes contained in the entity, so that the modal features of the attribute modality can more accurately characterize the features corresponding to the entity, enhance the granularity of the modal features, and thereby improve the accuracy of the encoding.

[0105] In step 3014, image feature extraction processing is performed on the image modality data to obtain image modality features of the sample entity.

[0106] In some embodiments, step 3014 is implemented by: performing multi-level convolution operations and pooling operations on the image modality data to obtain an image feature vector of the entity; performing dimensionality reduction mapping processing on the image feature vector to obtain image modality features of the sample entity.

[0107] Continue to refer Figure 7 The image encoder 603 includes a convolutional neural network 701C and a feedforward neural network 702C. In the embodiment of the present application, the convolutional neural network 701C is a 152-layer ResNet residual network model. The convolutional neural network 701C can perform multiple levels of convolution operations and pooling operations on the image of the entity, and the pooling result of the previous pooling layer of the last fully connected layer in the convolutional neural network 701C is used as the image feature vector. The feedforward neural network 702C performs dimensionality reduction processing on the image feature vector to obtain the image modality feature. The matrix F composed of the image modality features of each entity in the knowledge graph I It can be expressed as the following formula (5):

[0108] F I =WI RESNET(L)+b I (5)

[0109] Among them, L represents the feature matrix composed of the image feature vectors of each entity in the knowledge graph; W I , b I Represents trainable parameters.

[0110] In step 302, the following processing is performed for each sample entity: based on each modal feature of the sample entity, the inter-modal sub-contrast loss corresponding to the sample entity is determined.

[0111] In some embodiments, step 302 is implemented by: determining a feature distance between any two modal features of the sample entity; and taking the sum of each feature distance as the inter-modal sub-contrast loss corresponding to the sample entity.

[0112] For example, in the knowledge graph, the structural features, attribute features, and image features of any entity are b H , b A , b I , inter-modality contrast loss L inter It is expressed as the following formula (11):

[0113] L inter =||b H -b A ||+||b H -b I ||+||b A -b I || (11)

[0114] Inter-modality contrast loss L inter The smaller it is, the closer the feature distance between the modal features of different modes of the same entity is, so that the modal features of different modes remain consistent.

[0115] In the embodiment of the present application, the inter-modal contrast loss is obtained, and then the parameters of the multi-modal coding model are updated based on the inter-modal contrast loss, thereby improving the accuracy of the coding processing of the multi-modal coding model. The modal features of different modes output by the multi-modal coding model are reused, the amount of calculation for calculating the contrast loss is reduced, and computing resources are saved.

[0116] In step 303, each sample entity in the first sample knowledge graph and the sample entities aligned with each sample entity in the second sample knowledge graph are combined into an aligned entity pair.

[0117] For example, the first sample knowledge graph and the second sample knowledge graph as training samples are pre-aligned, and step 303 can be implemented in the following way: based on the alignment relationship between the first sample knowledge graph and the second sample knowledge graph, determine every two sample entities in the first sample knowledge graph and the second sample knowledge graph that have an alignment relationship; every two sample entities that have an alignment relationship are formed into an aligned entity pair.

[0118] In some embodiments, step 303 may also be performed before step 301 to reduce the amount of calculation during the training process.

[0119] In step 304 , the following processing is performed for each aligned entity pair in each modality: based on the two modal features of the aligned entity pair, the intra-modal sub-contrast loss of the aligned entity pair is determined.

[0120] Here, the aligned entity pair includes a first sample entity and a second sample entity, the first sample entity belongs to the first sample knowledge graph, and the second sample entity belongs to the second sample knowledge graph.

[0121] For example, the intra-modal contrast loss can be characterized as the ratio of the similarity between an entity and its positive sample to the sum of the similarity between the entity and its negative sample and the similarity between the entity and its positive sample. In the embodiment of the present application, the positive sample is an entity aligned with the first sample entity, and the negative sample is an entity that cannot be aligned with the first sample entity.

[0122] In some embodiments, reference Figure 3C , Figure 3C It is a flowchart of the training method of the multimodal coding model provided in an embodiment of the present application; step 304 can be implemented through steps 3041 to 3043, which are described in detail below.

[0123] In step 3041, a first modal intra-sub-loss of the sample entity pair is determined based on a first feature similarity between a modal feature of the first sample entity and a modal feature of a second sample entity aligned with the first sample entity, and a second feature similarity between the modal feature of the first sample entity and the modal feature of each negative sample of the first sample entity.

[0124] Here, the negative sample of the first sample entity is the sample entity other than the second sample entity in the second sample knowledge graph.

[0125] For example, a negative sample is an entity that is not aligned with the first sample entity, that is, compared with the positive sample, the similarity between the negative sample and the first sample entity is lower. Assuming that each entity in the knowledge graph is different, then other sample entities in the second sample knowledge graph except the second sample entity aligned with the first sample entity can all be used as negative samples of the first sample entity.

[0126] For example, step 3041 can be implemented in the following manner: obtain the sum of each second feature similarity corresponding to the first sample entity, and use the sum of each second feature similarity and the sum of the first feature similarity as the first intermediate parameter; use the ratio of the first feature similarity to the first intermediate parameter as the first modal intra-sub-loss of the sample entity pair.

[0127] In step 3042, a second modal intra-sub-loss of the sample entity pair is determined based on a first feature similarity between the modal features of the first sample entity and the modal features of the second sample entity aligned with the first sample entity, and a third feature similarity between the modal features of the second sample entity and the modal features of each negative sample of the second sample entity.

[0128] Here, the negative sample of the second sample entity is the sample entity other than the first sample entity in the first sample knowledge graph.

[0129] For example, the principle of the negative sample corresponding to the second sample entity is the same as that of the first sample entity, which can be referred to above and will not be repeated here.

[0130] For example, step 3042 can be implemented in the following manner: obtain the sum of each third feature similarity corresponding to the second sample entity, and use the sum of each third feature similarity and the sum of the first feature similarity as the second intermediate parameter; use the ratio of the first feature similarity to the second intermediate parameter as the second modal intra-sub-loss of the sample entity pair.

[0131] In some embodiments, the negative samples in steps 3041 and 3042 can be expressed as the following formula (6.1) and formula (6.2), respectively:

[0132]

[0133]

[0134] Among them, Pair neg (KG1→KG2) means that the second sample knowledge graph KG2 is replaced by the second sample entity e 2 Other entities are taken as the first sample entity e 1 negative samples, forming a negative sample pair, n is a positive integer, and n is the number of entities in the knowledge graph. neg (KG2→KG1) means that the first sample knowledge graph KG1 is replaced by the first sample entity e 1 Other entities are used as the second sample entity e 2 negative samples to form negative sample pairs.

[0135] The first feature similarity between the first sample entity and the second sample entity is expressed as f(e 1 , e 2 ), the first sample entity e 1 With any negative sample The second feature similarity between The second sample entity e 2 With any negative sample The third feature similarity between

[0136] In step 3043, the average of the first modal intra-sub-loss and the second modal intra-sub-loss is used as the modal intra-sub-contrast loss of the aligned entity pair.

[0137] For example, the intra-modal contrast loss can be expressed as the following formula (7.1) and formula (7.2), formula (7.3):

[0138]

[0139]

[0140]

[0141] in, Represents entity e 1 The corresponding intra-modality contrastive loss, The smaller the entity e 1 With entity e 2 The closer the distance of the modal features, the 1 Same as KG2 except e 2 The farther the distance between the modal features of the entities outside. Represents entity e 2 The corresponding intra-modality contrastive loss, The smaller the entity e 2 With entity e 1 The closer the distance of the modal features, the 2 Same as KG1 except entity e 1 The farther the distance between the modal features of the entities outside. In each mode, the average of the intra-modal contrast losses corresponding to the two entities is taken as the intra-modal contrast loss L in each mode. intra .

[0142] In some embodiments, the scope of negative samples can be further expanded. Other sample entities in the first knowledge graph other than the first sample entity can also be used as negative samples of the first sample entity, that is, other entities in the same knowledge graph can be used as negative samples. Other sample entities in the second knowledge graph other than the second sample entity can also be used as negative samples of the second sample entity, that is, other entities in the same knowledge graph can be used as negative samples.

[0143] For example, the negative samples of the first sample entity are sample entities other than the first sample entity in the first sample knowledge graph, and sample entities other than the second sample entity in the second sample knowledge graph; the negative samples of the second sample entity are sample entities other than the second sample entity in the second sample knowledge graph, and sample entities other than the first sample entity in the first sample knowledge graph.

[0144] Then, the negative sample pair corresponding to the first sample entity and the negative sample pair corresponding to the second sample entity can be expressed as the following formula (8.1) and formula (8.2):

[0145]

[0146]

[0147] Where Pair neg (KG1) represents the first sample entity e in the first sample knowledge graph KG1 1 The corresponding negative sample pair; accordingly, Pair neg (KG2) represents the second sample entity e in the second sample knowledge graph KG2 2 Corresponding negative sample pairs. Based on the above negative sample pairs, the intra-modal contrast loss can be expressed as the following formula (7.1), formula (9.1), and formula (9.2):

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] Among them, the feature distance can be represented by similarity. For the sample entity pair (e 1 , e 2 ), The smaller it is, the smaller the first sample entity e 1The farther the feature distance of the entity that is not aligned with the second sample knowledge graph KG2 is, the farther the feature distance of other entities other than itself in the first sample knowledge graph KG1 is. Similarly, The smaller it is, the smaller the second sample entity e 2 The farther the feature distance is from the unaligned entity in the first sample knowledge graph KG1, the farther the feature distance is from other entities other than itself in the second sample knowledge graph KG2.

[0154] In an embodiment of the present application, by taking sample entities of the same knowledge graph and different knowledge graphs as negative samples, on the one hand, the range of negative samples is expanded, and on the other hand, the intra-modal contrast loss obtained based on the parameters related to the negative samples can make the encoding processing of the multimodal coding model more accurate, that is, the similarity between negative samples is made higher and the similarity between positive samples is made smaller, thereby improving the accuracy of the multimodal coding model in aligning different knowledge graphs (avoiding the alignment result of entities pointing to different objects being aligned).

[0155] In step 305, the inter-modality total contrastive loss of the multimodal coding model is determined based on each inter-modality sub-contrastive loss, and the intra-modality total contrastive loss of the multimodal coding model is determined based on each intra-modality sub-contrastive loss.

[0156] In some embodiments, step 305 can be implemented in the following manner: adding each inter-modal sub-contrast loss to obtain the total inter-modal contrast loss of the multimodal coding model; adding the intra-modal sub-contrast loss of each modality corresponding to each entity to obtain the total intra-modal contrast loss of the multimodal coding model.

[0157] For example, in the embodiments of the present application, three types of modalities (attribute, structure, and image) are used as examples to illustrate that the intra-modal contrast loss of the three modalities of the same entity is added together to obtain the intra-modal contrast loss corresponding to an entity, and the intra-modal contrast loss of each entity in the same knowledge graph is added together to obtain the total intra-modal contrast loss of the multimodal encoding model. Similarly, the inter-modal sub-contrast loss of each entity in the same knowledge graph is added together to obtain the total intra-modal contrast loss of the multimodal encoding model.

[0158] In step 306, the parameters of the multimodal coding model are updated based on the inter-modality total contrast loss and the intra-modality total contrast loss.

[0159] For example, the multimodal coding model can be back-propagated based on the inter-modal total contrast loss to obtain a trained multimodal coding model, and then the trained multimodal coding model can be back-propagated based on the intra-modal total contrast loss to obtain a trained multimodal coding model.

[0160] In some embodiments, step 306 can be implemented in the following manner: based on the total contrast loss between modalities and the total contrast loss within modality, back-propagation processing is performed on the multimodal coding model to obtain updated parameters of the multimodal coding model; based on the updated parameters, the corresponding original parameters in the multimodal coding model are replaced.

[0161] For example, the back propagation process is implemented in the following way: the modal features of different modes output by the multimodal coding model are used as the output results, the gradient corresponding to the original parameters in the multimodal coding model is obtained by derivation based on the contrast loss and the output results, and the partial derivative of the original parameters in the multimodal coding model is obtained by using the chain rule, the gradient is substituted into the partial derivative result, and the update formula of the corresponding original parameters in the multimodal coding model is obtained, and the update parameters are determined based on the preset learning rate, contrast loss and update formula. The original parameters of the multimodal coding model are replaced with the update parameters.

[0162] In some embodiments, after step 306, the trained multimodal coding model can also be used. Figure 4 , Figure 4 It is a flowchart of the training method of the multimodal encoding model provided in an embodiment of the present application; the knowledge graph is aligned through steps 401 to 404.

[0163] In step 401, the updated multimodal encoding model is called to encode the third knowledge graph and the fourth knowledge graph to be aligned, and the modal features of each modality of each entity in the third knowledge graph and the fourth knowledge graph are obtained.

[0164] For example, the encoding process can refer to step 301, which will not be described in detail here.

[0165] In step 402, the following processing is performed for each entity: feature splicing processing is performed based on each modal feature of the entity to obtain the splicing feature of the entity.

[0166] For example, the splicing process can be implemented by weighted average calculation, where each modal feature is multiplied by the corresponding weight and added, and the sum is divided by the number of modalities to obtain the splicing feature of the entity.

[0167] In step 403, the following processing is performed for each entity in the third knowledge graph: the similarity between the concatenated features of the entity and the concatenated features of each entity in the fourth knowledge graph is determined; and at least one entity in the fourth knowledge graph at the head of the entity in descending order of similarity is used as the alignment result of the entity.

[0168] For example, continue to refer to Figure 7, the multimodal encoding model 700 also includes a feature fusion module 703 and a feature alignment module 704. Each encoder encodes the modal data corresponding to the entity in the knowledge graph to obtain the modal features of each modality of the entity. The feature fusion module 703 concatenates each modal feature of the entity to obtain a fused feature. The feature alignment module 704 determines the similarity between the fused feature of the entity and the modal features of each entity in other knowledge graphs, and takes at least one entity at the head in the descending order of similarity as the alignment result of the entity.

[0169] For example, you can also set an interval for the similarity, and use the entities in the fourth knowledge graph corresponding to the similarity within the similarity interval as the alignment entities of each entity in the third knowledge graph. For example: the similarity interval is greater than or equal to 0.95 and less than or equal to 1. Assuming that the similarity corresponding to entity F in the third knowledge graph is sorted in descending order as (0.98 / 0.95 / 0.95 / 0.90...), then the entities in the fourth knowledge graph corresponding to the similarity of 0.99 / 0.95 / 0.95 are selected as the alignment entities of entity F, forming three alignment entity pairs, and the three alignment entity pairs are used as the alignment results of entity F. For another example: the similarity interval is greater than or equal to 0.99 and less than or equal to 1, and the similarity corresponding to entity F in the third knowledge graph is sorted in descending order as (0.98 / 0.95 / 0.95 / 0.90...), and any similarity in the descending similarity sorting is not in the similarity interval, then there is no entity in the fourth knowledge graph that is aligned with entity F, and the alignment result of entity F is that there is no aligned entity.

[0170] In step 404, the alignment results of each entity in the third knowledge graph are combined to obtain the alignment results between the third knowledge graph and the fourth knowledge graph.

[0171] For example, the alignment results between the knowledge graphs that have been aligned correspond to each other. By counting the alignment results of each entity in the third knowledge graph, or by counting the alignment results of each entity in the fourth knowledge graph, the alignment results between the third knowledge graph and the fourth knowledge graph can be obtained.

[0172] In an embodiment of the present application, the multimodal coding model is trained by obtaining the intra-modal contrast loss and the inter-modal contrast loss corresponding to each modality, thereby optimizing the encoding accuracy of the multimodal coding model for data of different modalities, so that the feature distance between the modal features of the positive samples under each modality obtained by encoding is closer, and the feature distance between the modal features of the negative samples is farther away. Thus, based on the multimodal coding model, a more accurate alignment result of the knowledge graph can be obtained; based on the alignment result, the data in the knowledge graph can be integrated, redundant data can be eliminated, and the storage space of the database can be saved; when information in the knowledge graph is recommended to users based on the alignment result, the recommended information can better meet the needs of users, thereby improving the accuracy of the recommendation processing.

[0173] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0174] The training method of the multimodal coding model provided in the embodiment of the present application can be applied in the following application scenarios:

[0175] The goal of entity alignment is to find an entity that represents the same object for each entity in the other knowledge graphs in a knowledge graph to be aligned. When the knowledge graph is relatively complete, the aligned entities usually have similar structures (the relationship between entities). In addition, attributes are usually used as characteristics of the entity itself. If the attributes of two entities are very similar, then there is a high probability that the two are aligned. However, both the structure and the attributes are strongly dependent on the completeness of knowledge, and because the diversity of attributes usually introduces a lot of noise, images can be used as supplementary information to determine whether the two entities express the same semantic clues (Image 1 corresponds to the name "Johnson", Image 2 corresponds to the name "Johnson", Image 1 and Image 2 are photos of the same user from different angles. It can be determined that they are the same user through image judgment, but because the names are in different languages, it is difficult to directly determine that they are the same user through name attributes). In the alignment method for multimodal knowledge graphs, the related technology uses a simple fusion method to fuse the multimodal information in the knowledge graph, which is difficult to fully reflect the consistency and difference between the multimodal information. The multimodal encoding model of the related technology has a poor alignment effect on the multimodal knowledge graph.

[0176] The training method of the multimodal coding model provided in the embodiment of the present application is a method in which the trained multimodal coding model can align knowledge graphs of different granularities (for example, aligning general knowledge graphs in the medical field, aligning knowledge graphs in the vertical field of medication, etc.), thereby broadening the knowledge graph's ability to express complex medical knowledge, enriching the existing knowledge dimensions, and will also be of great help to cross-domain integration.

[0177] For example, entity structure highlights the relevance of entities to other entities, expressed through the relationship between entities. Entity attributes highlight the characteristics contained in the entity itself. For example, for a drug entity, its attributes may include drug ingredients, medication time, usage and dosage, etc. Images can express high-level semantics of entities and can perceive the concreteness of entities at the visual level.

[0178] refer to Figure 8 , Figure 8 This is an optional flow chart of the training method of the multimodal coding model provided in the embodiment of the present application. Figure 8 The steps shown are explained.

[0179] In step 801, the multimodal encoding model to be trained is called to encode the sample knowledge graph to obtain the modal features of each entity.

[0180] For example, for the convenience of explanation, the embodiment of the present application uses a pair of sample knowledge graphs as training samples for illustration, and the number of sample knowledge graphs can be greater. The first sample knowledge graph KG1 and the second sample knowledge graph KG2 are used as training samples, and the first sample knowledge graph KG1 and the second sample knowledge graph KG2 are pre-aligned to reduce the amount of calculation in the training process, save the computing resources consumed by the training model, and improve the training efficiency.

[0181] For example, the types of modalities include structural modality, attribute modality, and image modality. Figure 6 , Figure 6 60 is a schematic diagram of the encoding process of the multimodal encoding model provided in the embodiment of the present application; the multimodal encoding model includes a structure encoder 601, an attribute encoder 602 and an image encoder 603. Figure 7 , Figure 7 It is a structural diagram of the multimodal encoding model provided in an embodiment of the present application. The multimodal encoding model 700 also includes a feature fusion module 703 and a feature alignment module 704. Each encoder encodes the modal data corresponding to the entity in the knowledge graph to obtain the modal features of each modality of the entity. The feature fusion module 703 performs splicing processing on each modal feature of the entity to obtain a fused feature. The feature alignment module 704 determines the similarity between the fused feature of the entity and the modal features of each entity in other knowledge graphs, takes at least one entity at the head in the descending order of similarity as the alignment result of the entity, obtains the alignment result between the knowledge graphs based on the alignment result of each entity in the knowledge graph, and outputs the alignment result between the knowledge graphs.

[0182] For structural modes, structure refers to the relationship between entities, refer to Figure 7 , Figure 7It is a structural schematic diagram of the multimodal coding model structure encoder 700 provided in an embodiment of the present application; the structure encoder 601 can be a multi-level graph convolutional neural network model, including multiple graph convolutional layers 701A. The graph convolutional neural network model maps each knowledge graph to the same feature space, converts it into an entity relationship graph corresponding to each entity, and encodes the entity relationship graph of each entity. The entity relationship graph includes entity nodes (corresponding to entities) and neighbor nodes (corresponding to other entities associated with the entity). The features of the neighboring nodes of each graph convolutional layer 701A are aggregated with the features of the entity nodes by weighted calculation (the features of the neighboring nodes are integrated into the features of the entity nodes), that is, the entity neighbors are used to represent the semantic features of the entity. l is an integer greater than or equal to zero, and the structural modal properties of the outputs of the multiple graph convolutional layers 701A of the l+1th layer can be expressed as the following formula (1):

[0183]

[0184] in,[·] + Represents the ReLU activation function; is the adjacency matrix M of KG1 and KG2 (the adjacency matrix represents the relationship between nodes and is an n-order square matrix, where n is the number of nodes and is greater than or equal to 1. The adjacency matrix of the knowledge graph is a symmetric matrix) and the diagonal matrix I N The sum of (l) Represents a layer-specific trainable parameter matrix; Representation Matrix degree matrix (the degree matrix Degree is a diagonal matrix, and the elements on the diagonal are the degrees of each node in the knowledge graph. The degree of a node represents the number of edges associated with the node.); H (l) The output result H of the last graph convolution layer 701A is used in this embodiment. (l+1) As the structural modal features of the entity (which can be represented in the form of feature vectors), since the knowledge graph contains the relationship between each entity and other entities, the entity structural features can also express the structural information of the knowledge graph.

[0185] Regarding attribute modality, since the total number of attributes in the knowledge graph is large, but the number of attributes for a single entity is small, in some embodiments, only the most frequent attributes (for example, 1,000) can be considered to obtain attribute modal features. A one-hot vector is constructed based on the attribute data corresponding to each entity in the knowledge graph. Each dimension of the one-hot vector corresponds to an attribute. The component value corresponding to the attribute can be represented by 0 or 1. 0 indicates that the entity contains this attribute, and 1 indicates that the entity does not have this attribute. For example: each dimension of the one-hot vector is: dosage, usage interval, whether it is a prescription drug, etc. The attributes included for a certain drug are: dosage, usage interval, and over-the-counter drug. Then the one-hot vector of this drug entity is [1.1.0]. Continue to refer to Figure 7 , the attribute encoder 602 includes a data conversion unit 701B and a feedforward neural network 702B. For example, the attribute unique hot vector is a high-dimensional sparse vector. In high-dimensional statistics, information is sparse in features, that is, not every feature is equally important, and the characteristic that important features account for a very small proportion is called sparse. The feedforward neural network 702B is used to perform dimensionality reduction processing on the attribute unique hot vector of the entity to obtain attribute modal features (represented in the form of feature vectors).

[0186] The attribute feature matrix F composed of the attribute modal features of each entity in the knowledge graph A , which can be expressed as the following formula (2):

[0187] F A =W A ·A+b A (2)

[0188] Where A represents the feature matrix composed of the unique hot vectors of the attributes of each entity in the knowledge graph; W A , b A Represents trainable parameters.

[0189] In some embodiments, in order to improve the accuracy of the attribute modal feature, the relationship corresponding to the entity can also be used as an attribute to construct an attribute unique hot vector representing the relationship between entities. Each dimension of the unique hot vector corresponds to a relationship, and the parameter corresponding to the relationship can be represented by 0 or 1. 0 indicates that the entity contains this relationship, and 1 indicates that the entity does not have this relationship. The feedforward neural network 702B is called to perform dimensionality reduction processing on the attribute unique hot vector representing the relationship between entities to obtain the structural attribute modal feature (represented in the form of a feature vector).

[0190] The attribute feature matrix F composed of the structural attribute modal features of each entity in the knowledge graph R , which can be expressed as the following formula (3):

[0191] FR =W R ·R+b R (3)

[0192] Where R represents the feature matrix composed of the attribute unique hot vectors of the entity relationship of each entity in the knowledge graph; W R , b R Represents trainable parameters.

[0193] In some embodiments, the attribute encoder 602 further includes a neural network layer for converting F A and F R Further fusion obtains the feature matrix F of all entity attribute modes T , expressed as the following formula (4):

[0194] F T =σ(W T ·[F A ; F R ]+b T ) (4)

[0195] Where [;] represents the concatenation operation between matrices; W T , b T represents a trainable parameter; σ represents the ReLU activation function. T It represents the attribute features corresponding to each entity in the attribute modality. If the attributes of two entities are similar, the feature vectors obtained by modeling the two entities separately are also similar, which provides a strong reference for the multimodal encoding model.

[0196] For image mode, continue to refer to Figure 7 The image encoder 603 includes a convolutional neural network 701C and a feedforward neural network 702C. In the embodiment of the present application, the convolutional neural network 701C is a 152-layer ResNet residual network model. The convolutional neural network 701C can perform multiple levels of convolution operations and pooling operations on the image of the entity, and the pooling result of the previous pooling layer of the last fully connected layer in the convolutional neural network 701C is used as the image feature vector. The feedforward neural network 702C performs dimensionality reduction processing on the image feature vector to obtain the image modality feature. The matrix F composed of the image modality features of each entity in the knowledge graph I It can be expressed as the following formula (5):

[0197] F I =W I RESNET(L)+b I (5)

[0198] Among them, L represents the feature matrix composed of the image feature vectors of each entity in the knowledge graph; W I, b I Represents trainable parameters.

[0199] In step 802, the contrast loss of the multimodal coding model is obtained.

[0200] For example, contrast loss includes intra-modality contrast loss and inter-modality contrast loss. Adding the contrast loss corresponding to each modality can obtain the intra-modality contrast loss and inter-modality contrast loss of the multimodal encoding model. The intra-modality contrast loss under each modality is obtained in the same way, and the following takes the attribute modality as an example for explanation.

[0201] For aligned entity pairs (e 1 , e 2 ), the entity e encoded by the expected multimodal encoding model 1 The attribute characteristics and entity e 2 The attribute features of entity e can be close in the feature space (that is, the similarity is higher), while entity e 1 The distance between the attribute features of the entities that are not aligned is farther (that is, the similarity is lower). To achieve the above purpose, it is necessary to construct qualified positive sample pairs and negative sample pairs.

[0202] Aligned entity pairs (e 1 , e 2 ) naturally constitutes a positive sample. Since entity alignment usually assumes a one-to-one mapping, in the knowledge graph KG2, in addition to entity e 2 All entities other than 1 Negative samples, similarly in the knowledge graph KG1 except for entity e 1 All entities other than 2 Negative samples. The negative sample pair can be expressed as the following formula (6.1) and formula (6.2):

[0203]

[0204]

[0205] Among them, Pair neg (KG1→KG2) means that all entities in the second knowledge graph KG2 except entity e 2 Each entity of 1 negative samples, forming a negative sample pair, n is a positive integer, which is the number of entities in the knowledge graph. neg (KG2→KG1) means that all entities in the first knowledge graph KG1 except entity e 1 Each entity of 2In the embodiment of the present application, the smaller the intra-modal contrast loss is, the closer the distance between each entity and its aligned entity in the feature space is, and the farther the distance between the non-aligned entity is, then the intra-modal contrast loss can be expressed as the following formula (7.1) and formula (7.2), formula (7.3):

[0206]

[0207]

[0208]

[0209] Where f(·) represents a similarity measurement function, such as cosine similarity (the distance between entities is negatively correlated with cosine similarity. The closer the distance, the closer the similarity is to 1, and the farther the distance, the closer the cosine similarity is to 0); Represents entity e 1 The corresponding intra-modality contrastive loss, The smaller the entity e 1 With entity e 2 The closer the distance of the modal features, the 1 In addition to e in the second knowledge graph KG2 2 The farther the distance between the modal features of the entities outside. Represents entity e 2 The corresponding intra-modality contrastive loss, The smaller the entity e 2 With entity e 1 The closer the distance of the modal features, the 2 In the first knowledge graph KG1, except for entity e 1 The farther the distance between the modal features of the entities outside the modality is, the greater the distance between the modal features of the entities outside the modality is. In each modality, the average value of the intra-modal contrast loss corresponding to the two entities is taken as the intra-modal contrast loss L intra .

[0210] In some embodiments, since each entity in the knowledge graph is independent and non-repetitive, in order to increase the distinction between different entities in the same knowledge graph, the entities in the same modality and the same knowledge graph can be further compared and learned. 1 Any entity in the knowledge graph KG1 can be regarded as a negative sample pair. Among them, the negative sample pairs in the knowledge graph KG1 can be expressed as the following formula (8.1) and formula (8.2):

[0211]

[0212]

[0213] Where Pair neg (KG1) represents entity e 1 In the negative sample pair in the first knowledge graph KG1, any two entities constitute a negative sample pair; accordingly, Pair neg (KG2) represents entity e 2 Negative sample pairs in the second knowledge graph KG2. Furthermore, based on the above negative sample pairs, the intra-modal contrast loss can be expressed as the following formula (7.1), formula (9.1), and formula (9.2):

[0214]

[0215]

[0216]

[0217]

[0218]

[0219] Formula (7.1), Formula (9.1), Formula (9.2) represent that for two aligned entity pairs (e 1 , e 2 ), The smaller the entity e 1 The farther the distance from the entities that are not aligned in the knowledge graph KG2, the farther the distance from other entities in the knowledge graph KG1 that are not itself. Similarly, The smaller the entity e 2 The farther the distance from the entities that are not aligned in the knowledge graph KG1, the farther the distance from other entities in the knowledge graph KG2 that are not itself.

[0220] The following example shows that "Nifedipine Tablets" (belonging to knowledge graph KG1) and "Xintongding Tablets" (belonging to knowledge graph KG2) are two aligned entities. The purpose of contrastive learning is to shorten the feature distance between the modal features of these two entities and to increase the feature distance between the modal features of the two non-aligned entities "Nifedipine Tablets" and "Weitongding Tablets" (belonging to knowledge graph KG2). In addition, although the names of "Xintongding Tablets" and "Weitongding Tablets" are very similar, they are actually two completely different entities. In order to enhance the discriminability of the multimodal encoding model and the distinguishability between entities, the distance between the features of "Xintongding Tablets" and "Weitongding Tablets" output by the trained multimodal encoding model should be farther than before the optimization training.

[0221] For example, the intra-modal contrast loss is calculated for each modality, and the total intra-modal contrast loss is the sum of the intra-modal contrast losses of each modality, which is expressed as the following formula (10):

[0222]

[0223] Among them, H represents structural mode; A represents attribute mode; and I represents image mode.

[0224] Regarding the contrast loss between modalities, for the same entity, although different modal features represent different modalities, the high-level semantic features represented by different modal features are consistent. The embodiment of the present application can enhance the consistency of different modalities by additionally obtaining the contrast loss between modalities and reducing the contrast loss between modalities of the multimodal encoding model. Assumption: In the knowledge graph KG1, entity e 1 The structural features, attribute features and image features are b H , b A , b I , the inter-modality contrast loss is expressed as the following formula (11):

[0225] L inter =||b H -b A ||+||b H -b I ||+||b A -b I || (11)

[0226] Inter-modality contrast loss L inter The smaller it is, the closer the feature distance between the modal features of different modes of the same entity is, so that the modal features of different modes remain consistent.

[0227] In step 803, back-propagation processing is performed on the multimodal coding model based on the contrastive loss to update the parameters of the multimodal coding model.

[0228] For example, based on the intra-modal contrast loss L intra And the inter-modal contrast loss L inter , the multimodal feature encoder model is trained by back propagation to obtain the optimized multimodal encoding model.

[0229] In step 804, the knowledge graphs to be aligned are aligned based on the trained multimodal encoding model.

[0230] For example, obtain the third knowledge graph KG3 to be aligned and the fourth knowledge graph KG4 to be aligned, call the trained multimodal feature encoder to encode the third knowledge graph KG3 and the fourth knowledge graph KG4, and obtain the structural features, attribute features and image features of each entity in each knowledge graph. Obtain the features of each modality of each entity in the knowledge graph, and perform weighted calculation based on each modality feature for each entity to obtain the fusion feature of the entity. Determine the cosine similarity between the fusion feature of the entity to be aligned in the first knowledge graph and the fusion feature of each entity in the second knowledge graph, sort all cosine similarities in descending order, and take at least one entity at the head of the descending order as the alignment result of the entity to be aligned.

[0231] The embodiment of the present application improves the accuracy of multimodal knowledge graph encoding by obtaining intra-modal contrast loss and inter-modal contrast loss, so that the feature distance between the modal features of the encoded positive samples is closer, while the distance between the modal features of the negative samples is farther, thereby improving the distinction between the fusion features of the entities, and obtaining a more accurate knowledge graph alignment result, which is beneficial to improving the recommendation effect of information recommendation based on the knowledge graph.

[0232] The following continues to describe the exemplary structure of the multimodal coding model training device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the training device 455 of the multimodal coding model stored in the memory 450 may include: an encoding module 4551, configured to call the multimodal coding model to be trained, and encode each sample entity in the first sample knowledge graph and the second sample knowledge graph to obtain modal features of at least two modalities of each sample entity, wherein the first sample knowledge graph and the second sample knowledge graph are aligned with each other; a loss acquisition module 4552, configured to perform the following processing on each sample entity: based on each modal feature of the sample entity, determine the inter-modal sub-contrast loss corresponding to the sample entity; the loss acquisition module is also configured to convert each sample entity in the first sample knowledge graph into a loss feature; The first sample knowledge graph and the sample entities aligned with each sample entity in the second sample knowledge graph form an aligned entity pair; the loss acquisition module 4552 is further configured to perform the following processing for each aligned entity pair under each modality: based on the two modal features of the aligned entity pair, determine the intra-modal sub-contrast loss of the aligned entity pair; the loss acquisition module 4552 is configured to determine the total inter-modal contrast loss of the multimodal coding model based on each inter-modal sub-contrast loss, and determine the total intra-modal contrast loss of the multimodal coding model based on each intra-modal sub-contrast loss; the training module 4553 is configured to update the parameters of the multimodal coding model based on the total inter-modal contrast loss and the total intra-modal contrast loss.

[0233] In some embodiments, the loss determination module 4552 is further configured to determine a feature distance between any two modal features of the sample entity; and the sum of each feature distance is used as the inter-modal sub-contrast loss corresponding to the sample entity.

[0234] In some embodiments, the aligned entity pair includes a first sample entity and a second sample entity, the first sample entity belongs to the first sample knowledge graph, and the second sample entity belongs to the second sample knowledge graph; the loss determination module 4552 is also configured to determine the first modal intra-sub-loss of the sample entity pair based on the first feature similarity between the modal features of the first sample entity and the modal features of the second sample entity aligned with the first sample entity, and the second feature similarity between the modal features of the first sample entity and the modal features of each negative sample of the first sample entity; wherein the negative sample of the first sample entity is a sample entity other than the second sample entity in the second sample knowledge graph; based on the first feature similarity between the modal features of the first sample entity and the modal features of the second sample entity aligned with the first sample entity, and the third feature similarity between the modal features of the second sample entity and the modal features of each negative sample of the second sample entity, determine the second modal intra-sub-loss of the sample entity pair; wherein the negative sample of the second sample entity is a sample entity other than the first sample entity in the first sample knowledge graph; the average of the first modal intra-sub-loss and the second modal intra-sub-loss is used as the modal intra-sub-contrast loss of the aligned entity pair.

[0235] In some embodiments, the loss determination module 4552 is also configured to obtain the sum of each second feature similarity corresponding to the first sample entity, and use the sum of each second feature similarity and the sum of the first feature similarity as a first intermediate parameter; and use the ratio of the first feature similarity to the first intermediate parameter as the first modal intra-sub-loss of the sample entity pair.

[0236] In some embodiments, the loss determination module 4552 is further configured to obtain the sum of each third feature similarity corresponding to the second sample entity, and use the sum of each third feature similarity and the sum of the first feature similarity as the second intermediate parameter; and use the ratio of the first feature similarity to the second intermediate parameter as the second modal intra-sub-loss of the sample entity pair.

[0237] In some embodiments, the aligned entity pair includes a first sample entity and a second sample entity; the first sample entity belongs to the first sample knowledge graph, and the second sample entity belongs to the second sample knowledge graph; the loss determination module 4552 is further configured to determine the first intra-modal sub-loss of the sample entity pair based on the first feature similarity between the modal features of the first sample entity and the modal features of the second sample entity aligned with the first sample entity, and the second feature similarity between the modal features of the first sample entity and the modal features of each negative sample of the first sample entity, wherein the negative sample of the first sample entity is a sample entity other than the first sample entity in the first sample knowledge graph, and the second sample knowledge graph is a sample entity other than the first sample entity in the second sample knowledge graph. The method comprises the following steps: determining a second modal intra-sub-loss of the sample entity pair based on a first feature similarity between the modal feature of the first sample entity and the modal feature of the second sample entity aligned with the first sample entity, and a second feature similarity between the modal feature of the second sample entity and the modal feature of each negative sample of the second sample entity, wherein the negative samples of the second sample entity are sample entities other than the second sample entity in the second sample knowledge graph and sample entities other than the first sample entity in the first sample knowledge graph; taking the average of the first modal intra-sub-loss and the second modal intra-sub-loss as the modal intra-sub-contrast loss of the aligned entity pair.

[0238] In some embodiments, the training module 4553 is configured to perform back-propagation processing on the multimodal coding model based on the total contrast loss between modalities and the total contrast loss within modality to obtain updated parameters of the multimodal coding model; and replace the corresponding original parameters in the multimodal coding model based on the updated parameters.

[0239] In some embodiments, the loss determination module 4552 is further configured to add each inter-modal sub-contrast loss to obtain the total inter-modal contrast loss of the multimodal coding model; and add the intra-modal sub-contrast loss of each modality corresponding to each entity to obtain the total intra-modal contrast loss of the multimodal coding model.

[0240] In some embodiments, the types of modalities include: structural modality, attribute modality and image modality; the encoding module 4551 is also configured to call the multimodal encoding model to perform the following processing on each sample entity in the first sample knowledge graph and the second sample knowledge graph: based on the sample knowledge graph corresponding to the sample entity, determine the structural modal data, attribute modal data and image modal data corresponding to the sample entity, wherein the structural modal data characterizes the relationship between the sample entity and other sample entities; perform mapping processing on the structural modal data, and perform convolution processing on the entity relationship graph obtained by the mapping processing to obtain the structural modal features of the sample entity; perform combination processing on the attribute modal data, and perform dimensionality reduction processing on the combined feature vector obtained by the combination processing to obtain the attribute modal features of the sample entity, wherein the dimension of the combined feature vector is higher than the dimension of the attribute modal features; perform image feature extraction processing on the image modal data to obtain the image modal features of the sample entity.

[0241] In some embodiments, the encoding module 4551 is also configured to perform mapping processing on the structural modal data to obtain an entity relationship diagram of the entity, wherein the entity relationship diagram includes: an entity node and multiple neighbor nodes corresponding to the entity node; multiple feature aggregation processing is performed based on the entity node and each neighbor node in the entity relationship diagram to obtain the structural modal features of the sample entity.

[0242] In some embodiments, the encoding module 4551 is further configured to perform combination processing on each attribute corresponding to the attribute modal data to obtain a combined feature vector of the entity; and perform dimensionality reduction mapping processing on the combined feature vector to obtain the attribute modal characteristics of the sample entity.

[0243] In some embodiments, the types of attribute modal data include: relationship attribute data, parameter attribute data; the encoding module 4551 is also configured to combine each attribute corresponding to the parameter attribute data to obtain a parameter combination feature vector of the entity; determine the relationship feature vector of the entity based on the relationship attribute data, wherein the component value corresponding to each dimension in the relationship feature vector represents: the relationship between the sample entity and the entity corresponding to each dimension; perform dimensionality reduction mapping processing on the parameter high-dimensional feature vector and the relationship feature vector, respectively, and fuse the mapped parameter low-dimensional feature vector and the relationship low-dimensional feature vector to obtain the attribute modal characteristics of the sample entity.

[0244] In some embodiments, the encoding module 4551 is also configured to perform multi-level convolution operations and pooling operations on the image modality data to obtain an image feature vector of the entity; and perform dimensionality reduction mapping processing on the image feature vector to obtain image modality features of the sample entity.

[0245] In some embodiments, the encoding module 4551 is also configured to call the updated multimodal encoding model to perform encoding processing on the third knowledge graph and the fourth knowledge graph to be aligned, and obtain the modal features of each modality of each entity in the third knowledge graph and the fourth knowledge graph; perform the following processing on each entity: perform feature splicing processing based on each modal feature of the entity to obtain the splicing features of the entity; perform the following processing on each entity in the third knowledge graph: determine the similarity between the splicing features of the entity and the splicing features of each entity in the fourth knowledge graph; take at least one entity of the fourth knowledge graph at the head of the entity sorted in descending order of similarity as the alignment result of the entity; combine the alignment results of each entity in the third knowledge graph to obtain the alignment result between the third knowledge graph and the fourth knowledge graph.

[0246] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the multimodal coding model described in the embodiment of the present application.

[0247] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the training method of the multimodal encoding model provided in the present application embodiment, for example, Figure 3A The training method of the multimodal encoding model is shown.

[0248] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0249] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0250] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0251] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0252] To summarize, by obtaining the intra-modality contrast loss and the inter-modality contrast loss through the embodiments of the present application, the multimodal knowledge graph encoding model can reduce the encoding loss from both inter-modality and intra-modality aspects, making the encoding result more accurate, thereby improving the accuracy of the multimodal knowledge graph encoding, and then the multimodal encoding model can perform more accurate alignment processing on the knowledge graph.

[0253] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A training method for a multimodal coding model, characterized in that: The method comprises: Calling the multimodal encoding model to be trained, encoding each sample entity in the first sample knowledge graph and the second sample knowledge graph, and obtaining modal features of at least two modalities of each of the sample entities, wherein the first sample knowledge graph and the second sample knowledge graph are aligned with each other, and the at least two modalities include at least an image modality, and the encoding process at least includes: determining image modality data based on the sample knowledge graph corresponding to the sample entity, performing image feature extraction on the image modality data, and obtaining image modality features of the sample entity; The following processing is performed for each of the sample entities: based on each of the modal features of the sample entity, determining the inter-modal sub-contrast loss corresponding to the sample entity; Each of the sample entities in the first sample knowledge graph and the sample entities aligned with each of the sample entities in the second sample knowledge graph form an aligned entity pair; Under each of the modalities, the following processing is performed for each of the aligned entity pairs: based on the two modal features of the aligned entity pair, an intra-modal sub-contrast loss of the aligned entity pair is determined; Determining an inter-modality total contrast loss of the multimodal coding model based on each of the inter-modality sub-contrast losses, and determining an intra-modality total contrast loss of the multimodal coding model based on each of the intra-modality sub-contrast losses; Based on the inter-modality total contrast loss and the intra-modality total contrast loss, the parameters of the multimodal encoding model are updated, wherein the updated multimodal encoding model is used to align the knowledge graph containing the image modality to obtain an alignment result.

2. The method according to claim 1, characterized in that The determining, based on each of the modal features of the sample entity, an inter-modal sub-contrast loss corresponding to the sample entity comprises: Determining a feature distance between any two modal features of the sample entity; The sum of each of the feature distances is used as the inter-modal sub-contrast loss corresponding to the sample entity.

3. The method according to claim 1, characterized in that The aligned entity pair includes a first sample entity and a second sample entity, the first sample entity belongs to the first sample knowledge graph, and the second sample entity belongs to the second sample knowledge graph; The determining, based on the two modal features of the aligned entity pair, the intra-modal sub-contrast loss of the aligned entity pair comprises: Determine a first modal intra-sub-loss of the sample entity pair based on a first feature similarity between a modal feature of the first sample entity and a modal feature of the second sample entity aligned with the first sample entity, and a second feature similarity between the modal feature of the first sample entity and the modal feature of each negative sample of the first sample entity; wherein the negative sample of the first sample entity is a sample entity other than the second sample entity in the second sample knowledge graph; Determine a second modal intra-sub-loss of the sample entity pair based on a first feature similarity between a modal feature of the first sample entity and a modal feature of the second sample entity aligned with the first sample entity, and a third feature similarity between the modal feature of the second sample entity and the modal feature of each negative sample of the second sample entity; wherein the negative sample of the second sample entity is a sample entity other than the first sample entity in the first sample knowledge graph; The average of the first modal intra-sub-loss and the second modal intra-sub-loss is used as the modal intra-sub-contrast loss of the aligned entity pair.

4. The method according to claim 3, characterized in that The determining of a first intra-modal sub-loss of the sample entity pair based on a first feature similarity between the modal feature of the first sample entity and the modal feature of the second sample entity aligned with the first sample entity, and a second feature similarity between the modal feature of the first sample entity and the modal feature of each negative sample of the first sample entity, comprises: Obtaining a sum of the similarities of each of the second features corresponding to the first sample entity, and using the sum of the similarities of each of the second features and the sum of the similarities of the first features as a first intermediate parameter; Using the ratio of the first feature similarity to the first intermediate parameter as the first modal intra-sub-loss of the sample entity pair; The determining of a second modality intra-sub-loss of the sample entity pair based on a first feature similarity between the modality feature of the first sample entity and the modality feature of the second sample entity aligned with the first sample entity, and a third feature similarity between the modality feature of the second sample entity and the modality feature of each negative sample of the second sample entity, comprises: Obtaining a sum of similarities of each third feature corresponding to the second sample entity, and taking the sum of each third feature similarity and the sum of the first feature similarity as a second intermediate parameter; The ratio of the first feature similarity to the second intermediate parameter is used as the second modal intra-sub-loss of the sample entity pair.

5. The method according to claim 1, characterized in that The aligned entity pair includes a first sample entity and a second sample entity; the first sample entity belongs to the first sample knowledge graph, and the second sample entity belongs to the second sample knowledge graph; The determining, based on the two modal features of the aligned entity pair, the intra-modal sub-contrast loss of the aligned entity pair comprises: Determine a first intra-modal sub-loss of the sample entity pair based on a first feature similarity between a modal feature of the first sample entity and a modal feature of the second sample entity aligned with the first sample entity, and a second feature similarity between the modal feature of the first sample entity and the modal feature of each negative sample of the first sample entity, wherein the negative samples of the first sample entity are sample entities other than the first sample entity in the first sample knowledge graph, and sample entities other than the second sample entity in the second sample knowledge graph; Determine a second modal intra-sub-loss of the sample entity pair based on a first feature similarity between a modal feature of the first sample entity and a modal feature of the second sample entity aligned with the first sample entity, and a second feature similarity between the modal feature of the second sample entity and the modal feature of each negative sample of the second sample entity, wherein the negative samples of the second sample entity are sample entities other than the second sample entity in the second sample knowledge graph, and sample entities other than the first sample entity in the first sample knowledge graph; The average of the first modal intra-sub-loss and the second modal intra-sub-loss is used as the modal intra-sub-contrast loss of the aligned entity pair.

6. The method according to claim 1, characterized in that The updating of the parameters of the multimodal coding model based on the inter-modality total contrast loss and the intra-modality total contrast loss includes: Based on the inter-modality total contrast loss and the intra-modality total contrast loss, back-propagation processing is performed on the multimodal coding model to obtain updated parameters of the multimodal coding model; The corresponding original parameters in the multimodal coding model are replaced based on the updated parameters.

7. The method according to claim 1, characterized in that The determining of the inter-modality total contrast loss of the multimodal coding model based on each of the inter-modality sub-contrast losses, and the determining of the intra-modality total contrast loss of the multimodal coding model based on each of the intra-modality sub-contrast losses, comprises: Adding each of the inter-modal sub-contrastive losses to obtain a total inter-modal contrastive loss of the multimodal coding model; The intra-modality sub-contrast loss of each of the modalities corresponding to each of the entities is added together to obtain the total intra-modality contrast loss of the multi-modal coding model.

8. The method according to claim 1, characterized in that The types of modes include: structural mode and attribute mode; The calling of the multimodal encoding model to be trained to encode each sample entity in the first sample knowledge graph and the second sample knowledge graph to obtain modal features of at least two modalities of each sample entity includes: The multimodal encoding model is called to perform the following processing on each of the sample entities in the first sample knowledge graph and the second sample knowledge graph: Based on the sample knowledge graph corresponding to the sample entity, determine the structural modal data and attribute modal data corresponding to the sample entity, wherein the structural modal data represents the relationship between the sample entity and other sample entities; Performing mapping processing on the structural modal data, and performing convolution processing on the entity relationship graph obtained by the mapping processing to obtain the structural modal features of the sample entity; The attribute modal data are combined and processed, and a combined feature vector obtained based on the combined processing is subjected to dimensionality reduction processing to obtain the attribute modal features of the sample entity, wherein the dimension of the combined feature vector is higher than the dimension of the attribute modal features.

9. The method according to claim 8, characterized in that The mapping process is performed on the structural modal data, and the convolution process is performed on the entity relationship graph obtained by the mapping process to obtain the structural modal features of the sample entity, including: Mapping the structural modal data to obtain an entity relationship graph of the entity, wherein the entity relationship graph includes: an entity node and a plurality of neighbor nodes corresponding to the entity node; Multiple feature aggregation processes are performed based on the entity nodes in the entity relationship graph and each of the neighboring nodes to obtain the structural modal features of the sample entity.

10. The method according to claim 8, characterized in that The combining process of the attribute modal data and the dimensionality reduction process of the combined feature vector obtained by the combining process to obtain the attribute modal features of the sample entity include: Combining each attribute corresponding to the attribute modal data to obtain a combined feature vector of the entity; The combined feature vector is subjected to dimensionality reduction mapping processing to obtain the attribute modal features of the sample entity.

11. The method according to claim 8, characterized in that The types of attribute modal data include: relationship attribute data and parameter attribute data; The combining process of the attribute modal data and the dimensionality reduction process of the combined feature vector obtained by the combining process to obtain the attribute modal features of the sample entity include: Combining each attribute corresponding to the parameter attribute data to obtain a parameter combination feature vector of the entity; Determine a relational feature vector of the entity based on the relational attribute data, wherein the component value corresponding to each dimension in the relational feature vector represents: the relationship between the sample entity and the entity corresponding to each dimension; The parameter high-dimensional feature vector and the relationship feature vector are respectively subjected to dimensionality reduction mapping processing, and the mapped parameter low-dimensional feature vector and the relationship low-dimensional feature vector are fused to obtain the attribute modal features of the sample entity.

12. A training device for a multimodal coding model, characterized in that: The training device of the multimodal coding model comprises: The encoding module is configured to call the multimodal encoding model to be trained, perform encoding processing on each sample entity in the first sample knowledge graph and the second sample knowledge graph, and obtain modal features of at least two modalities of each of the sample entities, wherein the first sample knowledge graph and the second sample knowledge graph are aligned with each other, and the at least two modalities include at least an image modality, and the encoding processing at least includes: determining image modal data based on the sample knowledge graph corresponding to the sample entity, performing image feature extraction processing on the image modal data, and obtaining image modal features of the sample entity; The loss acquisition module is configured to perform the following processing for each of the sample entities: based on each of the modal features of the sample entity, determine the inter-modal sub-contrast loss corresponding to the sample entity; The loss acquisition module is further configured to form an aligned entity pair with each of the sample entities in the first sample knowledge graph and the sample entities aligned with each of the sample entities in the second sample knowledge graph; The loss acquisition module is further configured to perform the following processing for each of the aligned entity pairs in each of the modalities: determining the intra-modal sub-contrast loss of the aligned entity pair based on the two modal features of the aligned entity pair; The loss acquisition module is configured to determine the inter-modality total contrast loss of the multimodal coding model based on each of the inter-modality sub-contrast losses, and determine the intra-modality total contrast loss of the multimodal coding model based on each of the intra-modality sub-contrast losses; A training module is configured to update the parameters of the multimodal encoding model based on the inter-modality total contrast loss and the intra-modality total contrast loss, wherein the updated multimodal encoding model is used to align the knowledge graph containing the image modality to obtain an alignment result, and the alignment result is used for recommendation processing between entities.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, used to implement the training method of the multimodal coding model described in any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the training method of the multimodal coding model described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, it implements the training method of the multimodal coding model described in any one of claims 1 to 11.

Images