Entity linking method and device, equipment and storage medium

Abstract

The invention discloses an entity linking method and device, equipment and a storage medium, relates to the technical field of computers, and is used for realizing a mode of carrying out entity linking based on visual contents so as to expand the application field of an entity linking technology and ensure that information obtained based on the entity linking technology is richer and more accurate. The method comprises the following steps: acquiring a visual feature vector of visual content to be linked and an entity feature vector corresponding to at least one candidate entity in a knowledge graph; wherein the entity feature vectors comprise attribute dimensions in the multimedia information corresponding to the corresponding candidate entities, and the attribute dimensions comprise at least one of images and texts; respectively determining a target similarity between the at least one obtained entity feature vector and the visual feature vector; and in the at least one candidate entity, screening out a target entity of which the target similarity meets a preset similarity condition, and taking the target entity as an entity linking result of the to-be-linked visual content.

Description

Background Art

[0002] With the development of science and technology, online information continues to increase, and more and more applications provide information search, content recommendation and other functions to provide users with information they are interested in. For example, shopping, video, news and information applications all provide search services, and users can enter search content in the search box of each application to obtain relevant information.

[0003] At present, it is proposed to provide information to users through entity linking technology; that is, link to the target entity in the knowledge graph through entity linking technology, and provide the user with the information associated with the target entity. However, the current entity linking technology is only used to link the mentioned words (that is, the entity name) in the natural language text to the target entity in the knowledge graph, that is, link the target entity based on text information.

[0004] In real life, information can be in the form of videos, images, graphics, and text in addition to text information. However, the current entity linking technology is limited to text applications and cannot be extended to the visual field. Therefore, it is impossible to accurately link to the target entities associated with videos, images, graphics, and text, that is, less information is obtained through entity linking technology.

[0005] Therefore, it is necessary to learn how to perform entity linking based on visual content in order to expand the application field of entity linking and ensure that the acquired information is richer and more accurate. Summary of the invention

[0006] The embodiments of the present application provide an entity linking method, apparatus, device and storage medium for implementing entity linking based on visual content, so as to expand the application field of entity linking technology and ensure that the information obtained based on entity linking technology is richer and more accurate.

[0007] In a first aspect, an embodiment of the present application provides an entity linking method, the method comprising:

[0008] Obtaining a visual feature vector of the visual content to be linked, and an entity feature vector corresponding to each of at least one candidate entity in the knowledge graph; wherein the entity feature vector includes: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, and the attribute dimension includes at least one of an image and a text;

[0009] respectively determining a target similarity between the obtained at least one entity feature vector and the visual feature vector;

[0010] In at least one candidate entity, a target entity whose target similarity satisfies a preset similarity condition is screened out, and the target entity is used as an entity linking result of the visual content to be linked.

[0011] In a second aspect, an embodiment of the present application provides a physical link device, the device comprising:

[0012] An acquisition unit is used to acquire a visual feature vector of the visual content to be linked, and an entity feature vector corresponding to each of at least one candidate entity in the knowledge graph; wherein the entity feature vector includes: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, and the attribute dimension includes at least one of an image and a text;

[0013] A determination unit, used to respectively determine the target similarity between the obtained at least one entity feature vector and the visual feature vector;

[0014] The screening unit is used to screen out a target entity whose target similarity meets a preset similarity condition from at least one candidate entity, and use the target entity as an entity linking result of the visual content to be linked.

[0015] In a possible implementation, the acquiring unit is specifically configured to:

[0016] Using the trained contrastive learning model, perform the following operations:

[0017] encoding the visual content to be linked by using a visual encoder in a trained contrastive learning model to obtain a visual feature vector of the visual content to be linked; and

[0018] The multimedia information corresponding to at least one candidate entity is encoded respectively through the entity encoder in the trained contrastive learning model to obtain the entity feature vector corresponding to each of the at least one candidate entity.

[0019] In a possible implementation, the multimedia information includes image information and text information, and the entity encoder includes: an image encoder, a text encoder, and a cross-modal encoder; then the acquisition unit is specifically used to:

[0020] Encoding the image information through an image encoder to obtain an image feature vector, and encoding the text information through a text encoder to obtain a text feature vector;

[0021] The image feature vector and the text feature vector are fused through a cross-modal encoder to obtain a multimodal feature vector, which is used as the entity feature vector.

[0022] In one possible implementation, the contrastive learning model is obtained as follows:

[0023] Based on the training sample set, a cyclic iterative training is performed on the contrastive learning model to be trained to obtain a trained contrastive learning model, wherein the following steps are performed during one cyclic iterative process:

[0024] Selecting training samples from a training sample set; wherein the training samples include: sample visual content, and an entity label set corresponding to the sample visual content;

[0025] Input the training sample into the contrastive learning model to obtain the sample visual feature vector of the sample visual content, and obtain the label entity feature vector corresponding to each entity label in the entity label set;

[0026] Based on the predicted similarity between the feature vector of each label entity and the sample visual feature vector, a target loss function is constructed, and the target loss function is used to adjust the parameters.

[0027] In a possible implementation, the entity tag set includes at least: a positive entity tag and a negative entity tag;

[0028] A positive entity label is an entity in the knowledge graph whose relevance to the sample visual content reaches the first relevance threshold;

[0029] Negative entity labels are entities in the knowledge graph, except for positive entity labels, whose relevance to the sample visual content reaches the second relevance threshold;

[0030] The first correlation threshold is greater than the second correlation threshold.

[0031] In one possible implementation, a target loss function is constructed based on the predicted similarity between each entity label feature vector and the sample visual feature vector, including:

[0032] Constructing a target loss function based on the number of positive entity tags in the entity tag set and the first similarity ratios associated with each positive entity tag;

[0033] The first similarity ratio is: the ratio of the predicted similarity associated with the positive entity tag to the first comprehensive similarity associated with the entity tag set; the first comprehensive similarity is: the sum of the similarities associated with all entity tags in all entity tag sets.

[0034] In a possible implementation, the entity tag set further includes: an entity tag queue of a fixed length;

[0035] The entity tag queue is updated in the following way:

[0036] After the current loop iteration round ends, the positive entity tag used in the current loop iteration round is stored in the entity tag queue; and

[0037] Based on the number of used positive entity tags, entity tags are selected and deleted from the entity tag queue in the order of the time when the entity tags were stored.

[0038] In a possible implementation, based on the predicted similarity between each label entity feature vector and the sample visual feature vector, a target loss function is constructed, including:

[0039] Constructing a target loss function based on the target soft labels associated with each entity label and the second similarity ratios associated with each entity label;

[0040] Among them, the target soft label represents: the similarity between the entity label and the sample visual content; the second similarity ratio is: the ratio between the predicted similarity associated with the entity label and the second comprehensive similarity associated with the entity label set; the second comprehensive similarity is: the sum of the similarities associated with all entity labels in all entity label sets.

[0041] In a possible implementation, the target soft label is determined by:

[0042] For each entity tag in the entity tag set, perform the following operations:

[0043] determining a predicted similarity between the entity label and the sample visual content based on a label entity feature vector of the entity label and a sample visual feature vector of the sample visual content associated with the entity label;

[0044] Perform normalized mapping processing on the predicted similarity to obtain the corresponding predicted soft label;

[0045] The predicted soft label and the original hard label are superimposed in a preset ratio to obtain a target soft label; wherein the original hard label is determined based on whether the entity label is a positive entity label.

[0046] In a possible implementation manner, the determining unit is specifically configured to:

[0047] Normalizing at least one entity feature vector respectively to obtain at least one entity feature vector, each of which is normalized to a target entity feature vector in a hyperspherical space;

[0048] Normalizing the visual feature vector to obtain a visual feature vector normalized to a hyperspherical space;

[0049] The similarity between at least one target entity feature vector and the target visual feature vector is determined respectively.

[0050] In a possible implementation, the screening unit is further configured to:

[0051] In at least one candidate entity, a target entity whose target similarity meets a preset similarity condition is screened out, and after the target entity is used as an entity linking result of the visual content to be linked, multimedia information associated with the target entity is recommended to a target object browsing the visual content to be linked.

[0052] In a third aspect, an embodiment of the present application provides a computing device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the steps of the entity linking method provided in the embodiment of the present application.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the entity linking method provided in the embodiment of the present application are implemented.

[0054] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when the processor of a computing device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the computing device executes the steps of the entity linking method provided in the embodiment of the present application.

[0055] The beneficial effects of this application are as follows:

[0056] The embodiments of the present application provide an entity linking method, apparatus, device and storage medium, and relate to the field of computer technology. In the present application, a visual feature vector of the visual content to be linked and an entity feature vector corresponding to at least one candidate entity in the knowledge graph are first obtained, and the entity feature vector includes: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, and the attribute dimension includes at least one of an image and a text, that is, the entity feature vector includes at least one of an image feature vector and a text feature vector. Then, the target similarity between the obtained at least one entity feature vector and the visual feature vector is determined respectively, and a target entity whose target similarity meets a preset similarity condition is screened out from at least one candidate entity, and the target entity is used as the entity linking result of the visual content to be linked. Obviously, the embodiments of the present application propose a method for implementing entity linking based on visual content.

[0057] In the process of entity linking based on visual content, the target similarity between the visual feature vector of the visual content and the text feature vector of the entity, the target similarity between the visual feature vector and the image feature vector of the entity can be determined, and the target similarity between the visual feature vector and the multimodal feature vector of the entity determined based on the text feature vector and the image feature vector can also be determined, and the target entity can be determined based on the target similarity. At this time, the visual feature vector based on the visual content can be linked to the text content, image content, and text + image content, so that entity linking is no longer limited to the text field, which expands the application field of entity linking technology and ensures that the information obtained based on entity linking technology is richer and more accurate.

[0058] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 This is an architecture diagram of an entity linking system provided in the related art;

[0061] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the structure of a comparative learning model provided in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of the structure of another comparative learning model provided in an embodiment of the present application;

[0064] Figure 5 A flow chart of a training method for a comparative learning model provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of constructing an entity tag set provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of another method for constructing an entity tag set provided in an embodiment of the present application;

[0067] Figure 8A training diagram of a contrastive learning model provided in an embodiment of the present application;

[0068] Fig. 9 A schematic diagram for comparing entity linking methods provided in an embodiment of the present application;

[0069] Fig.10 A schematic diagram of a named entity comparison result provided in an embodiment of the present application;

[0070] Fig.11 A schematic diagram of a conceptual entity comparison result provided in an embodiment of the present application;

[0071] Fig.12 A flow chart of an entity linking method provided in an embodiment of the present application;

[0072] Fig.13 A schematic diagram of a specific implementation method of entity linking provided in an embodiment of the present application;

[0073] Fig.14 A schematic diagram of an entity linking application scenario provided in an embodiment of the present application;

[0074] Fig.15 A physical linking device provided in an embodiment of the present application;

[0075] Fig.16 A structural diagram of a computing device provided in an embodiment of the present application;

[0076] Fig.17 A structural diagram of another computing device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and beneficial effects of this application clearer, the technical solution in the embodiment of this application will be clearly and completely described below in conjunction with the drawings in the embodiment of this application. Obviously, the described embodiment is only a part of the embodiment of this application, not all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0078] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0079] Entity Linking (EL), also known as entity linking, is a task that requires identifying the content representing entities in unstructured data (i.e., mentions, which refer to a certain entity) and finding the entity represented by the mention from the knowledge base (domain vocabulary, knowledge graph, etc.). The technology required to complete the entity linking task is called entity linking technology. Entity linking plays a very important basic role in many fields, such as question answering, semantic search, information extraction, content recommendation, etc.

[0080] Entity is the basic unit in the knowledge graph and an important unit that carries information.

[0081] Mentions are content used to represent entities. For example, in natural text, a mention is a language fragment that expresses an entity; in images, a mention is a part of the image that expresses the entity, that is, the region of interest.

[0082] Knowledge Graph, also known as knowledge domain visualization or knowledge domain mapping map, is a series of various graphics that show the development process and structural relationship of knowledge. It uses visualization technology to describe knowledge resources and their carriers, and mines, analyzes, constructs, draws and displays knowledge and their mutual connections, aiming to describe the conceptual entities of the objective world and the relationships between them. It is a modern theory that combines the theories and methods of applied mathematics, graphics, information visualization technology, information science and other disciplines with methods such as metrology citation analysis and co-occurrence analysis, and uses visualized graphs to vividly display the core structure, development history, frontier fields and overall knowledge architecture of the discipline to achieve the purpose of multidisciplinary integration. Knowledge graphs are composed of triples, such as: <entity 1, relationship, entity 2> or <entity, attribute, attribute value>.

[0083] Multimedia information refers to media information in the form of text, images, videos, sounds, animations, etc.; for example, videos, graphics, etc.

[0084] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0085] The terms "first" and "second" in this document are used for descriptive purposes only and should not be understood as explicitly or implicitly indicating relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, unless otherwise specified, the meaning of "multiple" is two or more.

[0086] The following is a brief introduction to the design concept of the embodiments of the present application.

[0087] With the development of technology, online information continues to increase, and more and more applications provide functions such as information search and content recommendation to provide users with information of interest.

[0088] At present, it is proposed to provide information to users through entity linking technology; that is, link to the target entity in the knowledge graph through entity linking technology, and provide the user with the information associated with the target entity. However, the current entity linking technology is only used to link the mentioned words (that is, the entity name) in the natural language text to the target entity in the knowledge graph, that is, link the target entity based on text information.

[0089] In life, information can be in the form of videos, images, graphics, etc. in addition to text information. However, the current entity linking technology is limited to the application in the text field and cannot be extended to the visual field.

[0090] Therefore, a visual entity linking is proposed in the related art. Visual entity linking is the embodiment of the multimodal entity linking task in a single visual modality, focusing on linking visual mentions (objects of interest) in the input image that do not contain text information to the knowledge graph. Multimodal entity linking aims to use visual information as an auxiliary to text content and link the mention words that appear in the text context with the corresponding entities in the knowledge graph.

[0091] See also Figure 1 , Figure 1 The invention discloses an entity linking system in the related art. The visual entity linking method proposed based on the entity linking system mainly includes: object recognition model generation: entity link library generation and entity linking; wherein, the object recognition model generation part includes: collecting and annotating pictures, preprocessing the collected and annotated pictures, building an object recognition model, and training the object recognition model; the entity link library generation part includes: collecting entity corpus, associating entities with picture labels, and obtaining an entity link library; the entity link generation part includes: preprocessing the pictures obtained by shooting, and then inputting them into the object recognition model to obtain object recognition results, and searching the object recognition results in the entity link library to obtain the text results of the entity. The purpose of entity disambiguation is achieved through object recognition of pictures, and multimodal entity linking from pictures to text is realized. Specifically, common objects in life are photographed by camera equipment, and then the objects in the pictures are recognized, and finally the object recognition results are linked to the corresponding entities, thereby realizing entity linking from pictures to text.

[0092] It can be seen that the visual entity linking technology matches objects in images to common entities in the knowledge graph (such as people, clothing, etc.), which is limited to coarse-grained entity links. In addition, the visual entity linking in the related art fine-tunes the multimodal pre-trained model by constructing a one-to-one input image-entity image pair or input image-entity text pair, and then uses the multimodal pre-trained model to achieve effective linking of entities through encoding-retrieval or encoding-generation; however, this fine-tuning method is difficult to effectively migrate to the visual multi-entity linking task, resulting in low recall and accuracy of visual multi-entity linking. Among them, visual multi-entity linking is an improvement on the visual entity linking task, and the main differences include:

[0093] Multiple visual mentions: The input image can contain more than one visual mention linked to the knowledge graph. Visual mentions are objects of interest or concepts associated in the image. Multi-type entities: Both named entities (such as people, places, and organizations) and conceptual entities (such as sports categories and terms) are considered. Incomplete multimodal knowledge: The knowledge graph contains incomplete multimodal knowledge. Only a small number of entities have complete text descriptions and associated images, while a large number of entities are missing a certain modality.

[0094] In view of this, the embodiments of the present application propose an entity linking method, apparatus, device and storage medium, which are applied to the visual multi-entity linking task to implement an entity linking method based on visual content, so as to expand the application field of entity linking technology, improve the recall rate and accuracy of visual multi-entity linking, and ensure that the information obtained based on entity linking technology is richer and more accurate.

[0095] In an embodiment of the present application, a visual feature vector of the visual content to be linked and an entity feature vector corresponding to each of at least one candidate entity in the knowledge graph are first obtained, the entity feature vector comprising: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, the attribute dimension comprising at least one of an image and a text, that is, the entity feature vector comprises at least one of an image feature vector and a text feature vector; then, the target similarity between the obtained at least one entity feature vector and the visual feature vector is respectively determined, and a target entity whose target similarity meets a preset similarity condition is screened out from at least one candidate entity, and the target entity is used as the entity linking result of the visual content to be linked; obviously, an embodiment of the present application proposes a method for implementing entity linking based on visual content.

[0096] In the process of entity linking based on visual content, the target similarity between the visual feature vector of the visual content and the text feature vector of the entity, the target similarity between the visual feature vector and the image feature vector of the entity can be determined, and the target similarity between the visual feature vector and the multimodal feature vector of the entity determined based on the text feature vector and the image feature vector can also be determined, and the target entity can be determined based on the target similarity. At this time, the visual feature vector based on the visual content can be linked to the text content, image content, and text + image content, so that entity linking is no longer limited to the text field, which expands the application field of entity linking technology and ensures that the information obtained based on entity linking technology is richer and more accurate.

[0097] In an embodiment of the present application, in order to more accurately obtain the visual feature vector of the visual content to be linked, and the entity feature vector corresponding to at least one candidate entity in the knowledge graph, a contrastive learning model is proposed, which includes a visual encoder and an entity encoder; the visual content to be linked is encoded by the visual encoder to obtain the visual feature vector; and the multimedia information corresponding to at least one candidate entity is encoded by the entity encoder to obtain the corresponding entity feature vector.

[0098] Among them, the part of the comparative learning model involves artificial intelligence (AI) and machine learning technology, and is designed based on speech technology, natural language processing technology and machine learning (ML) in artificial intelligence.

[0099] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0100] Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology mainly includes computer vision technology, natural language processing technology, and machine learning / deep learning. With the research and progress of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, robots, smart medical care, etc. It is believed that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0101] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared with data mining, which seeks mutual characteristics between big data, machine learning focuses more on algorithm design, allowing computers to automatically "learn" rules from data and use the rules to predict unknown data.

[0102] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. Reinforcement learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent can maximize its rewards or achieve specific goals through learning strategies during its interaction with the environment.

[0103] After introducing the design concept of the embodiment of the present application, the application scenario set in the present application is briefly described below. It should be noted that the following scenario is only used to illustrate the embodiment of the present application and is not limited. In the specific implementation, the technical solution provided by the embodiment of the present application can be flexibly applied according to actual needs.

[0104] See also Figure 2 , Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application. The application scenario includes a terminal device 210 and a server 220, and the terminal device 210 and the server 220 can communicate with each other through a communication network.

[0105] In an optional implementation, the communication network may be a wired network or a wireless network. Therefore, the terminal device 210 and the server 220 may be directly or indirectly connected via a wired or wireless communication mode. For example, the terminal device 210 may be indirectly connected to the server 220 via a wireless access point, or the terminal device 210 may be directly connected to the server 220 via the Internet, and this application does not limit this.

[0106] Among them, the terminal device 210 includes but is not limited to mobile phones, tablet computers, laptops, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; various clients can be installed on the terminal device, and the client can be an online platform or application (such as a browser, game software, shopping software, video player, etc.) that supports functions such as information preview and information playback, or a web page, a small program, etc.; for example, the client instructs video playback, and through the client, a large amount of the latest or hottest video content can be watched online, providing a free, high-definition, and smooth video viewing experience, and videos can be quickly found through channel recommendations, viewing history, favorite videos, and search functions.

[0107] The server 220 is a background server corresponding to the client installed in the terminal device 210. The server 220 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0108] It should be noted that the entity linking method in the embodiment of the present application can be executed by a computing device, which can be a server 220 or a terminal device 210, that is, the method can be executed by the server 220 or the terminal device 210 alone, or can be executed jointly by the server 220 and the terminal device 210.

[0109] When the terminal device 210 executes alone, for example, the terminal device 210 first obtains the visual feature vector of the visual content to be linked, and the entity feature vector corresponding to each of the at least one candidate entity in the knowledge graph; further, the target similarity between the obtained at least one entity feature vector and the visual feature vector is determined respectively; finally, among the at least one candidate entity, the target entity whose target similarity meets the preset similarity condition is screened out, and the target entity is used as the entity linking result of the visual content to be linked.

[0110] When the server 220 is executed alone, for example, the server 220 first obtains the visual feature vector of the visual content to be linked, and the entity feature vector corresponding to each of the at least one candidate entity in the knowledge graph; further, the target similarity between the obtained at least one entity feature vector and the visual feature vector is determined respectively; finally, among the at least one candidate entity, the target entity whose target similarity meets the preset similarity condition is screened out, and the target entity is used as the entity linking result of the visual content to be linked.

[0111] When the server 220 and the terminal device 210 execute together, for example, the terminal device 210 may obtain a visual feature vector of the visual content to be linked, and transmit the visual feature vector to the server 220, and an entity feature vector corresponding to at least one candidate entity in the knowledge graph of the server 220; further, the server 220 respectively determines the target similarity between the obtained at least one entity feature vector and the visual feature vector; finally, the server 220 selects a target entity whose target similarity meets a preset similarity condition from at least one candidate entity, and uses the target entity as the entity linking result of the visual content to be linked.

[0112] In a possible implementation, the entity linking method involved in the embodiment of the present application can be applied to the fields of multimedia information recommendation, advertising, and information retrieval.

[0113] In a possible implementation, in an entity linking method, a contrastive learning model is involved; exemplary, a visual encoder in a trained contrastive learning model is used to encode visual content to be linked, and a visual feature vector of the visual content to be linked is obtained; and multimedia information corresponding to at least one candidate entity is encoded by an entity encoder in the trained contrastive learning model, and an entity feature vector corresponding to at least one candidate entity is obtained. Further, the target similarity between the obtained at least one entity feature vector and the visual feature vector is determined respectively by the contrastive learning model.

[0114] In order to ensure that the entities obtained by the entity linking method proposed in the embodiment of the present application are more accurate and rich, the embodiment of the present application also involves a training method of a comparative learning model. The training method of the comparative learning model can be executed by a computing device, which can be a server 220 or a terminal device 210, that is, the method can be executed by the server 220 or the terminal device 210 alone, or can be executed by the server 220 and the terminal device 210 together.

[0115] It should be noted that Figure 2 The figure is only an example. In fact, the number of terminal devices 210 and servers 220 is not limited, and no specific limitation is made in the embodiment of the present application. In the embodiment of the present application, when there are multiple servers 220, multiple servers 220 can form a blockchain, and the server 220 is a node on the blockchain.

[0116] To further illustrate the technical solution provided by the embodiment of the present application, the following is an example of a server executing alone, and the entity linking method provided by the exemplary embodiment of the present application, and the training method of the model used in the entity linking process are described in conjunction with the accompanying drawings. It should be noted that the above application scenario is only shown for the sake of facilitating the understanding of the spirit and principle of the present application, and the implementation of the present application is not subject to any restrictions in this regard. In addition, although the operation of the method of the present application is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0117] The entity linking method proposed in the embodiment of the present application is implemented based on the contrastive learning model. In order to ensure the accuracy of the entity linking method, the training method of the contrastive learning model is proposed. At the same time, in order to verify that the entity linking method based on the contrastive learning model has higher accuracy and recall rate than other methods in the relevant technology, a contrastive evaluation is proposed for the trained contrastive learning model in the embodiment of the present application.

[0118] Below, the overall implementation method of the embodiment of the present application is explained from the perspectives of the contrastive learning model training process, the contrastive learning model contrast evaluation process, and the entity linking process based on the contrastive learning model.

[0119] Example 1: Comparative learning model training process

[0120] See also Figure 3 , Figure 3 A structural diagram of a contrastive learning model provided in an embodiment of the present application, the contrastive learning model includes: a visual encoder, an entity encoder and a contrast matcher; wherein:

[0121] A visual encoder is used to encode the visual content to be linked to obtain a visual feature vector of the visual content to be linked; the visual encoder includes but is not limited to: a visual transformer and a Swin transformer;

[0122] An entity encoder is used to encode the multimedia information corresponding to at least one candidate entity in the knowledge graph to obtain an entity feature vector corresponding to each of the at least one candidate entity;

[0123] A contrast matcher is used to compare and match the visual feature vector output by the visual encoder with the entity feature vectors corresponding to at least one candidate entity output by the entity encoder, and determine the target similarity between at least one entity feature vector and the visual feature vector.

[0124] In the embodiment of the present application, considering that an entity in the knowledge graph is used to represent a multimedia information, the multimedia information includes information such as video, graphics, and text, which contains both text and images; therefore, in order to ensure the accuracy of the entity feature vector output by the entity encoder for each entity, an image encoder, a text encoder, and a cross-modal encoder are provided in the entity encoder; see Figure 4 , Figure 4 A schematic diagram of the structure of another contrastive learning model provided in an embodiment of the present application; wherein:

[0125] An image encoder is used to encode the image of the candidate entity in the knowledge graph to obtain the image feature vector of the entity; it should be noted that in the embodiment of the present application, in order to avoid introducing additional parameter storage costs, the image encoder in the entity encoder adopts an architecture that is completely consistent with the visual encoder and shares model parameters;

[0126] The text encoder is used to encode the text information of the candidate entities in the knowledge graph to obtain a text feature vector. The structure of the text encoder is consistent with the text pre-training model BERT, and a [CLS] tag is added to the beginning of the text information input to summarize the entire sentence. Specifically, for each entity, the entity name and text description are combined through a pre-defined sentence template to obtain the corresponding text information. For example, the text information is: an image of [entity name] can be described as [text description];

[0127] Cross-modal encoder: By introducing an additional cross-attention layer, the image feature vector of the entity is injected into the text feature vector of the entity to obtain a multimodal feature vector.

[0128] In order to ensure the accuracy of the contrastive learning model, the embodiment of the present application also provides a contrastive learning model training method for the above contrastive learning model; specifically, based on the training sample set, the contrastive learning model to be trained is subjected to cyclic iterative training to obtain a trained contrastive learning model; since the operations performed during each cyclic iterative training process are consistent, the operations performed during one cyclic iterative training process are taken as an example to illustrate the training process of the contrastive learning model.

[0129] See also Figure 5 , Figure 5 A flow chart of a training method for a contrastive learning model provided in an embodiment of the present application includes the following steps:

[0130] Step S500: obtaining a training sample set, wherein each training sample in the training sample set includes: sample visual content, and an entity label set corresponding to the sample visual content.

[0131] In an embodiment of the present application, the entity label set corresponding to the sample visual content includes at least one positive entity label and one negative entity label; wherein the positive entity label is an entity in the knowledge graph, whose correlation with the visual sample content reaches a first correlation threshold; the negative entity label is an entity in the knowledge graph, whose correlation with the visual sample content reaches a second correlation threshold, and the first correlation threshold is greater than the second correlation threshold.

[0132] For example, see Figure 6 , Figure 6 A schematic diagram of constructing an entity label set is provided for an embodiment of the present application; specifically, the text description of the sample visual content is matched with the text description of the entity in the knowledge graph, the correlation with each entity in the sample vision is determined, and based on the correlation, entities whose correlation reaches a first correlation threshold are selected from the knowledge graph as positive label entities; for example, the first correlation threshold is 99%, then among the entities in the knowledge graph, entities whose correlation with the sample visual content reaches 99% are screened out as positive label entities; similarly, entities whose correlation reaches a second correlation threshold and is less than the first correlation threshold are selected from the knowledge graph as negative label entities; for example, the second correlation threshold is 97%, then in the knowledge graph, entities whose correlation with the sample vision reaches 97% and is less than 99% are screened out as negative label entities.

[0133] It should be noted that Figure 6 This is only an exemplary description. In actual application, there may be more than one positive entity label in the entity label set.

[0134] In the implementation of this application, considering that the knowledge graph contains a large number of entities, it will take a lot of time to determine the relevance of each entity in the knowledge graph with the sample visual content and sort the relevance. Therefore, in order to alleviate the time-consuming problem, some entities can be randomly screened out from the knowledge graph, and the relevance between the screened entities and the sample visual content can be determined.

[0135] In another possible implementation, the negative entity tags can also be screened from the knowledge graph based on the screened positive entity tags. Exemplarily, for each positive entity tag, the bm25 algorithm is used to retrieve the most matching entity other than the positive entity tag itself in the knowledge graph as the negative entity tag.

[0136] By adopting the method of constructing a training sample set provided in an embodiment of the present application, different numbers of positive entity labels are set for the sample visual content of different training samples, that is, the number of positive entity labels changes with the change of the sample visual content, so that during the training process, the data of each training batch is adaptively recombined through the loss function, and a specific target is constructed for each sample visual content, thereby solving the problem of erroneous division of positive and negative sample pairs caused by inconsistent positive entity labels; and improving the difficulty of negative samples in the contrastive learning process, further ensuring the accuracy of the trained contrastive learning model.

[0137] Step S501: Select training samples from a training sample set.

[0138] In an embodiment of the present application, training samples are randomly selected from a training sample set, and the training samples include: sample visual content, and an entity label set corresponding to the sample visual content; the entity label set includes at least one positive entity label and one negative entity label.

[0139] Step S502: input the training sample into the contrastive learning model to obtain the sample visual feature vector of the sample visual content, and obtain the label entity feature vector corresponding to each entity label in the entity label set.

[0140] In a possible implementation, an image of the sample visual content is input into a visual encoder of a contrastive learning model, and the image of the sample visual content is encoded by the visual encoder to obtain a sample visual feature vector.

[0141] In the process of encoding the image of the sample visual content, considering that the visual mention may not be able to unambiguously correspond to a certain area in the image of the sample visual content, in an embodiment of the present application, the image of the sample visual content is directly divided into image blocks, the image is divided into multiple uniform image blocks, and a visual encoder is used to directly encode the uniformly divided image blocks.

[0142] In a possible implementation, the entity label is input into an entity encoder of a contrastive learning model, and the entity label is encoded by the entity encoder to obtain a corresponding label entity feature vector.

[0143] When encoding the entity tag, if the multimedia information associated with the entity tag only includes an image, the image is encoded only by the image encoder in the entity encoder, and the obtained image feature vector is used as the tag entity feature vector;

[0144] If the multimedia information associated with the entity tag contains only text, the text is encoded only by the text encoder in the entity encoder, and the obtained text feature vector is used as the tag entity feature vector;

[0145] If the multimedia information associated with the entity tag contains both images and text, the image is first encoded by the image encoder in the entity encoder to obtain an image feature vector, and the text is encoded by the text encoder in the entity encoder to obtain a text feature vector. Then, the image feature vector and the text feature vector are fused by the cross-modal encoder in the entity encoder to obtain a multimodal feature vector, and the multimodal feature vector is used as the label entity feature vector.

[0146] In this application, unimodal feature vectors and multimodal feature vectors are unified, and multimodal feature vectors are also used as comparison items to construct sample visual feature vector-entity multimodal feature pairs, embedding unimodality and multimodality into a unified space to promote entity alignment under different modal representations and alleviate the problem of inconsistent entity representation caused by missing descriptive text or associated images.

[0147] Step S503, constructing a target loss function based on the predicted similarity between each label entity feature vector and the matched sample visual feature vector, and using the target loss function to adjust parameters.

[0148] In an embodiment of the present application, after obtaining the sample visual feature vector of the sample visual content in the training sample and the label entity feature vector of each entity label in the entity label set, the predicted similarity between the sample visual feature vector and each matched label entity feature vector is determined.

[0149] Since the implementation method of determining the predicted similarity between each label entity feature vector and the matched sample visual feature vector is consistent, the following takes determining the predicted similarity between a label entity feature vector and the matched sample visual feature vector as an example to illustrate the determination of the predicted similarity.

[0150] First, normalizing the label entity feature vector, normalizing the label entity feature vector to the hypersphere space, and obtaining the normalized label entity feature vector; and normalizing the matched sample visual feature vector, normalizing the matched sample visual feature vector to the hypersphere space, and obtaining the normalized sample visual feature vector;

[0151] Afterwards, the predicted similarity between the label entity feature vector and the sample visual feature vector is determined by calculating the cosine similarity of the two vectors.

[0152] For example, for the image i associated with the sample visual content (m) , the visual encoder f(u) first encodes the image i (m) Perform encoding processing to obtain a sample visual feature vector u ′ =fu (i (m) )∈i l×d . For image i (m) Each match is described by the text t (e) And there may be associated images i (e) The entity label represented by is encoded using the entity encoder f(v) to obtain the label entity feature vector v ′ =f v (t (e) ,i (e) )∈i l×d Next, the sample visual feature vector of the sample visual content and the label entity feature vector of the matched entity label are normalized onto the hypersphere: And by calculating the cosine similarity s=uv of the two vectors T Calculate their similarity.

[0153] After obtaining the sample visual feature vector and the label entity feature vector, the two are uniformly planned into the hypersphere space to unify the two spaces, and further similarity calculation is performed to ensure the accuracy of the similarity calculation.

[0154] In a possible implementation, after determining the predicted similarities between the sample visual feature vector and the matched feature vectors of each label entity, a target loss function is constructed based on each predicted similarity.

[0155] Optionally, a target loss function is constructed based on the number of positive entity tags in the entity tag set and the first similarity ratio associated with each positive entity tag; wherein the first similarity ratio is: the ratio of the predicted similarity associated with the positive entity tag to the first comprehensive similarity associated with the entity tag set; the first comprehensive similarity is: the sum of the similarities associated with all entity tags in all entity tag sets.

[0156] Exemplarily, the target loss function constructed based on each prediction similarity is:

[0157] Formula 1:

[0158] Among them, B represents the sample visual content set in the current training process, i (m) represents a sample visual content in the sample visual content set, E(i) represents the positive entity label set of the sample visual content, |E(i)| represents the number of positive entity labels, k represents a positive entity label in the positive entity label set, A represents the union of the entity label sets matching all sample visual contents, j represents an entity label in the entity label set, τ is a hyperparameter used to control the intensity of the penalty for difficult negative samples, represents the predicted similarity of a positive entity label that matches the sample visual content, Represents the first comprehensive similarity.

[0159] In a possible implementation, the target loss function constructed based on each prediction similarity can also be a noise contrast estimation (NCE) loss, denoted by L MIL-NCE ; When using L MIL-NCE When the training sample set {z i The i-th training sample z in i , set P i is the training sample z i The indices of all positive entity labels in N i is the training sample z i All negative entity label indices in , record:

[0160] is a positive sample pair;

[0161] is a negative sample pair;

[0162] in, Represents the training sample z i Sample visual content in z j Represents the training sample z i Entity tags in ;

[0163] And set the similarity between the sample visual content and the entity label to: or

[0164] Therefore, the target loss function L constructed based on the similarity of each prediction is MIL -NCE is:

[0165] Formula 2:

[0166] Among them, B represents the sample visual content set in the current training process, i (m) represents a sample visual content in the sample visual content set, represents the predicted similarity between the sample visual content and the matching positive entity label, Represents the first comprehensive similarity. MIL-NCE loss is used to maximize the similarity scores and Minimize the similarity scores of all negative sample pairs and

[0167] In a possible implementation, the target loss function constructed based on each prediction similarity can also be a supervised contrastive learning (SCL) loss, denoted by L SupCon ;

[0168] When using L SupCon When the training sample set {z i The i-th training sample z in i , set P i is the training sample z i The indices of all positive entity labels in N i is the training sample z i All negative entity label indices in , record:

[0169] is a positive sample pair;

[0170] is a negative sample pair;

[0171] in, Represents the training sample z i Sample visual content in z j Represents the training sample z i Entity tags in ;

[0172] And set the similarity between the sample visual content and the entity label to: or

[0173] Therefore, the target loss function L constructed based on the similarity of each prediction is SupCon for:

[0174]

[0175] Where E represents the expectation; and the predicted similarity between each sample visual content and the matching positive entity label The predicted similarity associated with all entity tags By using labeled data, L SupCon Normalized embeddings from the same class are encouraged to be pulled closer together, while embeddings from different classes are pulled further apart.

[0176] In a possible implementation, the target loss function constructed based on each prediction similarity can also be a multiple positive sample noise contrast estimation Multi-positive, MP-NCE) loss, denoted by L MP-NCE ; The target loss function L constructed based on the similarity of each prediction MP-NCE for:

[0177] Formula 4:

[0178] Where E represents the expectation; D(i, j) represents the domain combination of the sample visual content and the matching entity label, ω D(i，j) is a domain-specific hyperparameter that enables intra-domain supervision (image-image pairs) and inter-domain supervision (image-text pairs, image-multimodal pairs) to contribute equally to the final loss. For example, when a training sample contains N input image-entity image pairs, N input image-entity multimodal pairs, and M input image-entity text pairs, then When they are image-image pairs, image-text pairs, and image-multimodal pairs, ω D(i，j) Set to 1, 1,

[0179] In the embodiment of the present application, in order to increase the number of negative entity tags, a fixed-length entity tag queue is also set in the entity tag set, and the entity tags in the entity tag queue are set as negative sample tags. Figure 7 , Figure 7 A schematic diagram of another method for constructing an entity tag set is provided in an embodiment of the present application; and, in order to ensure the accuracy of model training, the entity tag queue is continuously updated according to the training samples in the model training process.

[0180] Exemplarily, after the current loop iteration round ends, or after it is determined that a trained comparative learning model is obtained based on the current training sample, the positive entity tags currently in use are stored in the entity tag queue; and based on the number of positive entity tags in use, entity tags are selected and deleted from the entity tag queue in the order of the time when the entity tags were stored.

[0181] For example, when training a contrastive learning model based on a current training sample, after the current loop iteration number ends or a trained contrastive learning model is obtained for the current training sample, the positive entity label in the training sample is added to the entity label queue;

[0182] If it is determined after adding that the entity tag queue is not full, no tags in the entity tag queue will be dequeued; if it is determined after adding that the entity tag queue is full, no tags in the entity tag queue will be dequeued; if it is determined after adding that the entity tag queue is full and there is no extra space to store the remaining positive entity tags, the earliest enqueued entity tag will be deleted from the entity tag queue in the order of the time when the entity tags were stored.

[0183] When an entity tag queue is set in the entity tag set, considering that the entity tags in the entity tag queue may be positive entity tags of the sample visual content, but are used as negative entity tags during use, resulting in inaccurate training, therefore, soft tags are proposed in an embodiment of the present application, and the similarity between the entity tags and the sample visual content is represented by the soft tags.

[0184] And when constructing a target loss function based on the predicted similarity between each label entity feature vector and the sample visual feature vector, the target loss function is constructed based on the used training samples, the target soft labels associated with each entity label in the training samples, and the second similarity ratio associated with each entity label; wherein the target soft label represents: the degree of similarity between the entity label and the matching sample visual content; the second similarity ratio is: the ratio between the predicted similarity associated with the entity label and the second comprehensive similarity associated with the entity label set to which the entity label belongs; the second comprehensive similarity is: the sum of the similarities associated with all entity labels in all entity label sets.

[0185] Exemplarily, the objective loss function is:

[0186] Formula 5:

[0187] Among them, B represents the sample visual content set in the current training process, i (m) represents a sample visual content in the sample visual content set, l represents an entity label in the entity label set, and y i(m),k represents the target soft label between the sample visual content and the matching entity label, A represents the union of all entity label sets matching the sample visual content, j represents an entity label in the entity label set, τ is a hyperparameter used to control the intensity of the penalty for difficult negative samples, represents the predicted similarity of a positive entity label that matches the sample visual content, Represents the second comprehensive similarity.

[0188] In a possible implementation, the target soft label is determined by:

[0189] For each entity tag in the entity tag set, perform the following operations:

[0190] determining a predicted similarity between the entity label and the sample visual content based on a label entity feature vector of the entity label and a sample visual feature vector of the sample visual content associated with the entity label;

[0191] Perform normalized mapping processing on the predicted similarity to obtain the corresponding predicted soft label;

[0192] The predicted soft label and the original hard label are superimposed in a preset ratio to obtain a target soft label; wherein the original hard label is determined based on whether the entity label is a positive entity label.

[0193] In this application, soft labels are proposed and a method for calculating soft labels is given to ensure that the training process is more accurate.

[0194] Taking the multimedia information associated with entity tags, which contains both images and text, as an example, the training process of the contrastive learning model is generally described. Figure 8 , Figure 8 A schematic diagram of a comparative learning model training provided in an embodiment of the present application, from Figure 8 It can be seen that:

[0195] Dividing the image of the sample visual content into multiple uniform image blocks, and directly encoding the uniformly divided image blocks using a visual encoder to obtain a sample visual feature vector of the sample visual content;

[0196] The entity name and text description of the entity tag are combined through a pre-defined sentence template to obtain the corresponding text information. A [CLS] tag is added to the beginning of the text information input to summarize the entire sentence. The entire sentence is then input into the text encoder of the entity encoder for encoding to obtain a text feature vector.

[0197] Input the image of the entity label into the image encoder of the entity encoder for encoding to obtain the image feature vector;

[0198] The image feature vector and the text feature vector are input into the cross-modal encoder of the text entity encoder, and the image feature vector of the entity label is injected into the text feature vector of the entity label to obtain the multimodal feature vector of the entity label;

[0199] The sample visual feature vector of the sample visual content and the multimodal feature vector of the entity label are input into the contrast matcher to obtain the predicted similarity between the sample visual content and the entity label.

[0200] The above method can be used to determine the predicted similarity between the sample visual content and each entity label before, and then a loss function is constructed based on multiple predicted similarities, and the parameters of the contrastive learning model are adjusted based on the loss function to obtain a trained contrastive learning model.

[0201] In the present application, a contrastive learning model is trained for sample visual content containing different numbers of positive entity labels and different numbers of negative entity labels, as well as candidate entities containing multiple modal information, so as to effectively perform entity linking tasks based on the trained contrastive learning model.

[0202] Example 2: Comparative evaluation process

[0203] In order to verify that the entity linking method based on the trained comparative learning model proposed in the embodiment of the present application has higher accuracy and recall rate than other entity linking methods in the prior art, it is proposed to compare the implementation method and the output results.

[0204] See also Fig. 9 , Fig. 9 A schematic diagram for comparing entity linking methods provided in an embodiment of the present application; wherein: Fig. 9 The entity linking methods involved include: performing entity linking based on the trained contrastive learning model proposed in the embodiments of the present application, performing entity linking based on the supervised contrast learning (Supervised Contrast) model in the related technology, and performing entity linking based on the contrastive language-image model in the related technology; the contrastive language-image model includes but is not limited to: contrastive language-image pre-training (Contrastive Language-Image Pre-Training, CLIP) model, large-scale image and noisy text embedding (ALarge-scale Image and Noisy-text embedding, ALIGN) model.

[0205] Exemplarily, the supervised contrastive learning model in the related art introduces additional positive sample pairs for training through the image labels, which is mainly used for entity linking between images; the contrastive language-image model in the related art is trained based on a one-to-one mapping relationship between image and text, which is mainly used for entity linking between image and text; and the contrastive learning model proposed in the implementation of this application is trained based on image features, text features and fused multimodal features of multiple positive labels, which is used for image-image entity linking, image-text entity linking and image-multimodal entity linking.

[0206] Further, the embodiment of the present application proposes to use specific data for effect evaluation. Exemplarily, experiments are conducted using the SportsDiverse visual multi-entity linking dataset and the WikiSports multimodal sports knowledge graph. Among them, SportsDiverse is a dataset for visual multi-entity linking tasks, and is also the largest visual entity linking dataset considering conceptual entities; it contains about 82,000 input images, covering sports figures, place names, organization names, and all other named entities identified by names, as well as conceptual entities such as sports events and terms. Conceptual entities refer to abstract entities that require official definitions in specific fields; therefore, SportsDiverse can provide a unified evaluation benchmark for visual multi-entity links in the sports field. WikiSports is the largest public multimodal knowledge graph in the sports field, containing about 1.96 million entities, 224,000 entity images, and 15.25 million relationship and attribute triples, which can provide rich external multimodal knowledge for search and recommendation in the sports field.

[0207] Specifically, UniCL, ALBEF, BLIP, X-VLM and the implementation method of this application are experimentally compared. The experiment uses the Top-K recall rate (Recall@K) and R accuracy (R-Precision) under different K values ​​for evaluation. Fig.10 , Fig.10 A schematic diagram of a named entity comparison result provided in an embodiment of the present application, and see Fig.11 , Fig.11 A schematic diagram of a concept entity comparison result provided in an embodiment of the present application; wherein, the Top-K accuracy rate refers to the ratio of the number of correctly linked entities in the first K results to the number of all associated entities, and the R accuracy rate refers to the ratio of the number of correctly linked entities in the first |Rel| results to the number of all associated entities, and |Rel| represents the number of all associated entities. The larger the Top-K accuracy rate and the R accuracy rate, the better the result of entity linking.

[0208] Therefore, from Fig.10 and Fig.11 It can be seen that the technical solution proposed in the embodiment of the present application has generally achieved better entity linking results in multiple tasks, that is, it has higher accuracy and recall rate.

[0209] Example 3: Entity linking process based on trained contrastive learning model

[0210] See also Fig.12 , Fig.12 A flow chart of an entity linking method provided in an embodiment of the present application includes the following steps:

[0211] Step S1200, obtaining a visual feature vector of the visual content to be linked, and an entity feature vector corresponding to at least one candidate entity in the knowledge graph; wherein the entity feature vector includes: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, and the attribute dimension includes at least one of an image and a text.

[0212] In one possible implementation, the visual content to be linked is encoded by a visual encoder in a trained contrastive learning model to obtain a visual feature vector of the visual content to be linked; and the multimedia information corresponding to at least one candidate entity is encoded by an entity encoder in a trained contrastive learning model to obtain an entity feature vector corresponding to at least one candidate entity.

[0213] In a possible implementation, the entity encoder includes: an image encoder, a text encoder, and a cross-modal encoder; when the multimedia information includes image information and text information, the multimedia information is encoded by the entity encoder in the trained contrastive learning model to obtain an entity feature vector, including:

[0214] The image information is encoded by an image encoder to obtain an image feature vector, and the text information is encoded by a text encoder to obtain a text feature vector. The image feature vector and the text feature vector are fused by a cross-modal encoder to obtain a multimodal feature vector, and the multimodal feature vector is used as an entity feature vector.

[0215] Step S1201 : determining target similarities between at least one entity feature vector and a visual feature vector respectively.

[0216] In a possible implementation, respectively determining the target similarity between the obtained at least one entity feature vector and the visual feature vector includes:

[0217] Normalizing at least one entity feature vector respectively to obtain at least one entity feature vector, each of which is normalized to a target entity feature vector in a hyperspherical space; normalizing the visual feature vector to obtain a visual feature vector normalized to a hyperspherical space; and determining the similarity between at least one target entity feature vector and a target visual feature vector respectively.

[0218] Step S1202: Filter out a target entity whose target similarity satisfies a preset similarity condition from at least one candidate entity, and use the target entity as an entity linking result of the visual content to be linked.

[0219] In a possible implementation, entities whose target similarity reaches a similarity threshold are screened out as target entities.

[0220] In another possible implementation, at least one candidate entity is sorted according to the target similarity in descending order, and an entity whose ranking reaches a rank threshold is obtained as the target entity.

[0221] See also Fig.13 , Fig.13 A schematic diagram of a specific implementation method of entity linking provided in an embodiment of the present application, from Fig.13 It can be seen that:

[0222] A candidate entity is selected from the knowledge graph, and the image and text contained in the multimedia information associated with the candidate entity are input into the entity encoder of the trained contrastive learning model for encoding processing to obtain an entity feature vector; when the multimedia information only contains images, the entity feature vector is an image feature vector; when the multimedia feature vector only contains text, the entity feature vector is a text feature vector; when the multimedia feature vector contains both images and text, the entity feature vector is a multimodal feature vector containing an image feature vector and a text feature vector;

[0223] The image of the visual content to be linked is divided into multiple uniform image blocks, and the uniformly divided image blocks are directly encoded using a visual encoder of a trained contrastive learning model to obtain a visual feature vector of the visual content to be linked;

[0224] Inputting the visual feature vector of the visual content to be linked and the entity feature vector of the candidate entity into the contrast matcher of the trained contrastive learning model to determine the target similarity between the entity feature vector and the visual feature vector;

[0225] By adopting the above method, the target similarity between the visual content to be linked and each candidate entity can be determined, and then based on the target similarity, the target entity whose target similarity meets the preset similarity condition is screened out from at least one candidate entity, and the target entity is used as the entity linking result of the visual content to be linked.

[0226] In one possible implementation, among at least one candidate entity, a target entity whose target similarity meets a preset similarity condition is screened out, and after the target entity is used as the entity linking result of the visual content to be linked, the multimedia information associated with the target entity is recommended to the target object browsing the visual content to be linked.

[0227] See also Fig.14 , Fig.14 A schematic diagram of an entity link application scenario provided in an embodiment of the present application, from Fig.14 It can be seen that: in Fig.13After obtaining the entity linking result, the entity linking result is recommended to the terminal device associated with the object of the visual content to be linked.

[0228] In the present application, firstly, a visual feature vector of the visual content to be linked and an entity feature vector corresponding to each of at least one candidate entity in the knowledge graph are obtained, the entity feature vector comprising: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, the attribute dimension comprising at least one of an image and a text, that is, the entity feature vector comprises at least one of an image feature vector and a text feature vector; then, the target similarity between the obtained at least one entity feature vector and the visual feature vector is determined respectively, and among the at least one candidate entity, a target entity whose target similarity meets a preset similarity condition is screened out, and the target entity is used as the entity linking result of the visual content to be linked; obviously, the embodiment of the present application proposes a method for implementing entity linking based on visual content.

[0229] In the process of entity linking based on visual content, the target similarity between the visual feature vector of the visual content and the text feature vector of the entity, the target similarity between the visual feature vector and the image feature vector of the entity can be determined, and the target similarity between the visual feature vector and the multimodal feature vector of the entity determined based on the text feature vector and the image feature vector can also be determined, and the target entity can be determined based on the target similarity. At this time, the visual feature vector based on the visual content can be linked to the text content, image content, and text + image content, so that entity linking is no longer limited to the text field, which expands the application field of entity linking technology and ensures that the information obtained based on entity linking technology is richer and more accurate.

[0230] Based on the same inventive concept, the embodiment of the present application also provides a physical linking device; Fig.15 As shown, the physical linking device 1500 includes:

[0231] The acquisition unit 1501 is used to acquire a visual feature vector of the visual content to be linked, and an entity feature vector corresponding to each of at least one candidate entity in the knowledge graph; wherein the entity feature vector includes: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, and the attribute dimension includes at least one of an image and a text;

[0232] A determination unit 1502, configured to respectively determine a target similarity between the obtained at least one entity feature vector and the visual feature vector;

[0233] The screening unit 1503 is used to screen out a target entity whose target similarity meets a preset similarity condition from at least one candidate entity, and use the target entity as an entity linking result of the visual content to be linked.

[0234] In a possible implementation, the acquiring unit 1501 is specifically configured to:

[0235] Using the trained contrastive learning model, perform the following operations:

[0236] encoding the visual content to be linked by using a visual encoder in a trained contrastive learning model to obtain a visual feature vector of the visual content to be linked; and

[0237] The multimedia information corresponding to at least one candidate entity is encoded respectively through the entity encoder in the trained contrastive learning model to obtain the entity feature vector corresponding to each of the at least one candidate entity.

[0238] In a possible implementation, the multimedia information includes image information and text information, and the entity encoder includes: an image encoder, a text encoder, and a cross-modal encoder; then the acquisition unit 1501 is specifically used to:

[0239] Encoding the image information through an image encoder to obtain an image feature vector, and encoding the text information through a text encoder to obtain a text feature vector;

[0240] The image feature vector and the text feature vector are fused through a cross-modal encoder to obtain a multimodal feature vector, which is used as the entity feature vector.

[0241] In one possible implementation, the contrastive learning model is obtained as follows:

[0242] Based on the training sample set, a cyclic iterative training is performed on the contrastive learning model to be trained to obtain a trained contrastive learning model, wherein the following steps are performed during one cyclic iterative process:

[0243] Selecting training samples from a training sample set; wherein the training samples include: sample visual content, and an entity label set corresponding to the sample visual content;

[0244] Input the training sample into the contrastive learning model to obtain the sample visual feature vector of the sample visual content, and obtain the label entity feature vector corresponding to each entity label in the entity label set;

[0245] Based on the predicted similarity between the feature vector of each label entity and the sample visual feature vector, a target loss function is constructed, and the target loss function is used to adjust the parameters.

[0246] In a possible implementation, the entity tag set includes at least: a positive entity tag and a negative entity tag;

[0247] A positive entity label is an entity in the knowledge graph whose relevance to the sample visual content reaches the first relevance threshold;

[0248] Negative entity labels are entities in the knowledge graph, except for positive entity labels, whose relevance to the sample visual content reaches the second relevance threshold;

[0249] The first correlation threshold is greater than the second correlation threshold.

[0250] In one possible implementation, a target loss function is constructed based on the predicted similarity between each entity label feature vector and the sample visual feature vector, including:

[0251] Constructing a target loss function based on the number of positive entity tags in the entity tag set and the first similarity ratios associated with each positive entity tag;

[0252] The first similarity ratio is: the ratio of the predicted similarity associated with the positive entity tag to the first comprehensive similarity associated with the entity tag set; the first comprehensive similarity is: the sum of the similarities associated with all entity tags in all entity tag sets.

[0253] In a possible implementation, the entity tag set further includes: an entity tag queue of a fixed length;

[0254] The entity tag queue is updated in the following way:

[0255] After the current loop iteration round ends, the positive entity tag used in the current loop iteration round is stored in the entity tag queue; and

[0256] Based on the number of used positive entity tags, entity tags are selected and deleted from the entity tag queue in the order of the time when the entity tags were stored.

[0257] In a possible implementation, based on the predicted similarity between each label entity feature vector and the sample visual feature vector, a target loss function is constructed, including:

[0258] Constructing a target loss function based on the target soft labels associated with each entity label and the second similarity ratios associated with each entity label;

[0259] Among them, the target soft label represents: the similarity between the entity label and the sample visual content; the second similarity ratio is: the ratio between the predicted similarity associated with the entity label and the second comprehensive similarity associated with the entity label set; the second comprehensive similarity is: the sum of the similarities associated with all entity labels in all entity label sets.

[0260] In a possible implementation, the target soft label is determined by:

[0261] For each entity tag in the entity tag set, perform the following operations:

[0262] determining a predicted similarity between the entity label and the sample visual content based on a label entity feature vector of the entity label and a sample visual feature vector of the sample visual content associated with the entity label;

[0263] Perform normalized mapping processing on the predicted similarity to obtain the corresponding predicted soft label;

[0264] The predicted soft label and the original hard label are superimposed in a preset ratio to obtain a target soft label; wherein the original hard label is determined based on whether the entity label is a positive entity label.

[0265] In a possible implementation manner, the determining unit 1502 is specifically configured to:

[0266] Normalizing at least one entity feature vector respectively to obtain at least one entity feature vector, each of which is normalized to a target entity feature vector in a hyperspherical space;

[0267] Normalizing the visual feature vector to obtain a visual feature vector normalized to a hyperspherical space;

[0268] The similarity between at least one target entity feature vector and the target visual feature vector is determined respectively.

[0269] In a possible implementation, the screening unit 1503 is further configured to:

[0270] In at least one candidate entity, a target entity whose target similarity meets a preset similarity condition is screened out, and after the target entity is used as an entity linking result of the visual content to be linked, multimedia information associated with the target entity is recommended to a target object browsing the visual content to be linked.

[0271] It should be noted that although several units (or modules) of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units (or modules) described above can be concretized in one unit (or module). Conversely, the features and functions of one unit (or module) described above can be further divided into multiple units (or modules) for concretization. Of course, when implementing the present application, the functions of each unit (or module) can also be implemented in the same one or more software or hardware.

[0272] After introducing the entity linking method and apparatus according to an exemplary embodiment of the present application, another exemplary embodiment of the present application, a computing device, is introduced next.

[0273] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0274] In a possible implementation, the computing device provided in the embodiment of the present application may include at least a processor and a memory. The memory stores a program code, and when the program code is executed by the processor, the processor executes any step of the entity linking method of various exemplary embodiments in the present application.

[0275] In this embodiment, the structure of the computing device can be as follows Fig.16 As shown, it includes a memory 1601 , a communication module 1603 and one or more processors 1602 .

[0276] The memory 1601 is used to store computer programs executed by the processor 1602. The memory 1601 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0277] The memory 1601 may be a volatile memory, such as a random-access memory (RAM); the memory 1601 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1601 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1601 may be a combination of the above memories.

[0278] The processor 1602 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1602 is used to implement the above entity linking method when calling the computer program stored in the memory 1601 .

[0279] The communication module 1603 is used to communicate with terminal devices and other servers.

[0280] The specific connection medium between the memory 1601, the communication module 1603 and the processor 1602 is not limited in the embodiment of the present application. Fig.16 In the embodiment, the memory 1601 and the processor 1602 are connected via a bus 1604. The bus 1604 is connected to the processor 1602 via a bus 1604. Fig.16 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1604 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.16 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0281] The memory 1601 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the entity linking method of the embodiment of the present application. The processor 1602 is used to execute the above-mentioned entity linking method.

[0282] In another possible implementation, the computing device may be a terminal device, and the structure of the computing device may be as follows: Fig.17 As shown, it includes: a communication component 1710, a memory 1720, a display unit 1730, a camera 1740, a sensor 1750, an audio circuit 1760, a Bluetooth module 1770, a processor 1780 and other components.

[0283] The communication component 1710 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The computing device can help users send and receive information through the WiFi module.

[0284] The memory 1720 can be used to store software programs and data. The processor 1780 executes various functions and data processing of the terminal device by running the software programs or data stored in the memory 1720. The memory 1720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one type of disk storage device, flash memory device, or other volatile solid-state storage device. The memory 1720 stores an operating system that enables the terminal device to run. In the present application, the memory 1720 can store an operating system and various application programs, and can also store code for executing the multimedia information recommendation method of the embodiment of the present application.

[0285] The display unit 1730 can also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device. Specifically, the display unit 1730 may include a display screen 1732 disposed on the front of the terminal device. The display screen 1732 may be configured in the form of a liquid crystal display, a light emitting diode, etc.

[0286] The display unit 1730 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device. Specifically, the display unit 1730 may include a touch screen 1731 arranged on the front of the terminal device, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0287] The touch screen 1731 can be covered on the display screen 1732, or the touch screen 1731 and the display screen 1732 can be integrated to realize the input and output functions of the terminal device, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 1730 can display applications, multimedia information played by applications, and corresponding operation steps.

[0288] Camera 1740 can be used to capture static images. Camera 1740 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 1780 to convert it into a digital image signal.

[0289] The terminal device may further include at least one sensor 1750, such as an acceleration sensor 1751, a distance sensor 1752, a fingerprint sensor 1753, and a temperature sensor 1754. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0290] The audio circuit 1760, the speaker 1761, and the microphone 1762 can provide an audio interface between the user and the terminal device. The audio circuit 1760 can transmit the electrical signal converted from the received audio data to the speaker 1761, which is converted into a sound signal for output. The terminal device can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1760 and converted into audio data, and then the audio data is output to the communication component 1710 to be sent to, for example, another terminal device, or the audio data is output to the memory 1720 for further processing.

[0291] The Bluetooth module 1770 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable computing device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1770 to exchange data.

[0292] The processor 1780 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1720 and calling data stored in the memory 1720. In some embodiments, the processor 1780 may include one or more processing units; the processor 1780 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1780. In the present application, the processor 1780 can run the operating system, application programs, user interface display and touch response, as well as the multimedia information recommendation method of the embodiment of the present application. In addition, the processor 1780 is coupled to the display unit 1730.

[0293] In some possible implementations, various aspects of the entity linking method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computing device, the program code is used to enable the computing device to execute the steps of the entity linking method according to various exemplary implementations of the present application described above in this specification.

[0294] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0295] The program product of the cross-chain transaction processing of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited to this. In this document, the readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with a command execution system, device or device.

[0296] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0297] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0298] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user equipment, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0299] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0300] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable cross-chain transaction processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable cross-chain transaction processing device generate commands for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0301] These computer program commands may also be stored in a computer-readable memory that can direct a computer or other programmable cross-chain transaction processing device to operate in a specific manner, so that the commands stored in the computer-readable memory produce a manufactured product including a command device that implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0302] These computer program commands can also be loaded onto a computer or other programmable cross-chain transaction processing device, so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0303] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0304] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An entity linking method, characterized in that: The method comprises: Obtaining a visual feature vector of the visual content to be linked, and an entity feature vector corresponding to each of at least one candidate entity in the knowledge graph; wherein the entity feature vector comprises: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, the attribute dimension comprising at least one of an image and a text; respectively determining a target similarity between the obtained at least one entity feature vector and the visual feature vector; Among the at least one candidate entity, a target entity whose target similarity satisfies a preset similarity condition is screened out, and the target entity is used as an entity linking result of the visual content to be linked.

2. The method according to claim 1, characterized in that The method comprises: Using the trained contrastive learning model, perform the following operations: encoding the visual content to be linked by using the visual encoder in the trained contrastive learning model to obtain a visual feature vector of the visual content to be linked; and The multimedia information corresponding to the at least one candidate entity is encoded respectively by the entity encoder in the trained contrastive learning model to obtain the entity feature vector corresponding to each of the at least one candidate entity.

3. The method according to claim 2, characterized in that The multimedia information includes image information and text information, and the entity encoder includes: an image encoder, a text encoder and a cross-modal encoder; Then, the multimedia information is encoded by the entity encoder in the trained contrastive learning model to obtain the entity feature vector, including: The image information is encoded by the image encoder to obtain an image feature vector, and the text information is encoded by the text encoder to obtain a text feature vector; The image feature vector and the text feature vector are fused by the cross-modal encoder to obtain a multimodal feature vector, and the multimodal feature vector is used as the entity feature vector.

4. The method according to claim 2, characterized in that The contrastive learning model is obtained by: Based on the training sample set, a cyclic iterative training is performed on the contrastive learning model to be trained to obtain a trained contrastive learning model, wherein the following steps are performed during one cyclic iterative process: Selecting a training sample from the training sample set; wherein the training sample includes: sample visual content, and an entity label set corresponding to the sample visual content; Input the training sample into the contrastive learning model to obtain a sample visual feature vector of the sample visual content, and obtain a label entity feature vector corresponding to each entity label in the entity label set; Based on the predicted similarity between each label entity feature vector and the sample visual feature vector, a target loss function is constructed, and the target loss function is used to adjust parameters.

5. The method according to claim 4, characterized in that The entity tag set includes at least: a positive entity tag and a negative entity tag; The positive entity label is an entity in the knowledge graph whose correlation with the sample visual content reaches a first correlation threshold; The negative entity label is an entity in the knowledge graph, except for the positive entity label, whose correlation with the sample visual content reaches a second correlation threshold; Wherein, the first correlation threshold is greater than the second correlation threshold.

6. The method according to claim 4, characterized in that The target loss function is constructed based on the predicted similarity between each entity label feature vector and the sample visual feature vector, including: Constructing the target loss function based on the number of positive entity tags in the entity tag set and the first similarity ratios associated with each positive entity tag; The first similarity ratio is: the ratio of the predicted similarity associated with the positive entity tag to the first comprehensive similarity associated with the entity tag set; the first comprehensive similarity is: the sum of the similarities associated with all entity tags in all entity tag sets.

7. The method according to any one of claims 4 to 6, characterized in that: The entity tag set also includes: an entity tag queue of fixed length; Then the entity tag queue is updated in the following way: After the current loop iteration round ends, the positive entity tag used in the current loop iteration round is stored in the entity tag queue; and Based on the number of the used positive entity tags, entity tags are selected and deleted from the entity tag queue in the order of the entity tags being stored.

8. The method according to claim 7, characterized in that The target loss function is constructed based on the predicted similarity between each label entity feature vector and the sample visual feature vector, including: Constructing the target loss function based on the target soft tags associated with each entity tag and the second similarity ratios associated with each entity tag; Among them, the target soft label represents: the similarity between the entity label and the sample visual content; the second similarity ratio is: the ratio between the predicted similarity associated with the entity label and the second comprehensive similarity associated with the entity label set; the second comprehensive similarity is: the sum of the similarities associated with all entity labels in all entity label sets.

9. The method according to claim 8, characterized in that The target soft label is determined in the following manner: For each entity tag in the entity tag set, perform the following operations respectively: Determining a predicted similarity between the entity tag and the sample visual content based on a tag entity feature vector of the entity tag and a sample visual feature vector of the sample visual content associated with the entity tag; Performing normalized mapping processing on the predicted similarity to obtain corresponding predicted soft labels; The predicted soft label and the original hard label are superimposed according to a preset ratio to obtain the target soft label; wherein the original hard label is determined based on whether the entity label is a positive entity label.

10. The method according to any one of claims 1 to 3, characterized in that: The respectively determining the target similarity between the at least one entity feature vector and the visual feature vector includes: Normalizing the at least one entity feature vector respectively to obtain the at least one entity feature vector, each of which is normalized to a target entity feature vector in a hyperspherical space; Normalizing the visual feature vector to obtain a visual feature vector normalized to the hypersphere space; Determine respectively at least one target entity feature vector and the similarity between each of them and the target visual feature vector.

11. The method according to any one of claims 1 to 3, characterized in that: After selecting a target entity whose target similarity satisfies a preset similarity condition from the at least one candidate entity and using the target entity as the entity linking result of the visual content to be linked, the method further includes: The multimedia information associated with the target entity is recommended to the target object browsing the visual content to be linked.

12. A physical linking device, characterized in that: The device comprises: An acquisition unit, used to acquire a visual feature vector of the visual content to be linked, and an entity feature vector corresponding to each of at least one candidate entity in the knowledge graph; wherein the entity feature vector comprises: an attribute dimension in the multimedia information corresponding to the corresponding candidate entity, wherein the attribute dimension comprises at least one of an image and a text; a determination unit, configured to respectively determine a target similarity between the obtained at least one entity feature vector and the visual feature vector; The screening unit is used to screen out a target entity whose target similarity meets a preset similarity condition from the at least one candidate entity, and use the target entity as an entity linking result of the visual content to be linked.

13. A computing device, characterized in that: The computing device comprises: a processor and a memory, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 11 when executed by a processor.

Images