Visual entity linking method and device, computer equipment and readable storage medium

Abstract

The invention relates to a visual entity linking method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that an image is acquired, a visual object in the image is segmented, a pixel mask of the segmented visual object is generated, and the pixel mask marks the position of the visual object in the image; obtaining entity link indication information for the image, determining a to-be-matched visual object from the visual objects according to the entity link indication information, and determining a to-be-matched named entity; based on the pixel mask of the to-be-matched visual object, extracting visual object features of the to-be-matched visual object from the image; obtaining named entity characteristics of the named entity to be matched; and matching the visual object features of the to-be-matched visual object with the named entity features of the to-be-matched named entity, and linking the successfully matched to-be-matched visual object with the to-be-matched named entity. By adopting the method, the entity recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technologies, and in particular, to a visual entity linking method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technologies, visual entity linking technology has emerged. Visual Entity Linking (VEL) is a cross-task in the fields of natural language processing and computer vision, aiming to identify visual objects related to a text query in an image and link them to named entities in an entity knowledge base.

[0003] In traditional technologies, visual entity linking methods usually rely on text queries to identify entities in an image, that is, users need to provide detailed text descriptions to help the model accurately match visual objects in the image with named entities in the entity knowledge base.

[0004] However, traditional methods usually have difficulty constructing accurate text query statements when facing complex scenarios (such as entity linking for images with multiple objects or blurred backgrounds), resulting in a problem of low entity recognition accuracy. Summary of the Invention

[0005] Based on this, it is necessary to provide a visual entity linking method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve entity recognition accuracy for the above technical problems.

[0006] In a first aspect, the present application provides a visual entity linking method, including:

[0007] Obtain an image, segment visual objects in the image, generate a pixel mask for the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image;

[0008] Obtain entity linking indication information for the image, and determine a to-be-matched visual object and a to-be-matched named entity from the visual objects according to the entity linking indication information;

[0009] Extract visual object features of the to-be-matched visual object from the image based on the pixel mask of the to-be-matched visual object;

[0010] Obtain named entity features of the to-be-matched named entity;

[0011] Match the visual object features of the to-be-matched visual object with the named entity features of the to-be-matched named entity, and link the to-be-matched visual object and the to-be-matched named entity with successful matching.

[0012] In a second aspect, the present application further provides a visual entity linking device, including:

[0013] An image segmentation module, configured to obtain an image, segment visual objects in the image, generate a pixel mask of the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image;

[0014] An object selection module, configured to obtain entity linking indication information for the image, and determine a to-be-matched visual object and a to-be-matched named entity from the visual objects according to the entity linking indication information;

[0015] A feature extraction module, configured to extract visual object features of the to-be-matched visual object from the image based on the pixel mask of the to-be-matched visual object;

[0016] A feature acquisition module, configured to acquire named entity features of the to-be-matched named entity;

[0017] A feature matching module, configured to match the visual object features of the to-be-matched visual object with the named entity features of the to-be-matched named entity, and link the successfully matched to-be-matched visual object and to-be-matched named entity.

[0018] In a third aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0019] Obtain an image, segment visual objects in the image, generate a pixel mask of the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image;

[0020] Obtain entity linking indication information for the image, and determine a to-be-matched visual object and a to-be-matched named entity from the visual objects according to the entity linking indication information;

[0021] Extract visual object features of the to-be-matched visual object from the image based on the pixel mask of the to-be-matched visual object;

[0022] Acquire named entity features of the to-be-matched named entity;

[0023] Match the visual object features of the to-be-matched visual object with the named entity features of the to-be-matched named entity, and link the successfully matched to-be-matched visual object and to-be-matched named entity.

[0024] Fourthly, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0025] Obtain an image, segment visual objects in the image, generate a pixel mask for the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image;

[0026] Obtain entity link indication information for the image, and according to the entity link indication information, determine a visual object to be matched from the visual objects, and determine a named entity to be matched;

[0027] Based on the pixel mask of the visual object to be matched, extract visual object features of the visual object to be matched from the image;

[0028] Obtain named entity features of the named entity to be matched;

[0029] Match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched, and link the successfully matched visual object to be matched and the named entity to be matched.

[0030] Fifthly, the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0031] Obtain an image, segment visual objects in the image, generate a pixel mask for the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image;

[0032] Obtain entity link indication information for the image, and according to the entity link indication information, determine a visual object to be matched from the visual objects, and determine a named entity to be matched;

[0033] Based on the pixel mask of the visual object to be matched, extract visual object features of the visual object to be matched from the image;

[0034] Obtain named entity features of the named entity to be matched;

[0035] Match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched, and link the successfully matched visual object to be matched and the named entity to be matched.

[0036] The above visual entity linking method, device, computer device, computer-readable storage medium, and computer program product can obtain an image, segment visual objects in the image, and generate pixel masks for the segmented visual objects. By means of image segmentation, pixel masks that accurately mark the positions of visual objects in the image can be obtained. By obtaining entity linking indication information for the image, the entity linking indication information can be used to determine the to-be-matched visual objects and to-be-matched named entities. Furthermore, the pixel masks of the to-be-matched visual objects can be used to extract the visual object features of the to-be-matched visual objects from the image, and the named entity features of the to-be-matched named entities can be obtained. The visual object features of the to-be-matched visual objects are matched with the named entity features of the to-be-matched named entities, and the successfully matched to-be-matched visual objects and to-be-matched named entities are linked. Throughout the process, the linking of visual objects and named entities can be achieved through feature matching in combination with the image segmentation results, which can improve the accuracy of entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for the description in the embodiments of the present application or related technologies. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1 It is an application environment diagram of the visual entity linking method in an embodiment;

[0039] Figure 2 It is a flowchart of the visual entity linking method in an embodiment;

[0040] Figure 3 It is a schematic diagram of a pixel mask in an embodiment;

[0041] Figure 4 It is a schematic diagram of marking a to-be-matched visual object with marking points in an embodiment;

[0042] Figure 5 It is a schematic diagram of marking a to-be-matched visual object with a marking box in an embodiment;

[0043] Figure 6 It is a schematic diagram of specifying a to-be-matched named entity for linking in an embodiment;

[0044] Figure 7 It is a schematic diagram of expansion through associated entities in an embodiment;

[0045] Figure 8Schematic diagram of expansion through a text query statement in an embodiment;

[0046] Figure 9 Flow schematic diagram of a visual entity linking method in another embodiment;

[0047] Figure 10 Schematic diagram of determining a visual object to be matched and a named entity to be matched in an embodiment;

[0048] Figure 11 Flow schematic diagram of obtaining fused description information in an embodiment;

[0049] Figure 12 Flow schematic diagram of obtaining fused description information in another embodiment;

[0050] Figure 13 Technical principle diagram of a visual entity linking method in an embodiment;

[0051] Figure 14 Effect schematic diagram of visual entity linking in an embodiment;

[0052] Figure 15 Flow schematic diagram of visual semantic tokenization in an embodiment;

[0053] Figure 16 Structural block diagram of a visual entity linking device in an embodiment;

[0054] Figure 17 Internal structural diagram of a computer device in an embodiment. Specific implementation manners

[0055] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In order to clearly describe the technical solutions of the present application and facilitate understanding of the technical solutions of the present application, the key concepts involved in the present application will be explained below first.

[0057] 1. Visual object.

[0058] A visual object refers to an object in an image that can be recognized, located and described. For example, a visual object can specifically be a specific object, such as a car, a person, an animal, a building, food, etc. Also, for example, a visual object can specifically be a specific scene, such as a beach, a city street, a forest, etc.

[0059] 2. Pixel mask.

[0060] A pixel mask is a data structure related to an image and is typically used to specify which pixels in the image need to be processed and attended to. In this embodiment, the pixel mask is used to mark the position of a visual object in the image. For example, the pixel mask can specifically be a two-dimensional matrix of the same size as the image, where each element in the matrix corresponds to a pixel point in the image, and the value of the element is used to indicate the state or attribute of the corresponding pixel point. In this embodiment, the value of the element can be used to indicate whether the corresponding pixel belongs to the visual object, thus enabling accurate marking of the position of the visual object in the image.

[0061] 3. Entity linking indication information.

[0062] Entity linking indication information refers to information used to specify a visual object or named entity to be matched. For example, the entity linking indication information can specifically be information used to specify a visual object to be matched, that is, visual object marking information. Another example is that the entity linking indication information can specifically be information used to specify a named entity to be matched, that is, named entity specification information.

[0063] 4. Visual object to be matched.

[0064] A visual object to be matched is a visual object that needs to be matched with a named entity to be matched to determine whether it can be linked to the named entity to be matched, and can specifically be at least one visual object.

[0065] 5. Named entity to be matched.

[0066] A named entity to be matched is a named entity that needs to be matched with a visual object to be matched to determine whether it can be linked to the visual object to be matched, and can specifically be at least one named entity in an entity knowledge base. A named entity refers to a proper noun or entity of a specific concept with a specific meaning, and these entities are usually specific and unique and can refer to specific things such as people, places, and objects.

[0067] The visual entity linking method provided in the embodiments of this application can be applied to an application environment as Figure 1 shown. Among them, the terminal 102 acquires an image, segments the visual objects in the image, generates a pixel mask for the segmented visual objects, the pixel mask marks the position of the visual objects in the image, acquires entity linking indication information for the image, determines the visual objects to be matched from the visual objects according to the entity linking indication information, and determines the named entities to be matched. Based on the pixel masks of the visual objects to be matched, visual object features of the visual objects to be matched are extracted from the image, named entity features of the named entities to be matched are acquired, the visual object features of the visual objects to be matched are matched with the named entity features of the named entities to be matched, and the successfully matched visual objects to be matched and named entities to be matched are linked.

[0068] Among them, the terminal 102 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc.

[0069] In an exemplary embodiment, as Figure 2 shown, a visual entity linking method is provided. In this embodiment, this method is exemplified by being applied to a terminal. It can be understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 202 to step 210. Among them:

[0070] Step 202, obtain an image, segment the visual objects in the image, generate a pixel mask of the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image.

[0071] Among them, an image refers to a data structure that needs to identify the visual objects therein and link the visual objects with entities, and usually exists in the form of a pixel matrix. Its main role in the visual entity linking task is to provide visual information. For example, an image can specifically be a data structure containing multiple visual objects.

[0072] Among them, a visual object refers to an object that can be recognized, located, and described in an image. For example, a visual object can specifically be a specific object, such as a car, a person, an animal, a building, food, etc. Also for example, a visual object can specifically be a specific scene, such as a beach, a city street, a forest, etc.

[0073] Among them, a pixel mask is a data structure related to an image, and is usually used to specify which pixels in the image need to be processed and concerned. In this embodiment, the pixel mask is used to mark the positions of the visual objects in the image. For example, a pixel mask can specifically be a two-dimensional matrix with the same size as the image. Each element in the matrix corresponds to a pixel point in the image, and the value of the element is used to indicate the state or attribute of the corresponding pixel point. In this embodiment, the value of the element can be used to indicate whether the corresponding pixel belongs to the visual object, thus enabling accurate marking of the positions of the visual objects in the image.

[0074] Exemplarily, when visual entity linking is required, the terminal acquires an image, segments the visual objects in the image, generates a pixel mask for the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image.

[0075] In a specific application, when generating the pixel mask, the terminal can first predict the class distribution probability of each pixel point in the image. The class distribution probability includes the matching probability values of the pixel point belonging to each visual object in the image. For any pixel point, the visual object with the highest matching probability value is determined as the visual object matched by the pixel point. The local image regions where each visual object is located are respectively composed of all the matching pixel points. According to the visual objects matched by each pixel point, the local image regions where each visual object is located can be accurately segmented in the image, and then, based on the local image regions where each visual object is located, the pixel masks of each visual object can be accurately determined.

[0076] In a specific application, the pixel mask can specifically be a two-dimensional matrix with the same size as the image. Each element in the matrix corresponds to a pixel point in the image, and the value of the element can be used to indicate whether the corresponding pixel belongs to a visual object. Taking the values of the elements being 0 and 1, where 0 is used to indicate that the corresponding pixel does not belong to a visual object and 1 is used to indicate that the corresponding pixel belongs to a visual object as an example, the form of the pixel mask can be as Figure 3 shown. In the image, there are three visual objects, visual object 302, visual object 304, and visual object 306. Through segmentation, the pixel masks of the segmented visual objects can be obtained. The values of the elements in the pixel mask are used to indicate whether the corresponding pixel belongs to a visual object. Specifically, a value of 1 for the element indicates that the corresponding pixel belongs to a visual object, and a value of 0 for the element indicates that the corresponding pixel does not belong to a visual object.

[0077] In a specific application, the terminal can use a pre-trained image segmentation model to segment the visual objects in the image and generate a pixel mask. Among them, the pre-trained image segmentation model can be configured according to the actual application scenario. For example, the pre-trained image segmentation model can specifically be a SAM (Segment Anything Model) model, a SEEM (Segment Everything Every Moment) model, etc.

[0078] Step 204: Acquire entity link indication information for the image. According to the entity link indication information, determine the visual object to be matched from the visual objects, and determine the named entity to be matched.

[0079] Among them, the entity link indication information refers to the information used to specify the visual object or named entity to be matched. For example, the entity link indication information can specifically be the information used to specify the visual object to be matched, that is, the visual object marking information. Another example is that the entity link indication information can specifically be the information used to specify the named entity to be matched, that is, the named entity specification information.

[0080] Among them, the visual object to be matched refers to the visual object that needs to be matched with the named entity to be matched to determine whether it can be linked to the named entity to be matched, and can specifically be at least one visual object. The named entity to be matched refers to the named entity that needs to be matched with the visual object to be matched to determine whether it can be linked to the visual object to be matched, and can specifically be at least one named entity in the entity knowledge base. A named entity refers to a proper noun or an entity of a specific concept with a specific meaning. These entities are usually specific and unique and can refer to specific things such as people, places, and objects.

[0081] Among them, the entity knowledge base is a database or storage system that stores and organizes named entities and their relationships in a structured form. It is usually used for the construction and application of knowledge graphs and is an important part of the knowledge graph. It is used to store named entities, the relationships between named entities, and the attribute information of named entities. It organizes and integrates information in a graphical way, where nodes represent named entities such as people, places, and items; edges represent the relationships between named entities such as "belong to", "create", "be located in", etc.

[0082] Exemplarily, the terminal will obtain the entity link indication information for the image, and based on the entity link indication information, determine the visual object to be matched from the visual objects and determine the named entity to be matched. In a specific application, the terminal can determine the specified object according to the entity link indication information, and further can determine the object to be matched corresponding to the specified object according to the specified object.

[0083] Among them, the specified object can be the visual object to be matched or the named entity to be matched. When the specified object is the visual object to be matched, the object to be matched corresponding to the specified object is the named entity to be matched corresponding to the visual object to be matched. For example, when the specified object is the visual object to be matched, the object to be matched can specifically be all named entities in the entity knowledge base. When the specified object is the named entity to be matched, the object to be matched corresponding to the specified object is the visual object to be matched corresponding to the named entity to be matched. Another example is that when the specified object is the named entity to be matched, the object to be matched can specifically be all visual objects.

[0084] Step 206: Extract the visual object features of the visual object to be matched from the image based on the pixel mask of the visual object to be matched.

[0085] Among them, the visual object feature refers to a feature that can describe the object characteristics of the visual object to be matched. Through the visual object feature, the visual object to be matched can be distinguished from other visual objects. For example, the visual object feature can specifically be a feature vector that can describe the characteristics of the visual object to be matched, and can at least include the visual features of the visual object to be matched.

[0086] Exemplarily, the terminal will extract the first visual feature of the visual object to be matched from the image based on the pixel mask of the visual object to be matched, and then obtain the visual object feature based on the first visual feature.

[0087] In a specific application, the first visual feature refers to a feature extracted from the image and used to describe the visual characteristics of the visual object to be matched. Through the first visual feature, the visual feature to be matched can be distinguished from other visual objects in terms of the visual dimension. After extracting the first visual feature from the image, the terminal can directly use the first visual feature as the visual object feature, or can further obtain at least one of the entity link hint text for the image or the second visual feature of the remaining visual objects in the image, and interact the entity link hint text or at least one of the second visual features of the remaining visual objects in the image with the first visual feature to obtain a visual object feature that combines visual features and other features. Among them, the remaining visual objects refer to other visual objects except the visual object to be matched in the visual objects.

[0088] In a specific application, interacting the entity link hint text with the first visual feature can combine the first visual feature with semantic information to obtain a visual object feature that combines visual features and semantic features. Interacting the second visual feature of the remaining visual objects with the first visual feature can combine the first visual feature with context information to obtain a visual object feature that combines visual features and context features. If both the entity link hint text and the second visual feature of the remaining visual objects are used to interact with the first visual feature, the first visual feature can be combined with semantic information and context information to obtain a visual object feature that combines visual features, semantic features, and context features.

[0089] Step 208: Obtain the named entity feature of the named entity to be matched.

[0090] Among them, the named entity feature refers to a feature that can describe the entity characteristics of the named entity to be matched. Through the named entity feature, the named entity to be matched can be distinguished from other named entities. For example, the named entity feature can specifically be a feature vector that can describe the characteristics of the named entity to be matched.

[0091] Exemplarily, the terminal will obtain the named entity features of the named entity to be matched. The named entity features can be pre-generated or generated in real time. In a specific application, after determining the named entity to be matched, the terminal will obtain the named entity description information of the named entity to be matched from the entity knowledge base, and perform entity description expansion on the named entity to be matched to obtain the extended description information for the named entity to be matched. Then, the named entity features will be generated using the named entity description information and the extended description information.

[0092] Step 210: Match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched, and link the successfully matched visual object to be matched and the named entity to be matched.

[0093] Exemplarily, the terminal will match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched to obtain the feature similarity between the visual object to be matched and the named entity to be matched. Then, the successfully matched visual object to be matched and the named entity to be matched will be determined using the feature similarity, and the successfully matched visual object to be matched and the named entity to be matched will be linked.

[0094] The above visual entity linking method can obtain an accurate pixel mask marking the position of the visual object in the image through image segmentation by acquiring an image, segmenting the visual objects in the image, and generating a pixel mask for the segmented visual objects. By obtaining the entity linking indication information for the image, the entity linking indication information can be used to determine the visual object to be matched and the named entity to be matched. Furthermore, the pixel mask of the visual object to be matched can be used to extract the visual object features of the visual object to be matched from the image, and the named entity features of the named entity to be matched can be obtained. The visual object features of the visual object to be matched are matched with the named entity features of the named entity to be matched, and the successfully matched visual object to be matched and the named entity to be matched are linked. The entire process can combine the image segmentation results and achieve the linking of visual objects and named entities through feature matching, which can improve the entity recognition accuracy.

[0095] In an exemplary embodiment, extracting the visual object features of the visual object to be matched from the image based on the pixel mask of the visual object to be matched includes:

[0096] Based on the pixel mask of the visual object to be matched, extract the first visual feature of the visual object to be matched from the image, and obtain the entity linking hint text for the image;

[0097] Based on the entity linking hint text and the first visual feature, perform semantic marking on the visual object to be matched to obtain the first semantic marking feature of the visual object to be matched;

[0098] According to the first semantic marking feature, obtain the visual object feature of the visual object to be matched.

[0099] Among them, the entity link prompt text refers to the text information used to guide the matching of visual objects in an image with named entities in an entity knowledge base. For example, the entity link prompt text can be the description, title, etc. of the image, which are general information about the image, or the instruction task information indicating the visual entity link task, that is, the directive text used to guide the completion of the visual entity link task. Its role is to clearly tell the content or task objective that needs to be concerned, so as to improve the accuracy and efficiency of the link. Taking the image as Figure 4 shown as an example, the entity link prompt text can specifically be "I want to detect / recognize all entities in the image", "I want to detect / recognize vegetables in the image", "I want to detect / recognize broccoli in the image", etc.

[0100] Exemplarily, the terminal will, based on the pixel mask of the visual object to be matched, extract the first visual feature of the visual object to be matched from the image, and obtain the entity link prompt text for the image. Based on the entity link prompt text and the first visual feature, perform semantic marking on the visual object to be matched to obtain the first semantic marking feature of the visual object to be matched, and then according to the first semantic marking feature, obtain the visual object feature of the visual object to be matched. Among them, the entity link prompt text is pre-configured and can be configured by the user according to the actual application scenario.

[0101] In a specific application, when extracting the first visual feature, the terminal can first perform feature extraction on the image to obtain the image visual feature of the image, and then based on the pixel mask of the visual object to be matched, extract the first visual feature of the visual object to be matched from the image visual feature of the image. The terminal can also first, based on the pixel mask of the visual object to be matched, extract the local image where the visual object to be matched is located from the image, and then extract the first visual feature of the visual object to be matched from the local image.

[0102] In a specific application, when performing semantic marking, the terminal will extract the text feature from the entity link prompt text, and then perform multimodal fusion of the text feature and the first visual feature to obtain the first semantic marking feature of the visual object to be matched.

[0103] In a specific application, when extracting text features, the terminal first tokenizes the entity link hint text to obtain the text tokenization result, and then uses a pre-trained language model to extract text features. Among them, the pre-trained language model can be configured according to the actual application scenario. For example, the pre-trained language model can specifically be the BERT (Bidirectional Encoder Representations from Transformers) model. When performing multi-modal fusion, the terminal can use the attention mechanism or other interaction mechanisms to make the text features interact with the first visual features.

[0104] In a specific application, the process of semantic tagging can be represented by the following formula:

[0105] ;

[0106] Among them, refers to the first semantic tagging feature, is an operation that combines the entity link hint text with the first visual feature, that is, it represents performing semantic tagging, refers to the first visual feature, refers to the entity link hint text.

[0107] In a specific application, the semantic tagging in this embodiment can be implemented by a pre-trained large language model. By inputting the entity link hint text and the first visual feature into the large language model and giving appropriate prompt words, the large language model can output the first semantic tagging feature. Among them, the pre-trained large language model can be configured according to the actual application scenario, and the given prompt words can also be configured according to the actual application scenario. For example, the given prompt words can specifically be to perform multi-modal fusion of the entity link hint text and the first visual feature.

[0108] In this embodiment, using the pixel mask of the visual object to be matched, the first visual feature of the visual object to be matched can be accurately extracted from the image. Furthermore, on the basis of obtaining the entity link hint text for the image, the entity link hint text and the first visual feature can be used to perform semantic tagging on the visual object to be matched, obtaining the first semantic tagging feature of the visual object to be matched. According to the first semantic tagging feature, the visual object feature combining visual features and semantic features can be obtained to improve the fine-grained understanding ability of the visual object to be matched.

[0109] In an exemplary embodiment, extracting the first visual feature of the visual object to be matched from the image based on the pixel mask of the visual object to be matched includes;

[0110] Performing multi-level feature extraction on the image to obtain multi-level image visual features of the image;

[0111] Based on the pixel mask of the visual object to be matched, extract the multi-level object visual features of the visual object to be matched from the multi-level image visual features;

[0112] Fuse the multi-level object visual features to obtain the first visual feature of the visual object to be matched.

[0113] Exemplarily, the terminal will perform multi-level feature extraction on the image to obtain the multi-level image visual features of the image, and then, based on the pixel mask of the visual object to be matched and the feature dimensions of the image visual features at each level in the multi-level image visual features, determine the feature extraction mask for the image visual features at each level. For each level in the multi-level, the terminal will use the feature extraction mask of the image visual features at the targeted level to extract the object visual features of the visual object to be matched at the targeted level from the image visual features at the targeted level, aggregate the object visual features extracted from the image visual features at each level, obtain the multi-level object visual features, and fuse the multi-level object visual features to obtain the first visual feature of the visual object to be matched.

[0114] In a specific application, for each level in the multi-level, the terminal will adjust the dimension of the pixel mask of the visual object to be matched based on the feature dimension of the image visual features at the targeted level to obtain a feature extraction mask with the same feature dimension as the image visual features at the targeted level, and then multiply the obtained feature extraction mask element-wise with the image visual features at the targeted level to obtain the object visual features at the targeted level.

[0115] In a specific application, when fusing the multi-level object visual features, the terminal will first perform pooling on the object visual features at each level respectively to obtain the pooling features at each level, and then fuse the pooling features at each level to obtain the first visual feature of the visual object to be matched. Among them, the way of performing pooling can be average pooling or max pooling, or other pooling methods, which are not specifically limited here in this embodiment. When fusing the pooling features at each level, since the feature dimensions of the pooling features at each level are different, the terminal can convert the pooling features at each level into the same feature dimension as the pooling features at the adjacent level by sampling, and then fuse the pooling features at the two levels.

[0116] In this embodiment, by performing multi-level feature extraction on the image, multi-level image visual features of the image can be obtained. Furthermore, the pixel mask of the visual object to be matched can be used to extract multi-level object visual features of the visual object to be matched from the multi-level image visual features. By fusing the multi-level object visual features, the first visual feature of the visual object to be matched can be obtained. Throughout the process, the pixel mask can be used to accurately extract the first visual feature.

[0117] In an exemplary embodiment, obtaining the visual object feature of the visual object to be matched according to the first semantic marking feature includes:

[0118] Determine the remaining visual objects other than the visual object to be matched from the visual objects, and based on the pixel masks of the remaining visual objects, extract the second visual features of the remaining visual objects from the image;

[0119] Semantically mark the remaining visual objects based on the entity link prompt text and the second visual features to obtain the second semantic marking features of the remaining visual objects;

[0120] Interact the first semantic marking feature and the second semantic marking feature to obtain the visual object feature of the visual object to be matched.

[0121] Exemplarily, the terminal will also combine context features to generate the visual object feature of the visual object to be matched. Specifically, the terminal will determine the remaining visual objects other than the visual object to be matched from the visual objects, that is, the visual objects that do not belong to the visual object to be matched. Based on the pixel masks of the remaining visual objects, extract the second visual features of the remaining visual objects from the image. Semantically mark the remaining visual objects based on the entity link prompt text and the second visual features to obtain the second semantic marking features of the remaining visual objects, and then interact the first semantic marking feature and the second semantic marking feature to obtain the visual object feature of the visual object to be matched.

[0122] In a specific application, when extracting the second visual feature, the terminal can first perform feature extraction on the image to obtain the image visual features of the image, and then based on the pixel masks of the remaining visual objects, extract the second visual features of the remaining visual objects from the image visual features of the image. The terminal can also first, based on the pixel masks of the remaining visual objects, extract the local image where the remaining visual objects are located from the image, and then extract the second visual features of the remaining visual objects from the local image.

[0123] In a specific application, taking the feature extraction of an image as an example first, the method for extracting the second visual feature can be to perform multi-level feature extraction on the image to obtain the multi-level visual features of the image, and based on the pixel mask of the remaining visual object, extract the multi-level object visual features of the remaining visual object from the multi-level visual features of the image, and fuse the multi-level object visual features to obtain the second visual feature of the remaining visual object.

[0124] In a specific application, when performing semantic tagging, the terminal will extract text features from the entity link prompt text, and then perform multi-modal fusion of the text features and the second visual feature to obtain the second semantic tagging feature of the remaining visual object. In a specific application, when extracting text features, the terminal will first perform word segmentation on the entity link prompt text to obtain the text segmentation result, and then use a pre-trained language model to extract text features. Among them, the pre-trained language model can be configured according to the actual application scenario. When performing multi-modal fusion, the terminal can use an attention mechanism or other interaction mechanisms to make the text features and the second visual feature interact with each other. Among them, the attention mechanism used can specifically be a cross-attention mechanism.

[0125] In a specific application, the semantic tagging in this embodiment can be implemented through a pre-trained large language model. By inputting the entity link prompt text and the second visual feature into the large language model and giving appropriate prompt words, the large language model can output the second semantic tagging feature. Among them, the pre-trained large language model can be configured according to the actual application scenario, and the given prompt words can also be configured according to the actual application scenario. For example, the given prompt words can specifically be to perform multi-modal fusion of the entity link prompt text and the second visual feature.

[0126] In a specific application, the terminal can use an attention mechanism or other interaction mechanisms to interact the first semantic tagging feature and the second semantic tagging feature, so as to better capture the relationship between the first semantic tagging feature and the second semantic tagging feature, and combine visual features, semantic features and context features to obtain the visual object feature of the visual object to be matched. Among them, the attention mechanism used can specifically be a cross-attention mechanism.

[0127] In this embodiment, by determining the remaining visual objects other than the visual objects to be matched from the visual objects, and extracting the second visual features of the remaining visual objects from the image based on the pixel masks of the remaining visual objects, the acquisition of the second visual features as context features can be achieved. By semantically labeling the remaining visual objects based on the entity link prompt text and the second visual features to obtain the second semantic labeling features of the remaining visual objects, the second semantic labeling features combining visual features and semantic features can be obtained. Furthermore, by interacting the first semantic labeling features and the second semantic labeling features, the visual object features combining visual features, semantic features and context features can be obtained to improve the fine-grained understanding ability of the visual objects to be matched.

[0128] In an exemplary embodiment, interacting the first semantic labeling features and the second semantic labeling features to obtain the visual object features of the visual objects to be matched includes:

[0129] Obtaining multi-level image visual features of the image, performing multi-layer perception processing on the multi-level image visual features to obtain the image features of the image;

[0130] Interacting the image features, the first semantic labeling features and the second semantic labeling features to obtain the visual object features of the visual objects to be matched.

[0131] Exemplarily, when interacting the semantic labeling features, the terminal will obtain multi-level image visual features of the image, and further abstract the multi-level image visual features by performing multi-layer perception processing on the multi-level image visual features to obtain a higher-level data representation, that is, the image features of the image. On the basis of obtaining the image features of the image, the terminal will use the image features as the overall features to participate in the interaction, and interact the image features, the first semantic labeling features and the second semantic labeling features to obtain the visual object features of the visual objects to be matched.

[0132] In a specific application, when performing the interaction, the terminal will use an attention mechanism or other interaction mechanisms to interact the image features, the first semantic labeling features and the second semantic labeling features, so as to better capture the relationships among the image features, the first semantic labeling features and the second semantic labeling features, and combine the overall visual features, the local visual features of the visual objects to be matched, semantic features and context features to obtain the visual object features of the visual objects to be matched. Among them, the attention mechanism used can specifically be a cross-attention mechanism.

[0133] In this embodiment, by extracting the image features of an image, interacting the image features, the first semantic marker features, and the second semantic marker features, a visual object feature of the visual object to be matched is obtained, which combines the overall visual features, the local visual features of the visual object to be matched, the semantic features, and the context features, thereby improving the fine-grained understanding ability of the visual object to be matched.

[0134] In an exemplary embodiment, interacting the image features, the first semantic marker features, and the second semantic marker features to obtain the visual object feature of the visual object to be matched includes:

[0135] Concatenating the image features and the second semantic marker features to obtain a concatenated feature;

[0136] Interacting the first semantic marker features and the concatenated feature through a cross-attention mechanism to obtain the visual object feature of the visual object to be matched.

[0137] Exemplarily, when performing feature interaction, the terminal first concatenates the image features and the second semantic marker features to obtain a concatenated feature, and then interacts the first semantic marker features and the concatenated feature through a cross-attention mechanism to obtain the visual object feature of the visual object to be matched.

[0138] In a specific application, when performing interaction, the terminal can use the first semantic marker features as query vectors, the concatenated features as key vectors and value vectors, project the query vectors, key vectors, and value vectors respectively through the projection matrix of the cross-attention mechanism, then calculate the attention weights using the projected query vectors and projected key vectors, and finally perform weighted summation using the attention weights and the projected value vectors to obtain the visual object feature of the visual object to be matched.

[0139] In this embodiment, by concatenating the image features and the second semantic marker features, a concatenated feature that serves as a reference for the first semantic marker features can be obtained. Furthermore, through the cross-attention mechanism, the first semantic marker features and the concatenated feature can be interacted, so that by combining the overall visual features, the local visual features of the visual object to be matched, the semantic features, and the context features, the visual object feature of the visual object to be matched can be obtained, thereby improving the fine-grained understanding ability of the visual object to be matched.

[0140] In an exemplary embodiment, according to the entity link indication information, determining the visual object to be matched from the visual objects and determining the named entity to be matched includes:

[0141] When the entity link indication information includes visual object marker information, based on the visual object marker information, determining the regional coordinate information of the visual object marker region;

[0142] Based on the regional coordinate information and the pixel mask of the visual object, perform labeled object recognition to determine the visual object labeled by the visual object labeling region;

[0143] Take the labeled visual object as the visual object to be matched, and select the named entity to be matched from the entity knowledge base.

[0144] Among them, the visual object labeling information refers to the information used to specify the visual object to be matched. For example, the visual object labeling information can specifically refer to the information of the visual object labeling region used to specify the visual object to be matched, which is used to clearly specify the position of the visual object to be matched in the image. The visual object labeling region here can specifically be a marking point, a marking box, etc.

[0145] For example, the visual object labeling information can specifically refer to the regional coordinate information of the visual object labeling region used to specify the visual object to be matched. If the labeling region is a marking point, the visual object labeling information is the coordinate information of the marking point. If the labeling region is a marking box, the visual object labeling information is the coordinate information of the marking box, such as the upper left corner coordinate and the lower right corner coordinate of the marking box. It should be noted that the coordinates here refer to the pixel coordinates in the image, and the pixel coordinates describe the position of the pixel points in the image, usually represented by a pair of numerical values in the format of (x, y).

[0146] Exemplarily, when the entity link indication information includes the visual object labeling information, indicating that the user has selected the visual object to be matched, the terminal will, based on the visual object labeling information, determine the regional coordinate information of the visual object labeling region, perform labeled object recognition based on the regional coordinate information and the pixel mask of the visual object, determine the visual object labeled by the visual object labeling region from the visual objects, take the labeled visual object as the visual object to be matched, and select the named entity to be matched from the entity knowledge base. The named entity to be matched can specifically be all the named entities in the entity knowledge base.

[0147] In a specific application, if the labeling region is a marking point, the terminal will, based on the regional coordinate information and the pixel mask of the visual object, determine the positional relationship between the marking point and the visual object, and take the visual object closest to the marking point as the visual object to be matched. In a specific application, taking an image as Figure 4 shown as an example, which contains three visual objects: potato 402, meat 404, and broccoli 406. If the labeling region is a marking point and the position is as Figure 4 shown, located on the broccoli 406, the terminal can determine that the visual object closest to the marking point is the broccoli, and take the broccoli as the visual object to be matched 408.

[0148] In a specific application, if the marked area is a marked box, the terminal will determine the area matching degree between the marked box and the local image area where the visual object is located based on the area coordinate information and the pixel mask of the visual object, and use the visual object in the local image area with the highest area matching degree with the marked box as the visual object to be matched. Among them, the area matching degree can specifically be the area coincidence degree. In a specific application, taking an image such as Figure 5 as an example, which contains three visual objects: potato 502, meat 504, and broccoli 506. If the marked area is a marked box and the position is as Figure 5 shown, the terminal can determine through calculation that the visual object in the local image area with the highest area matching degree with the marked box is broccoli, and use broccoli as the visual object to be matched 508.

[0149] In this embodiment, when the entity link indication information includes visual object marking information, it is possible to use the visual object marking information to determine the area coordinate information of the marked area of the visual object, and then use the area coordinate information and the pixel mask of the visual object to perform marked object recognition, and accurately locate the specified visual object to be matched at the pixel level. Thus, on the basis of selecting the named entity to be matched from the entity knowledge base, it is possible to accurately determine the visual object to be matched and the named entity to be matched.

[0150] In an exemplary embodiment, determining the visual object to be matched and the named entity to be matched from the visual objects according to the entity link indication information includes:

[0151] When the entity link indication information includes named entity specification information, perform named entity recognition on the named entity specification information to determine the named entity specified by the named entity specification information;

[0152] Use the specified named entity as the named entity to be matched, and use the visual object as the visual object to be matched.

[0153] Among them, the named entity specification information refers to the information used to specify the named entity to be matched. For example, the named entity specification information can specifically include the entity identifier used to specify the named entity to be matched, such as the entity name, etc.

[0154] Exemplarily, when the entity link indication information includes named entity specification information, it means that the user has selected the named entity to be matched. The terminal will perform named entity recognition on the named entity specification information to determine the named entity specified by the named entity specification information, use the specified named entity as the named entity to be matched, and use all visual objects as the visual objects to be matched.

[0155] In a specific application, taking an image such as Figure 6Taking the example shown, which includes three visual objects: potato 602, meat 604, and broccoli 606. If the named entity specified information is to identify broccoli, by performing named entity recognition on the named entity specified information, it can be determined that the specified named entity is broccoli. The terminal will take potato 602, meat 604, and broccoli 606 as the visual objects to be matched, and take broccoli as the specified visual object to be matched. The visual object features of the visual objects to be matched are matched with the named entity features of the named entity to be matched, so as to link broccoli with Figure 6 the broccoli 606 in

[0156] In this embodiment, when the entity link indication information includes the named entity specified information, by performing named entity recognition on the named entity specified information, it is possible to determine the specified named entity, that is, the named entity to be matched. Furthermore, based on taking the visual object as the visual object to be matched, it is possible to accurately determine the visual object to be matched and the named entity to be matched.

[0157] In an exemplary embodiment, obtaining the named entity features of the named entity to be matched includes:

[0158] Performing entity description expansion on the named entity to be matched to obtain extended description information for the named entity to be matched;

[0159] Obtaining the named entity description information for the named entity to be matched, and fusing the extended description information and the named entity description information to obtain the fused description information for the named entity to be matched;

[0160] Extracting the named entity features of the named entity to be matched from the fused description information. Among them, the extended description information refers to the information used to expand the description of the named entity to be matched outside the named entity description information. For example, the extended description information can specifically be at least one of the associated entity description information of the associated entity associated with the category of the named entity to be matched and the relationship description information for the visual object to be matched extracted from the text query statement.

[0161] Among them, the named entity description information refers to the information used to describe the named entity to be matched. For example, the named entity description information can specifically be a description image or a description text for describing the named entity to be matched.

[0162] Exemplarily, after determining the named entity to be matched, the terminal will perform entity description expansion on the named entity to be matched to obtain extended description information for the named entity to be matched, and obtain named entity description information for the named entity to be matched. The extended description information and the named entity description information are fused to obtain fused description information for the named entity to be matched. Then, from the fused description information, the named entity features of the named entity to be matched are extracted by means of text feature extraction. Among them, the method of text feature extraction is not limited in this embodiment. For example, text feature extraction can be performed through a pre-trained text feature extraction model.

[0163] In a specific application, when performing entity description expansion, it can be considered to perform description expansion starting from the named entity to be matched itself, or it can be considered to combine the named entity to be matched with the text query statement for description expansion. Specifically, when considering performing description expansion starting from the named entity to be matched itself, the terminal can query the associated entities associated with the named entity to be matched, and use the associated entity description information of the associated entities for description expansion. Among them, this association can be based on various relationships, including inheritance relationships, co-ownership relationships, etc. When considering combining the named entity to be matched with the text query statement for description expansion, the terminal can extract the relationship description of the named entity to be matched from the text query statement to utilize this relationship description to enrich the semantic information of the named entity to be matched.

[0164] In a specific application, when the named entity description information includes the description text for describing the named entity to be matched, the terminal can directly fuse the extended description information and the description text through semantic analysis to obtain the fused description information for the named entity to be matched, and then extract the named entity features of the named entity to be matched from the fused description information.

[0165] In a specific application, through semantic analysis, the terminal can obtain the limiting information for further limiting the description text from the extended description information, and then can combine the limiting information and the description text to obtain the fused description information. Here, the limiting information can be the limitation of attributes such as the type and location of the named entity to be matched. For example, the fusion in this embodiment can be realized through a pre-trained large language model. The terminal can obtain the fused description information by inputting the extended description information and the description text into the pre-trained large language model and giving appropriate prompt words. Among them, the prompt words can be configured according to the actual application scenario. For example, the prompt words can specifically be "Enrich the description of the description text through the extended description information to generate the fused description information".

[0166] In a specific application, taking the extended description information as an example of the associated entity description information of the associated entities, if the associated entity is a parent entity with an inheritance relationship, that is, when the named entity to be matched is a subclass of the associated entity, the form of the fused description information can specifically be an (associated entity) called (the named entity to be matched). If the associated entity is a co-owned entity, the form of the fused description information can specifically be a (named entity to be matched) with the same category as the (associated entity).

[0167] In a specific application, taking the extended description information as an example of the description of the named entity to be matched extracted from the text query statement, if the extracted description is a positional relationship description, such as located in the sky, the form of the fused description information can specifically be a (named entity to be matched) located in the sky. If the extracted description is a semantic relationship description, such as belonging to category A, the form of the fused description information can specifically be a (named entity to be matched) belonging to category A.

[0168] In a specific application, when the named entity description information includes a description image for describing the named entity to be matched, the terminal can first generate a description text for the named entity to be matched based on the description image, and then fuse the extended description information and the description text through semantic analysis to obtain the fused description information for the named entity to be matched. Among them, the description text can be obtained based on a pre-trained image-to-text model, and the image-to-text model can be configured according to the actual application scenario.

[0169] In this embodiment, by performing entity description extension on the named entity to be matched, the extended description information for the named entity to be matched can be obtained. Furthermore, on the basis of obtaining the named entity description information for the named entity to be matched, by fusing the extended description information and the named entity description information, the extension and enrichment of the entity description of the named entity to be matched can be realized, and the fused description information can be obtained. Furthermore, the named entity features that can accurately represent the named entity to be matched can be extracted from the fused description information.

[0170] In an exemplary embodiment, obtaining the named entity features of the named entity to be matched based on the named entity description features includes:

[0171] Perform an associated category query on the named entity to be matched to determine the associated entity associated with the category of the named entity to be matched;

[0172] Obtain the associated entity description information of the associated entity, and use the associated entity description information as the extended description information for the named entity to be matched.

[0173] Among them, the associated category refers to other categories associated with the named entity to be matched. This association can be based on various relationships, including inheritance relationships, co-subordination relationships, etc. For example, the associated category can specifically refer to the parent class (superclass) or subclass associated with the named entity to be matched. Another example is that the associated category can specifically refer to the category that belongs to the same parent class as the named entity to be matched.

[0174] Among them, the associated entity refers to the entity associated with the category of the named entity to be matched. The associated entity description information refers to the information used to describe the associated entity. For example, the associated entity description information can specifically be the description text for describing the associated entity, such as the name of the associated entity of the associated entity.

[0175] Exemplarily, the terminal will query the associated category of the named entity to be matched, determine the associated entity associated with the category of the named entity to be matched, and then obtain the associated entity description information of the associated entity, and use the associated entity description information as the extended description information for the named entity to be matched.

[0176] In a specific application, the terminal can query the associated category of the named entity to be matched by calling a query interface to query the associated entity associated with the category of the named entity to be matched from the Internet data. In addition, the terminal can also query the associated category of the named entity to be matched through a pre-trained large language model. Among them, the prompt word for querying using the large language model can be "query the associated entity associated with the category of the named entity to be matched". Among them, the terminal can query with the description text of the named entity to be matched, or can query with the description image of the named entity to be matched. When querying with the description image, the terminal can query the similar image of the description image, and then determine the associated entity based on the similar image.

[0177] In a specific application, after using the associated entity description information as the extended description information, the terminal can fuse the associated entity description information and the named entity description information to obtain the fused description information. It can also further obtain the text query statement for the image, obtain the remaining extended description information for the named entity to be matched from the text query statement, and then fuse the associated entity description information, the named entity description information, and the remaining extended description information to obtain the fused description information.

[0178] In a specific application, taking the fusion of the associated entity description information and the named entity description information to obtain the fused description information as an example, the fused description information can be represented by the following formula:

[0179] ;

[0180] Among them, refers to the fused description information, Refers to named entity description information. Indicates fusion. Refers to associated entity description information.

[0181] In a specific application, for example, if the named entity to be matched is a certain aircraft model, the associated category can specifically be the parent class (superclass) associated with this aircraft model, or a category that belongs to the same parent class as this aircraft model. As Figure 7 shown, when the named entity to be matched is Aircraft Model 1, by querying the associated category from the dataset (including named entities: Aircraft Model 1, Named Entity: Aircraft Model 2, Named Entity: Aircraft Model 3, Named Entity: Aircraft Model 4, etc.), it can be determined that the parent class of Aircraft Model 1 is the double-decker aircraft under Aircraft. Through this parent class of double-decker aircraft, the category that belongs to the same parent class as Aircraft Model 1 and is associated with it can be determined as Aircraft Model 2. Further, taking the fusion of the associated entity description information and the named entity description information to obtain the fusion description information as an example, the effect of this embodiment is described. The associated entity can be the parent aircraft, and the associated entity description information can be the associated entity name, that is, aircraft. Then, by using the associated entity description information as the extended description information for connotation extension and fusing it with the named entity description information (such as Figure 7 shown as the description text and description image of Aircraft Model 1), the obtained fusion description information can be a more comprehensive description text representing Aircraft Model 1 (such as Figure 7 shown as "an aircraft called Aircraft Model 1") and a similar image (not shown in Figure 7 ).

[0182] In this embodiment, the method of querying the associated entity and using the associated entity description information of the associated entity to enrich the entity description of the named entity to be matched is beneficial to achieving the accurate matching of the named entity to be matched and the visual object to be matched. It can be understood that through this method, the recognition ability can be significantly improved when the visual object to be matched is a long-tail entity, that is, it can still accurately understand and match the long-tail entity in the case of insufficient context. Among them, the long-tail entity refers to an entity that is relatively less mentioned in a large text dataset.

[0183] In an exemplary embodiment, based on the named entity description features, the named entity features of the named entity to be matched are obtained, including:

[0184] Obtain a text query statement for the image;

[0185] Perform semantic parsing on the text query statement to determine the query subject representing the visual object to be matched, and extract the relationship description information for the query subject;

[0186] The relationship description information is used as the extended description information for the named entity to be matched. Among them, the text query statement refers to the information provided by the user for identifying the visual object to be matched from the image. It provides the context information of the image content, helps to understand the user's needs, and extracts relevant information from the image. The relationship description information refers to the context information related to the query subject representing the visual object to be matched, which is determined by analyzing the spatial or semantic relationship in the text query statement. For example, the relationship description information can specifically refer to the information indicating the spatial position of the query subject representing the visual object to be matched, such as in the sky, in the water, etc. Another example is that the relationship description information can specifically refer to the information indicating the positional relationship between the query subject representing the visual object to be matched and other visual objects.

[0187] Exemplarily, when performing entity description extension, the terminal can obtain the text query statement for the image, perform semantic parsing on the text query statement, determine the query subject representing the visual object to be matched, and extract the relationship description information for the query subject, and use the relationship description information as the extended description information for the named entity to be matched.

[0188] In a specific application, the user can provide a text query statement for the image. When the user provides a text query statement for the image, the terminal can use the text query statement to enhance the extension information of the visual object to be matched to improve the entity recognition accuracy. When performing semantic parsing, the terminal will first perform word segmentation and part-of-speech tagging on the text query statement, then use syntactic analysis to locate the query subject from the text query statement, and then extract the relationship description information for the query subject. Among them, the query subject is usually the object or subject in the statement. In a specific application, the terminal can use a pre-trained large language model to perform semantic parsing on the text query statement, determine the query subject, and extract the relationship description information for the query subject. Among them, the pre-trained large language model can be trained according to the actual application scenario, and the form of the prompt words used can be "Extract the query subject to be queried from this sentence and extract the relationship description information for the query subject".

[0189] In a specific application, after using the relationship description information as the extended description information, the terminal can fuse the relationship description information and the named entity description information to obtain the fused description information. It can also further perform an associated category query on the named entity to be matched to obtain the associated entity, and after using the associated entity description information of the associated entity as the extended description information, fuse the associated entity description information, the named entity description information, and the relationship description information to obtain the fused description information.

[0190] In a specific application, taking the example of fusing relationship description information and named entity description information to obtain fused description information, the fused description information can be represented by the following formula:

[0191] ;

[0192] Wherein, refers to the fused description information, refers to the named entity description information, represents fusion refers to the relationship description information.

[0193] In a specific application, for example, if the visual object to be matched is a certain flight model and the text query statement contains context information indicating the spatial position of the visual object to be matched, such as "What is the object in the sky", the terminal can determine from the text query statement that the query subject representing the visual object to be matched is "object", extract the relationship description information "in the sky" for the query subject, and combine it with the named entity description information of the named entity to be matched to obtain the fused description information, so as to improve the context information related to the visual object to be matched and assist in understanding the visual object to be matched. Further, taking the example of fusing relationship description information and named entity description information to obtain fused description information, the effect of this embodiment is described. By using the relationship description information ("in the sky") as the extensible description information for extension and fusing it with the named entity description information (specifically, it can be the description text and description image of a certain aircraft model), the obtained fused description information can be a more comprehensive description text representing the aircraft model (such as Figure 8 shown as "is a 'commercial aircraft' or 'jet aircraft'").

[0194] In this embodiment, in this way, the relationship description information extracted from the text query statement can be used to enrich the entity description of the named entity to be matched, which is beneficial to realizing the accurate matching of the named entity to be matched and the visual object to be matched. It can be understood that in this way, when the visual object to be matched is a long-tail entity, the recognition ability can be significantly improved, that is, the long-tail entity can still be accurately understood and matched in the case of lack of sufficient context.

[0195] In an exemplary embodiment, as Figure 9 shown, a visual entity linking method is provided. In this embodiment, taking the application of this method to the terminal as an example, it can be understood that this method can also be applied to the server, and can also be applied to a system including the terminal and the server and realized through the interaction between the terminal and the server. In this embodiment, this method includes the following steps 902 to step 930. Wherein:

[0196] Step 902: Obtain an image, segment visual objects in the image, generate pixel masks for the segmented visual objects, and the pixel masks mark the positions of the visual objects in the image.

[0197] Step 904: Obtain entity link indication information for the image. Based on the entity link indication information, determine the visual objects to be matched from the visual objects and determine the named entities to be matched.

[0198] In a specific application, such as Figure 10 As shown, when the entity link indication information includes visual object marking information, the terminal will, based on the visual object marking information, determine the regional coordinate information of the visual object marking area. Based on the regional coordinate information and the pixel mask of the visual object, perform marked object recognition to determine the visual object marked by the visual object marking area, and use the marked visual object as the visual object to be matched, and select the named entity to be matched from the entity knowledge base. When the entity link indication information includes named entity specification information, the terminal will perform named entity recognition on the named entity specification information to determine the named entity specified by the named entity specification information, use the specified named entity as the named entity to be matched, and use the visual object as the visual object to be matched.

[0199] Step 906: Extract multi-level features from the image to obtain multi-level image visual features of the image.

[0200] Step 908: Based on the pixel mask of the visual object to be matched, extract multi-level object visual features of the visual object to be matched from the multi-level image visual features.

[0201] Step 910: Fuse the multi-level object visual features to obtain the first visual feature of the visual object to be matched, and obtain entity link hint text for the image.

[0202] Step 912: Based on the entity link hint text and the first visual feature, perform semantic marking on the visual object to be matched to obtain the first semantic marking feature of the visual object to be matched.

[0203] Step 914: From the visual objects, determine the remaining visual objects other than the visual object to be matched. Based on the pixel mask of the remaining visual objects, extract the second visual feature of the remaining visual objects from the image.

[0204] Step 916: Based on the entity link hint text and the second visual feature, perform semantic marking on the remaining visual objects to obtain the second semantic marking feature of the remaining visual objects.

[0205] Step 918: Perform multi-layer perception processing on the multi-level image visual features to obtain the image feature of the image.

[0206] Step 920: Concatenate the image features and the second semantic marker features to obtain concatenated features.

[0207] Step 922: Through a cross-attention mechanism, interact the first semantic marker features and the concatenated features to obtain the visual object features of the visual object to be matched.

[0208] Step 924: Expand the entity description of the named entity to be matched to obtain the extended description information for the named entity to be matched.

[0209] Step 926: Obtain the named entity description information for the named entity to be matched, and fuse the extended description information and the named entity description information to obtain the fused description information for the named entity to be matched.

[0210] In a specific application, as Figure 11 shown, the method of obtaining the fused description information may include the following steps 1102 to 1106. Among them: Step 1102: Perform an associated category query on the named entity to be matched to determine the associated entity associated with the category of the named entity to be matched. Step 1104: Obtain the associated entity description information of the associated entity, and use the associated entity description information as the extended description information for the named entity to be matched. Step 1106: Fuse the extended description information and the named entity description information to obtain the fused description information for the named entity to be matched.

[0211] In a specific application, as Figure 12 shown, the method of obtaining the fused description information may include the following steps 1202 to 1208. Among them: Step 1202: Obtain the text query statement for the image. Step 1204: Perform semantic parsing on the text query statement to determine the query subject representing the visual object to be matched, and extract the relationship description information for the query subject. Step 1206: Use the relationship description information as the extended description information for the named entity to be matched. Step 1208: Fuse the extended description information and the named entity description information to obtain the fused description information for the named entity to be matched.

[0212] Step 928: Extract the named entity features of the named entity to be matched from the fused description information.

[0213] Step 930: Match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched, and link the successfully matched visual object to be matched and the named entity to be matched.

[0214] In an exemplary embodiment, taking the application in a complex scenario (such as entity linking for an image with multiple objects or a blurred background) as an example, the visual entity linking method of the present application is described in detail.

[0215] With the rapid development of computer vision and artificial intelligence technology, visual entity linking, as an important visual understanding task, has been widely used in image retrieval, visual question answering, image annotation and other fields. The goal of the visual entity linking task is to match the visual entities in the image with the entities in the external knowledge base.

[0216] Traditional visual entity linking methods usually rely on text queries to identify entities in images, requiring users to provide detailed text descriptions or questions to help the model identify the target entities in the image. However, in complex or ambiguous scenes, constructing accurate text queries is often difficult. For example, when describing a specific visual object in an image, the user needs to have a detailed understanding of the spatial relationship and semantics in the image, which is a big challenge for general users and will result in low entity recognition accuracy. In addition, in the face of complex scenes, users usually need to provide text query statements multiple times to help the model accurately match the visual objects in the image with the entities in the entity knowledge base, which will result in low entity recognition efficiency.

[0217] In addition, the recognition of long-tail entities in large-scale open source entity knowledge bases remains a challenge. Many less popular or uncommon entities cannot be accurately matched by standard text query methods because they do not have enough training samples or relevant contextual information. Traditional methods are prone to recognition errors or missed detections when faced with these long-tail entities, resulting in limited performance of the overall system. Moreover, traditional visual understanding methods often remain at a relatively rough level in feature extraction of image regions. For example, methods based on image patches may not be able to effectively capture the fine-grained features of each region in the image, especially when the scene is complex and there is a close relationship between objects. Traditional methods may not be able to accurately distinguish different objects, resulting in insufficient in-depth visual understanding and affecting the final entity matching effect.

[0218] Based on this, this application provides a visual entity linking method, which aims to solve the shortcomings of traditional visual entity linking methods in complex images and multimodal scenes through innovative technologies. To this end, the key technical points of this application are as follows:

[0219] 1. Pixel-level visual entity linking and back-annotation framework.

[0220] This application proposes a visual entity linking method based on pixel masks, which can accurately link visual objects in images with named entities in entity knowledge bases through pixel-level input. Through the reverse annotation framework, the corresponding image regions can be derived from existing named entities and high-quality pixel masks can be generated. This innovative method effectively reduces the search space and improves the annotation accuracy, and is particularly suitable for processing fine-grained images and complex visual scenes.

[0221] 2. Knowledge Enhancement and Visual Semantic Tokenization Technology

[0222] This application combines knowledge enhancement technology with visual semantic tokenization method, while improving the accuracy of visual understanding, enhancing the semantic interaction between images and texts. Knowledge enhancement improves the accuracy of annotations and entity recognition ability through extensional description and relationship enhancement. The visual semantic tokenization technology makes the visual features of image regions more closely combined with semantic information, improves the capture and understanding of details, enables each pixel region in the image to interact efficiently with semantic information, and enhances the performance of the overall visual entity linking task.

[0223] It can be understood that through the above technological innovations, this application provides a more efficient and accurate visual entity linking method, with broad prospects for business implementation. Specifically, in the first aspect, by introducing the pixel-level visual entity linking method and using pixel masks to replace traditional text queries, this application enables visual objects in images to be identified through simple visual inputs from users (such as clicking, box selection, or region segmentation), without the need for complex text descriptions. This greatly reduces the input burden on users and improves the accurate recognition ability of entities in complex scenarios. In the second aspect, the reverse annotation framework proposed in this application derives pixel regions in the image from existing named entities for accurate pixel-level annotation. This method not only reduces the search space but also improves the annotation accuracy. By combining reverse annotation with advanced image segmentation technology, fine-grained features in images can be processed more effectively, improving the accuracy of region-entity matching. In the third aspect, by introducing knowledge enhancement technology to expand entity descriptions, querying external knowledge bases to enrich the descriptions of long-tail entities, and using semantic understanding models such as pre-trained large language models to extract the spatial or semantic relationships of entities, this application can effectively improve the recognition ability of long-tail entities, thereby improving the overall entity recognition accuracy. In the fourth aspect, this application proposes a visual semantic tokenization method. By combining region features with text instructions, each pixel region in the image can interact efficiently with semantic information. This method improves the understanding of image details by enhancing region interaction features and using semantic-based visual tokens, enabling the system to accurately identify and link fine-grained visual objects, especially in scenarios with multiple objects and complex backgrounds.

[0224] In specific applications, the visual entity linking method of this application includes five parts: one is to perform forward entity recognition on the input image I and pixel mask mi; the second is the reverse annotation framework, which derives the image region mi based on the entity e; the third is knowledge enhancement, which enriches entity information through extensional description and extended relationship description; the fourth is visual semantic tokenization, which combines region features with text instructions to generate semantic tokens; and the fifth is the final entity matching, which matches entities based on similarity and outputs the final result.

[0225] In a specific application, the technical principle diagram of the visual entity linking method of the present application can be as Figure 13 shown. Below, in combination with Figure 13 , taking the application to a terminal as an example, the visual entity linking method of the present application will be described in detail.

[0226] Among them, the pixel-level visual entity linking method refers to using a pixel mask as the input to directly refer to the visual object in the image, avoiding the ambiguity of text description.

[0227] Specifically, as Figure 13 shown, the terminal will acquire an image, segment the visual object in the image, generate a pixel mask for the segmented visual object, and the pixel mask marks the position of the visual object in the image. In the case of generating the pixel mask, the terminal will acquire the entity linking indication information for the image, and based on the entity linking indication information, determine the visual object to be matched from the visual objects, and determine the named entity to be matched.

[0228] In a specific application, the user can select the visual object to be recognized through simple inputs (such as clicking, box selection), etc. At this time, the entity linking indication information includes visual object marking information. The terminal can determine the regional coordinate information of the visual object marking area based on the visual object marking information, and further, in combination with the regional coordinate information and the pixel mask of the visual object, perform marked object recognition, determine the visual object marked by the visual object marking area from the visual objects, that is, the visual object to be matched, and select the named entity to be matched from the entity knowledge base. Based on the pixel mask of the visual object to be matched, the terminal can match the visual object to be matched with the named entity to be matched to ensure the accurate association of the visual object in the image with the named entity in the entity knowledge base and find the named entity that matches the visual object to be matched.

[0229] In a specific application, the effect of visual entity linking of the present application by using visual object marking information is described. As Figure 14 shown, in the case of needing to link the broccoli in the image with the broccoli in the entity knowledge base, the traditional method needs to achieve the link through multiple rounds of text queries, such as Figure 14As shown, there are three rounds. The text query statement in the first round is "What's in the plate?", the text query statement in the second round is "What's next to the fries on the plate?", and the text query statement in the third round is "Next to the fries on the tablet, what is the small tree-shaped vegetable?". There are problems of low entity recognition efficiency and dependence on text query statements, which may lead to inaccurate entity recognition. By using the pixel-level visual entity linking method in this application, users only need to click or box the broccoli in the image to accurately select the broccoli in the image, which can improve the entity recognition efficiency and entity recognition accuracy.

[0230] Specifically, as Figure 13 shown, the terminal will extract the visual object features of the visual object to be matched from the image based on the pixel mask of the visual object to be matched, and obtain the named entity features of the named entity to be matched, and match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched to match the visual object to be matched with the named entity to be matched. The feature similarity Sim(mi, e) of feature matching can be expressed by the following formula:

[0231] ;

[0232] Among them, refers to the feature similarity, refers to the pixel mask of the visual object to be matched, refers to the local image area where the visual object to be matched is located, refers to the visual object feature, e refers to the named entity to be matched, refers to the named entity feature.

[0233] It can be understood that the pixel-level visual entity linking method avoids the ambiguity of text queries and directly calibrates the image accurately through the pixel mask, greatly improving the entity recognition and linking accuracy in complex scenarios.

[0234] Among them, the reverse annotation framework improves the accuracy of annotation by deriving the area in the image from the named entity. That is, the user can specify the named entity, such as Figure 13As shown, at this time, the entity link indication information includes named entity specification information. The terminal will perform named entity recognition on the named entity specification information, determine the named entity specified by the named entity specification information, and determine the specified named entity as the named entity to be matched. On this basis, the possible position area of the named entity to be matched in the image is determined by using the named entity features of the named entity to be matched, that is, the visual object is used as the visual object to be matched, and the visual object features of the visual object to be matched are matched with the named entity features of the named entity to be matched to match the visual object to be matched with the named entity to be matched, and the visual object matching the specified named entity to be matched is found from the image.

[0235] It should be noted that the position of the visual object in the image is marked by a pixel mask. Therefore, here it is equivalent to finding the pixel region mi (i.e., the pixel mask) corresponding to the specified named entity to be matched on the basis of using image segmentation to generate the pixel mask m of the visual object. The key of the reverse annotation framework is to reverse-derive the correspondence between the named entity and the image region, which greatly reduces the search space and improves the annotation efficiency. Compared with the traditional method of mapping from the image region to the named entity, the reverse annotation framework can significantly reduce the possibility of incorrect matching. Among them, the formula for determining that the pixel region mi in the image I corresponds to the named entity e according to the named entity features of the named entity e is as follows:

[0236] ;

[0237] Among them, is an image segmentation operation used to extract the pixel region mi corresponding to the named entity e from the image I.

[0238] Among them, in order to improve the understanding and annotation accuracy of long-tail entities, this application introduces knowledge enhancement technology, and improves the recognition ability of rare entities by enriching entity descriptions. Specifically, in this application, extended descriptions and extended relationship descriptions are used to enhance the semantic information of entities. Specifically, it includes: extended descriptions, which enrich the connotative description of entities by querying the associated categories (specifically, superclasses or related categories) of named entities. Extended relationship descriptions, which enhance the extensional information of entities by using large language models, etc. to extract spatial or semantic relationships related to entities. It can be understood that through extended descriptions and relationship descriptions, the knowledge enhancement technology significantly improves the model's recognition ability of long-tail entities, enabling accurate understanding and matching of long-tail entities even in the absence of sufficient context.

[0239] In a specific application, such as Figure 13As shown in the figure, the method of extended description is as follows: by performing an associated category query on the named entity to be matched, obtaining the associated entity associated with the category of the named entity to be matched, and using the associated entity description information of the associated entity as the extended description information, combining it with the named entity description information of the named entity to be matched to obtain the fusion description information. Assume the associated entity is , and the named entity to be matched is e. Then, the fusion description information obtained by the method of extended description can be expressed as:

[0240] ; where refers to the fusion description information, refers to the named entity description information, represents fusion, refers to the associated entity description information.

[0241] In a specific application, as shown in Figure 13 , the method of extended relationship description is as follows: using a large language model to analyze the spatial or relationship semantics in the text query statement, and extracting the context information related to the visual object to be matched to help better understand the long-tail entity. That is, obtaining the text query statement for the image, performing semantic parsing on the text query statement, determining the query subject representing the visual object to be matched, and extracting the relationship description information for the query subject, and using the relationship description information as the extended description information. Assume the relationship description feature is , and the named entity to be matched is e. Then, the fusion description information obtained by the method of extended relationship description can be expressed as:

[0242] ;

[0243] where refers to the fusion description information, refers to the named entity description information, represents fusion, refers to the relationship description information.

[0244] Among them, for the problem that traditional visual understanding methods ignore the semantic information of fine-grained features of images, resulting in insufficient in-depth understanding of images, the present application introduces a visual semantic tokenization method, which improves the fine-grained understanding ability of images by combining regional features with semantic information.

[0245] Specifically, traditional methods usually rely on the representation of image patches and cannot fully capture the fine-grained semantic information in images, especially when there are multiple similar or closely arranged objects in the image. Therefore, this application introduces a visual semantic tokenization method that combines the regional features of image segmentation with text instructions (i.e., entity link prompt text). The specific steps include: based on obtaining the pixel masks of visual objects in the image through image segmentation, performing semantic tagging on each visual object, combining the visual features of the visual object with the text instructions to generate the semantic tagging features of the visual object, and respectively interacting the semantic tagging features of each visual object with the semantic tagging features of other visual objects to construct a comprehensive image representation.

[0246] Among them, the process of performing semantic tagging can be represented by the following formula:

[0247] ;

[0248] Among them, refers to the semantic tagging feature, is the operation of combining the entity link prompt text with the visual features of the visual object, that is, it represents performing semantic tagging, refers to the visual features of the visual object, refers to the entity link prompt text.

[0249] Among them, assuming the regional interaction function is Inter(S1, S2,..., Sn), the comprehensive image representation G can be expressed as: G = Inter(S1, S2,..., Sn), where Si represents the visual object features of visual object i.

[0250] In a specific application, the visual semantic tokenization method can specifically include two parts. One is regional feature extraction, that is, segmenting the visual objects in the image and extracting the visual features of each visual object. The other is semantic tagging generation, that is, combining the visual features of the visual object with the text instructions to generate semantic tagging features. These semantic tagging features help the model better understand the semantic information of each visual object, thereby enhancing the recognition and linking ability of each visual object in the image.

[0251] In a specific application, such as Figure 13As shown, the method of visual semantic tokenization can be as follows: perform multi-level feature extraction on an image to obtain the multi-level image visual features of the image; based on the pixel mask of the visual object to be matched, extract the multi-level object visual features of the visual object to be matched from the multi-level image visual features; fuse the multi-level object visual features to obtain the first visual feature of the visual object to be matched; obtain the entity link hint text for the image; based on the entity link hint text and the first visual feature, perform semantic tokenization on the visual object to be matched to obtain the first semantic tokenization feature of the visual object to be matched; determine the remaining visual objects other than the visual object to be matched from the visual objects; based on the pixel masks of the remaining visual objects, extract the second visual features of the remaining visual objects from the image; based on the entity link hint text and the second visual features, perform semantic tokenization on the remaining visual objects to obtain the second semantic tokenization features of the remaining visual objects; perform multi-layer perception processing on the multi-level image visual features to obtain the image features of the image; interact the image features, the first semantic tokenization feature, and the second semantic tokenization feature to obtain the visual object features of the visual object to be matched.

[0252] In a specific application, the process of visual semantic tokenization can be as Figure 15 shown. Through a visual encoder, perform multi-level feature extraction on an image to obtain the multi-level image visual features of the image; based on the pixel masks of the visual objects (including the visual object to be matched and the remaining visual objects) obtained by image segmentation, through a mask-aware visual extractor, extract the multi-level object visual features of the visual objects from the multi-level image visual features; fuse the multi-level object visual features to obtain the visual features of the visual objects (including the first visual feature of the visual object to be matched and the second visual feature of the remaining visual objects); through a multi-layer perceptron, extract the image features of the image from the multi-level image visual features; obtain the entity link hint text for the image; determine the position features of the visual objects; input the image features of the image, the entity link hint text, the position features of the visual objects, and the visual features of the visual objects into a pre-trained large language model to perform semantic tokenization on the visual objects to obtain the semantic tokenization features, and interact the semantic tokenization features of each visual object to obtain the visual object features of each visual object. It should be noted that in this way, the visual object features of each visual object can be obtained, but in this embodiment, the visual object features of the visual object to be matched are required.

[0253] It can be understood that the visual entity linking method of the present application solves multiple bottleneck problems in traditional visual entity linking technology by introducing pixel-level visual entity linking, reverse annotation framework, knowledge enhancement and model integration, and visual semantic tokenization methods. The relevant formulas and algorithm details not only contribute to theoretical understanding but also provide strong technical support for practical applications, ensuring efficient performance in complex image and long-tail entity scenarios and can be widely applied to multiple practical scenarios. For example, in the business of identifying car brands in news pictures, using the visual entity linking method of the present application can accurately identify the car brands in news pictures and efficiently match them with brand entities in relevant databases, greatly improving the accuracy and processing speed of brand identification. In addition, the visual entity linking method of the present application can be encapsulated into a plugin and quickly integrated into different application scenarios to provide a reusable solution. Through modular design, users can easily apply this plugin to various visual entity linking tasks, such as image classification, object detection, image search, etc., to meet the needs of different industries and scenarios.

[0254] It should be noted that the visual entity linking method of the present application can accurately use pixel masks to replace traditional text queries to link visual entities in images by introducing pixel-level visual entity linking. Especially in complex scenarios, it can effectively avoid the ambiguity of text descriptions and provide more accurate entity recognition and matching. Compared with traditional text query methods, the patent solution improves the accuracy of visual entity linking in complex image scenarios. Especially when the image contains multiple objects, a blurred background, or object occlusion, it can better locate and identify the target entity. The innovative method of this patent has been verified on multiple public datasets, including OVEN-Wiki (a dataset of Wikipedia related to visual entity recognition), Wikiperson (a visual person linking dataset built based on Wikipedia), and other multi-modal datasets. By using the visual entity linking method of the present application, the performance in the following tasks has been significantly improved: on the OVEN-Wiki dataset, the entity linking accuracy of this method has increased by 18 points compared with traditional text query methods. On the Wikiperson dataset, the recognition accuracy of long-tail entities has increased by 20%. In the visual question answering task, by combining visual semantic tokenization, the image understanding ability has been improved by 5 points.

[0255] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0256] Based on the same inventive concept, an embodiment of the present application also provides a visual entity linking device for implementing the visual entity linking method described above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the visual entity linking device provided below can refer to the limitations on the visual entity linking method in the above text, and will not be repeated here.

[0257] In an exemplary embodiment, as Figure 16 shown, a visual entity linking device is provided, including: an image segmentation module 1602, an object selection module 1604, a feature extraction module 1606, a feature acquisition module 1608, and a feature matching module 1610, where:

[0258] The image segmentation module 1602 is configured to obtain an image, segment visual objects in the image, generate a pixel mask of the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image;

[0259] The object selection module 1604 is configured to obtain entity linking indication information for the image, and determine a visual object to be matched and a named entity to be matched from the visual objects according to the entity linking indication information;

[0260] The feature extraction module 1606 is configured to extract visual object features of the visual object to be matched from the image based on the pixel mask of the visual object to be matched;

[0261] The feature acquisition module 1608 is configured to obtain named entity features of the named entity to be matched;

[0262] The feature matching module 1610 is configured to match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched, and link the successfully matched visual object to be matched and the named entity to be matched.

[0263] The above visual entity linking method can, by acquiring an image, segmenting visual objects in the image, and generating pixel masks for the segmented visual objects, obtain pixel masks that accurately mark the positions of visual objects in the image through image segmentation. By acquiring entity linking indication information for the image, it can use the entity linking indication information to determine the to-be-matched visual objects and to-be-matched named entities, and then can use the pixel masks of the to-be-matched visual objects to extract visual object features of the to-be-matched visual objects from the image and obtain named entity features of the to-be-matched named entities. By matching the visual object features of the to-be-matched visual objects with the named entity features of the to-be-matched named entities and linking the successfully matched to-be-matched visual objects and to-be-matched named entities, the entire process can, by combining the image segmentation results, achieve the linking of visual objects and named entities through feature matching, thereby improving the accuracy of entity recognition.

[0264] In an exemplary embodiment, the feature extraction module is further configured to, based on the pixel mask of the to-be-matched visual object, extract a first visual feature of the to-be-matched visual object from the image, and obtain entity linking hint text for the image. Based on the entity linking hint text and the first visual feature, perform semantic marking on the to-be-matched visual object to obtain a first semantic marking feature of the to-be-matched visual object, and obtain the visual object feature of the to-be-matched visual object according to the first semantic marking feature.

[0265] In an exemplary embodiment, the feature extraction module is further configured to perform multi-level feature extraction on the image to obtain multi-level image visual features of the image, and based on the pixel mask of the to-be-matched visual object, extract multi-level object visual features of the to-be-matched visual object from the multi-level image visual features, and fuse the multi-level object visual features to obtain a first visual feature of the to-be-matched visual object.

[0266] In an exemplary embodiment, the feature extraction module is further configured to determine, from the visual objects, the remaining visual objects other than the to-be-matched visual object, based on the pixel masks of the remaining visual objects, extract second visual features of the remaining visual objects from the image, based on the entity linking hint text and the second visual features, perform semantic marking on the remaining visual objects to obtain second semantic marking features of the remaining visual objects, and interact the first semantic marking feature and the second semantic marking features to obtain the visual object feature of the to-be-matched visual object.

[0267] In an exemplary embodiment, the feature extraction module is further configured to obtain multi-level image visual features of the image, perform multi-layer perceptron processing on the multi-level image visual features to obtain image features of the image, and interact the image features, the first semantic marking feature, and the second semantic marking features to obtain the visual object feature of the to-be-matched visual object.

[0268] In an exemplary embodiment, the feature extraction module is further configured to splice the image feature and the second semantic marker feature to obtain a spliced feature, and interact the first semantic marker feature and the spliced feature through a cross-attention mechanism to obtain the visual object feature of the visual object to be matched.

[0269] In an exemplary embodiment, the object selection module is further configured to, when the entity link indication information includes visual object marker information, determine the regional coordinate information of the visual object marker region based on the visual object marker information, perform marked object recognition based on the regional coordinate information and the pixel mask of the visual object, determine the visual object marked by the visual object marker region, use the marked visual object as the visual object to be matched, and select the named entity to be matched from the entity knowledge base.

[0270] In an exemplary embodiment, the object selection module is further configured to, when the entity link indication information includes named entity specification information, perform named entity recognition on the named entity specification information, determine the named entity specified by the named entity specification information, use the specified named entity as the named entity to be matched, and use the visual object as the visual object to be matched.

[0271] In an exemplary embodiment, the feature acquisition module is further configured to perform entity description expansion on the named entity to be matched to obtain extended description information for the named entity to be matched, obtain named entity description information for the named entity to be matched, fuse the extended description information and the named entity description information to obtain fused description information for the named entity to be matched, and extract the named entity feature of the named entity to be matched from the fused description information.

[0272] In an exemplary embodiment, the feature acquisition module is further configured to perform an associated category query on the named entity to be matched, determine an associated entity associated with the category of the named entity to be matched, obtain the associated entity description information of the associated entity, and use the associated entity description information as the extended description information for the named entity to be matched.

[0273] In an exemplary embodiment, the feature acquisition module is further configured to obtain a text query statement for the image, perform semantic parsing on the text query statement, determine a query subject representing the visual object to be matched, and extract relationship description information for the query subject, and use the relationship description information as the extended description information for the named entity to be matched.

[0274] Each module in the above visual entity link device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above modules.

[0275] In an exemplary embodiment, a computer device is provided. The computer device can be a terminal or a server. Taking the computer device as a terminal as an example, its internal structural diagram can be as shown in Figure 17 the following figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a visual entity linking method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0276] Those skilled in the art can understand that Figure 17 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0277] In an embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0278] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0279] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps in the above method embodiments.

[0280] It should be noted that the data involved in this application (including but not limited to data for analysis, stored data, displayed data, etc.) are all data authorized by users or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0281] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, database, or other medium in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, artificial intelligence (AI) processors, etc., without limitation.

[0282] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0283] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A visual entity linking method, characterized in that, The method includes: Obtain an image, segment visual objects in the image, generate a pixel mask for the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image; Obtain entity link indication information for the image, and based on the entity link indication information, determine a visual object to be matched from the visual objects, and determine a named entity to be matched; Based on the pixel mask of the visual object to be matched, extract visual object features of the visual object to be matched from the image; Obtain named entity features of the named entity to be matched; Match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched, and link the successfully matched visual object to be matched and the named entity to be matched.

2. The method according to claim 1, wherein The extracting, based on the pixel mask of the visual object to be matched, visual object features of the visual object to be matched from the image includes: Based on the pixel mask of the visual object to be matched, extract first visual features of the visual object to be matched from the image, and obtain entity link hint text for the image; Based on the entity link hint text and the first visual features, perform semantic marking on the visual object to be matched to obtain first semantic marking features of the visual object to be matched; Based on the first semantic marking features, obtain visual object features of the visual object to be matched.

3. The method according to claim 2, wherein The extracting, based on the pixel mask of the visual object to be matched, first visual features of the visual object to be matched from the image includes; Perform multi-level feature extraction on the image to obtain multi-level image visual features of the image; Based on the pixel mask of the visual object to be matched, extract multi-level object visual features of the visual object to be matched from the multi-level image visual features; Fuse the multi-level object visual features to obtain first visual features of the visual object to be matched.

4. The method according to claim 2, wherein The obtaining, based on the first semantic marking features, visual object features of the visual object to be matched includes: Determine remaining visual objects other than the visual object to be matched from the visual objects, and based on the pixel masks of the remaining visual objects, extract second visual features of the remaining visual objects from the image; Based on the entity link hint text and the second visual features, perform semantic marking on the remaining visual objects to obtain second semantic marking features of the remaining visual objects; Interact the first semantic marking features and the second semantic marking features to obtain visual object features of the visual object to be matched.

5. The method according to claim 4, characterized in that The interacting the first semantic marking features and the second semantic marking features to obtain visual object features of the visual object to be matched includes: Obtain multi-level image visual features of the image, perform multi-layer perception processing on the multi-level image visual features to obtain image features of the image; Interact the image features, the first semantic marking features and the second semantic marking features to obtain visual object features of the visual object to be matched.

6. The method according to claim 5, characterized in that Performing interaction on the image feature, the first semantic marker feature, and the second semantic marker feature to obtain the visual object feature of the visual object to be matched includes: Concatenating the image feature and the second semantic marker feature to obtain a concatenated feature; Performing interaction on the first semantic marker feature and the concatenated feature through a cross-attention mechanism to obtain the visual object feature of the visual object to be matched.

7. The method according to claim 1, wherein Determining the visual object to be matched from the visual object according to the entity link indication information and determining the named entity to be matched includes: When the entity link indication information includes visual object marker information, determining the regional coordinate information of the visual object marker region based on the visual object marker information; Performing marked object recognition based on the regional coordinate information and the pixel mask of the visual object to determine the visual object marked by the visual object marker region; Using the marked visual object as the visual object to be matched and selecting the named entity to be matched from the entity knowledge base.

8. The method according to claim 1, wherein Determining the visual object to be matched from the visual object according to the entity link indication information and determining the named entity to be matched includes: When the entity link indication information includes named entity specification information, performing named entity recognition on the named entity specification information to determine the named entity specified by the named entity specification information; Using the specified named entity as the named entity to be matched and using the visual object as the visual object to be matched.

9. The method according to any one of claims 1 to 8, characterized in that Obtaining the named entity feature of the named entity to be matched includes: Performing entity description extension on the named entity to be matched to obtain extended description information for the named entity to be matched; Obtaining named entity description information for the named entity to be matched, fusing the extended description information and the named entity description information to obtain fused description information for the named entity to be matched; Extracting the named entity feature of the named entity to be matched from the fused description information.

10. The method according to claim 9, characterized in that Performing entity description extension on the named entity to be matched to obtain extended description information for the named entity to be matched includes: Performing an associated category query on the named entity to be matched to determine associated entities associated with the category of the named entity to be matched; Obtaining the associated entity description information of the associated entities and using the associated entity description information as the extended description information for the named entity to be matched.

11. The method according to claim 9, wherein Performing entity description extension on the named entity to be matched to obtain extended description information for the named entity to be matched includes: Obtaining a text query statement for the image; Performing semantic parsing on the text query statement to determine a query subject representing the visual object to be matched and extracting relationship description information for the query subject; Using the relationship description information as the extended description information for the named entity to be matched.

12. A visual entity linking device, characterized in that, The device includes: An image segmentation module, configured to obtain an image, segment visual objects in the image, generate a pixel mask of the segmented visual objects, and the pixel mask marks the positions of the visual objects in the image; An object selection module, configured to obtain entity link indication information for the image, and determine a visual object to be matched and a named entity to be matched from the visual objects according to the entity link indication information; A feature extraction module, configured to extract visual object features of the visual object to be matched from the image based on the pixel mask of the visual object to be matched; A feature acquisition module, configured to acquire named entity features of the named entity to be matched; A feature matching module, configured to match the visual object features of the visual object to be matched with the named entity features of the named entity to be matched, and link the successfully matched visual object to be matched and named entity to be matched.

13. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images